Multi-job distributed training system and method

Abstract

The invention discloses a multi-job distributed training system and method, and the system comprises a storage service module which is used for managing stored training data and distributing the training data for each training task according to the ID of each training task and the number of iterations thereof; the naming control module is used for distributing an ID (Identity) for each training task to position training data of each training task, and each training task is completed by one computing node; the host agent module is used for acquiring distributed training data corresponding to a training task from the storage server module so as to enable a computing node to perform gradient computing, and sending gradient data obtained by the computing node to a network; and receiving aggregated data from the network and sending the aggregated data to each computing node, and the switch module is used for performing aggregation calculation according to the gradient data corresponding to each training task obtained from the network to obtain the aggregation data, and sending the aggregation data to each calculation node in the network through the network.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multi-job distributed training system and method, a storage server, a computer device, a programmable switch, a computer-readable storage medium, and a computer program product. Background Art

[0002] Training a deep neural network is a huge task that involves a large amount of data and computing. Single-machine training can no longer meet the training needs, and distributed training (DT) based on multiple machines has become a research hotspot.

[0003] Currently, multiple GPUs used to perform training tasks in a distributed training network train deep neural networks locally, and each GPU sends the calculated gradients to the parameter server for gradient aggregation. Subsequently, the parameter server sends the aggregated gradients to each GPU, and each GPU performs the next iterative training based on the obtained aggregated gradients. Since the data traffic transmitted by the network during distributed training is large, communication overhead becomes the main bottleneck. For example, data communication is performed for more than half of the training process, and communication overhead limits the efficiency and scalability of distributed training jobs.

[0004] Therefore, in a distributed training network, how to reduce communication overhead and thus improve the efficiency and scalability of distributed training is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the shortcomings of the related technologies mentioned above, the purpose of the present application is to provide a multi-job distributed training system and method, a storage server, a computer device, a programmable switch, a computer-readable storage medium and a computer program product to solve the technical problem of how to reduce communication overhead in a distributed training network to improve the efficiency and scalability of distributed training.

[0006] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a multi-job distributed training system, which is applied to a network for distributed training of a deep neural network, wherein the network includes multiple computing nodes for participating in the distributed training, and the multi-job distributed training system includes: a storage service module, which is used to manage the stored training data and distribute the training data to each training task according to the ID of each training task and its number of iterations; a naming control module, which is used to assign an ID to each training task to locate the training data of each training task, wherein each training task is completed by a computing node; a host agent module, which is used to obtain the distributed training data corresponding to a training task from the storage service module to enable the computing node to perform gradient calculation, and send the gradient data obtained by the computing node to the network; and receive aggregated data from the network and send it to each computing node; a switch module, which is used to perform aggregate calculation based on the gradient data corresponding to each training task obtained from the network to obtain the aggregated data, and send the aggregated data to each computing node in the network through the network.

[0007] A second aspect of the present application provides a multi-job distributed training method, which is applied to a network for distributed training of a deep neural network, wherein the network includes multiple computing nodes for participating in the distributed training, and the multi-job distributed training method includes the following steps: assigning an ID to each training task to locate the training data of each training task; wherein each training task is completed by a computing node; the training data is managed by a storage service module, and the storage service module is used to distribute the training data to each training task based on the ID of each training task and its iteration number; obtaining the training data of the computing task so that the computing node performs gradient calculation to obtain gradient data and sends it to the network; performing aggregation calculation on the gradient data corresponding to each training task obtained from the network to obtain aggregate data and broadcasting it to the network through the network; the computing node receives the aggregate data from the network to perform the next iteration of training.

[0008] A third aspect of the present application provides a storage server, which is applied to a network for distributed training of a deep neural network, wherein the network includes multiple computing nodes for participating in the distributed training, and the storage server includes: a memory for storing training data for distributed training of the deep neural network; a processor for managing the stored training data and distributing the training data to each training task based on the ID of each training task and its number of iterations; wherein the ID of each training task is used to locate the training data of each training task, and each training task is completed by a computing node; the computing node performs gradient calculation based on the distributed training data corresponding to a training task to obtain gradient data and sends it to the network and obtains aggregated data from the network; the aggregated data is obtained by performing aggregation calculation through a programmable switch.

[0009] A fourth aspect of the present application provides a computer device, which is applied to a network for distributed training of a deep neural network, wherein the network includes multiple computing nodes for participating in the distributed training, and the computer device includes: a memory; a processor, which is used to obtain distributed training data corresponding to a training task so that the computing node can perform gradient calculation, and send the gradient data obtained by the computing node to the network; and receive aggregated data from the network and send it to each computing node; wherein each training task is completed by a computing node; wherein the training data corresponding to a training task is distributed to each training task by a storage server based on the ID of each training task and its number of iterations; the ID of each training task is used to locate the training data of each training task; and the aggregated data is obtained by performing aggregate calculations through a programmable switch.

[0010] A fifth aspect of the present application provides a computer device, which is applied to a network for distributed training of a deep neural network, wherein the network includes multiple computing nodes for participating in the distributed training, and the computer device includes: a memory; a processor, which is used to assign an ID to each training task to locate the training data of each training task, wherein each training task is completed by a computing node; wherein the training data corresponding to a training task is distributed to each training task by a storage server based on the ID of each training task and its number of iterations; the computing node performs gradient calculation based on the distributed training data corresponding to a training task to obtain gradient data and sends it to the network and obtains aggregated data from the network; the aggregated data is obtained by performing aggregate calculation on a programmable switch.

[0011] In a sixth aspect, the present application provides a programmable switch, which is applied to a network for distributed training of a deep neural network, wherein the network includes a plurality of computing nodes for participating in the distributed training, and the programmable switch includes: a network port; an aggregation module, which is used to perform aggregation calculation based on the gradient data corresponding to each training task obtained from the network port to obtain the aggregated data, and send the aggregated data to each computing node in the network through the network port; wherein each training task is completed by a computing node; wherein the training data corresponding to a training task is distributed to each training task by a storage server based on the ID of each training task and its number of iterations; the ID of each training task is used to locate the training data of each training task; and the gradient data is obtained by each computing node performing gradient calculation based on the assigned training data corresponding to a training task.

[0012] The seventh aspect of the present application provides a computer device, comprising: a storage device for storing at least one program; a processing device connected to the storage device, for calling the at least one program from the storage device and executing it to implement the multi-job distributed training method as described in any embodiment disclosed in the second aspect of the present application.

[0013] An eighth aspect of the present application provides a computer-readable storage medium storing at least one program, which, when called and executed by a processor of a computer, implements the multi-job distributed training method as described in any embodiment disclosed in the second aspect of the present application.

[0014] In a ninth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the multi-job distributed training method as described in any embodiment disclosed in the second aspect of the present application.

[0015] In summary, the multi-job distributed training system and method, storage server, computer device, programmable switch, computer-readable storage medium and computer program product provided by the present application are

[0016] The switch module is used to aggregate the gradient data and send the aggregated data to each computing node in the network, and the storage service module is used to manage the training data. This can reduce the communication overhead of the parameter server and improve the utilization of the network, thereby improving the efficiency and scalability of distributed training.

[0017] Furthermore, by simultaneously sending the same gradient blocks to different computing nodes, the shortest waiting time during aggregation can be achieved, thereby improving the computational efficiency of distributed training and reducing the overhead of the aggregator; when the host agent module receives the last gradient data during an iteration, all computing nodes in the network are started at the same time so that each computing node starts the next iteration to further improve synchronization. Moreover, the present application uses a computing node as a backup parameter server, which is enabled only when data packets are lost, reducing the amount of communication in the network during normal aggregation. And when a computing node fails during training, the computing tasks on the computing node can be transferred to other computing nodes, thereby not affecting subsequent distributed training. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The specific features of the present application are shown in the attached claims. The features and advantages of the invention involved in the present application can be better understood by referring to the exemplary embodiments and drawings described in detail below. A brief description of the drawings is as follows:

[0019] Figure 1 Shown is a topology diagram of a network for distributed training of a deep neural network in one embodiment of the present application.

[0020] Figure 2 Shown is a schematic diagram of data transmission in a network when a computing node fails in one embodiment of the present application.

[0021] Figure 3a and Figure 3b Schematic diagrams of gradient blocks obtained after gradients are divided into blocks in different embodiments of the present application are respectively shown.

[0022] Figure 4 A comparison chart showing the time it takes to perform one iteration of calculation on different models using different distributed training systems.

[0023] Figure 5 The figure shows a comparison of GPU and network utilization of different distributed training systems.

[0024] Figure 6 The figure shows a comparison of the training time of different distributed training systems at different numbers of iterations when a failure is introduced at the 70th iteration.

[0025] Figure 7 Shown is a flowchart of a multi-task distributed training method in one embodiment of the present application.

[0026] Figure 8 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following is an explanation of the implementation of the present application by means of specific embodiments. People familiar with the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification.

[0028] As described in the background technology, when training deep neural networks, data parallel distributed training, when running on ordinary GPU servers, often requires a reasonable way to support economical, efficient and scalable deep learning. This work becomes complicated because DT jobs usually involve many sub-second DT tasks executing multiple iterations on the GPU. Each task loads input data and exchanges calculated gradients. If not carefully planned, the communication time may block GPU utilization, thereby limiting the efficiency and scalability of DT jobs.

[0029] Although a cluster network architecture proposed by the industry is an explorable optimization target, for example, the NVIDIA A100 architecture recommends using 8 200Gbps network interface cards per server for GPU communication and an additional one for training data access, this architecture leads to high network costs. With recent advances in programmable switches like Tofino, people also use in-network computing (INC) to perform gradient aggregation. These systems are often called Incremental Network Aggregation (INA) and significantly reduce latency and network bandwidth utilization by eliminating issues such as software overhead on most host-based parameter servers.

[0030] Although existing in-network aggregation (INA) systems have achieved good results, they usually use programmable switches as compute accelerators for only a single job. When executing concurrent training jobs in a shared cluster, performance may degrade, and loading training data using the same network link may cause tasks to slow down. Traditional in-network aggregation (INA) systems require all training data to be partitioned and replicated before the job starts. This manual and static data management method is not conducive to fast failover.

[0031] Therefore, in the distributed training network, how to reduce the communication overhead and thus improve the efficiency and scalability of distributed training is a technical problem that needs to be solved urgently. In the prior art, using programmable switches to perform advance aggregation on the network path for transmitting gradients can reduce the number of gradients that the parameter server needs to aggregate, thereby alleviating parameter server congestion and reducing the communication overhead of the parameter server. However, at present, the switch is simply used to perform the function of gradient aggregation, and the distributed training network lacks training data management and coordination. The lack of management and coordination of training data will lead to reduced network utilization and thus reduce the efficiency of distributed training.

[0032] In view of this, some embodiments provided in the present application disclose a multi-job distributed training system, a multi-job distributed training method, a storage server, a computer device, a programmable switch, a computer-readable storage medium, and a computer program product. In the present application, a switch module is used to aggregate gradient data and send the aggregated data to each computing node in the network, and a storage service module is used to manage the training data. In this way, the communication overhead of the parameter server can be reduced and the utilization rate of the network can be improved, thereby improving the efficiency and scalability of distributed training.

[0033] In the present application, the multi-job distributed training system is applied to a network for distributed training of deep neural networks. The deep neural networks include but are not limited to convolutional neural networks for image recognition, generative adversarial networks for generating data such as images, and long short-term memory networks for speech recognition. In order to speed up the training time of the deep neural network, the training is usually completed by distributed training. The distributed training refers to dividing the tasks in an iterative training process into multiple training tasks and executing them by multiple computing nodes at the same time. In some embodiments of the present application, the computing node is a computer device composed of one or more GPUs, and the multiple computing nodes establish communication with a server or switch in the network through a network.

[0034] See also Figure 1 , which is a topological diagram of a network for distributed training of a deep neural network in one embodiment of the present application. As shown in the figure, the network includes a multi-job distributed training system 1 and a plurality of computing nodes 2 for participating in the distributed training. The multi-job distributed system is used to communicate with the plurality of computing nodes to assign training tasks and training data for each iteration to each computing node and to obtain gradient data calculated by each computing node. Each computing node performs gradient calculation using the training tasks and training data it obtains. In one embodiment, a computing node 2 includes at least one GPU for performing calculations. For example, a computing node 2 includes 1, 2, 4, 8, 16, or 32 GPUs.

[0035] In one embodiment, please continue to refer to Figure 1 As shown in the figure, the multi-job distributed system 1 includes a storage service module 10, a naming control module 11, a host agent module 12, and a switch module 13.

[0036] The naming control module 11 is used to assign an ID to each training task to locate the training data of each training task. The storage service module 10 is used to manage the stored training data and distribute the training data to each training task according to the ID of each training task and its iteration number. The host agent module 12 is used to obtain the distributed training data corresponding to a training task from the storage service module to enable the computing node to perform gradient calculation and generate gradient data and send it to the host agent module 12. The host agent module 12 sends the obtained gradient data to the switch module 13 in the network. The switch module 13 is used to perform aggregation calculation based on the gradient data corresponding to each training task obtained from the network to obtain the aggregated data and send the aggregated data to each computing node in the network through the network. For example, the switch module 13 sends the aggregated data to the host agent module 12 in the network. The host agent module 12 communicates with each computing node 2 to send the aggregated data to each computing node so that each computing node can start the next iteration.

[0037] The storage service module 10 is a software tool or software module that can process data, and it processes data (such as training data) with the help of a computer device or a hardware device in a storage server described later and an operating environment provided by an operating system. In one example, the storage service module 10 can run on a processor (CPU) connected to a memory (for example, the memory includes at least one SSD). Since the training data is read-only, in one example, when the number of computing nodes is 40 and each computing node has 8 GPUs, the present application only requires 12 SSDs (each SSD has a transmission rate of not less than 4GB / s or each SSD has a bandwidth of not less than 32Gbps) and 4 100Gbps network interface cards. Such a configuration can meet the data transmission efficiency and reduce the configuration cost of the network.

[0038] In one embodiment, the storage service module 10 includes one or more physical or logical blocks of computer instructions organized as objects, programs, or functions. However, the executable files of the modules do not have to be physically located together, but may include different commands stored in different locations, which, when logically connected together, can achieve the specified goals of the storage service module 10. In one example, the storage service module 10 includes C++ code.

[0039] The storage service module 10 is used to manage the stored training data. In one embodiment, all training data for training a deep neural network are stored in a memory, and all stored training data are centrally managed by a unified storage service module 10. For example, the storage service module 10 allocates training data according to the number of iterations set by the user and the number of computing nodes in each iteration. The storage service module 10 stores each allocated data in the memory, and the storage service module 10 also records the storage address of each allocated data. For example, 10,000 training data (e.g., 10,000 images) are needed to train a deep neural network, the user sets the number of iterations to 10, and the number of computing nodes in each iteration is 10, then the storage service module 10 divides the 10,000 images into 100 parts and stores them separately in the memory. It should be noted that when the number of GPUs in the computing node is two or more, the computing node can be allocated according to the training data it obtains.

[0040] The storage service module 10 is used to distribute training data to each training task according to the ID of each training task and its iteration number.

[0041] In one embodiment, the naming control module 11 first divides the tasks required to train the deep neural network into multiple training tasks according to the number of computing nodes, and each computing node completes one training task. In other words, the calculation performed by each computing node is a training task. Then, the naming control module 11 assigns an ID to each training task to locate the training data of each training task. To this end, the naming control module 11 is also used to determine the corresponding relationship between the ID of each training task and its number of iterations and the training data, that is, to determine the mapping between the ID of each training task and the training data, for example, to determine the storage address of the training data corresponding to each number of iterations and each ID. It should be noted that the training data corresponding to the training tasks with different IDs at the same number of iterations are different, and the training data corresponding to the training tasks with the same ID at different numbers of iterations are also different. Then, the naming control module 11 also sends the corresponding relationship to the storage service module 10. Finally, the storage service module 10 distributes training data for the training task based on the corresponding relationship and the request data of a training task generated by the host agent module 12. Specifically, the host agent module 12 generates the request data according to the ID assigned by the naming control module 11 through the switch module 13, and the request data includes the ID of the training task and its iteration number. The storage service module 10 can locate which training data is required for the training task of this ID in this iteration based on the corresponding relationship and the ID of the training task and its iteration number included in the request data, and send the training data to the host agent module 12 cache through the switch module 13.

[0042] The naming control module 11 is a software tool or software module that can process data. It processes data with the help of the hardware device in the computer device and the operating environment provided by the operating system, for example, determining the corresponding relationship between the ID of each training task and its number of iterations and resources. In one example, the naming control module 11 can run on the processor (CPU) of the computer device.

[0043] In one embodiment, the naming control module 11 includes one or more physical or logical blocks of computer instructions organized as objects, programs or functions. However, the executable files of the modules do not have to be physically located together, but may include different commands stored in different locations, which, when logically connected together, can achieve the specified goals of the naming control module 11. In one example, the naming control module 11 includes Python code.

[0044] In one embodiment, the naming control module 11 is used to assign an ID to each training task to locate the training data of each training task, wherein each training task is completed by a computing node. Specifically, the naming control module 11 is used to assign an ID to each training task in a manner that is the same or similar to that described above and is not repeated here. It should be noted that when the number of GPUs in a computing node is two or more, the computing node can redistribute the acquired training tasks according to the number of GPUs.

[0045] The naming control module 11 is also used to determine the correspondence between the ID of each training task and its number of iterations and the training data, wherein the steps for the naming control module 11 to determine the correspondence between the ID of each training task and its number of iterations and the training data are the same or similar to those described above and will not be repeated here.

[0046] Furthermore, the naming control module also sends the corresponding relationship to the storage service module, and then the storage service module can determine the training data corresponding to the request data based on the corresponding relationship to send the training data to the host agent module through the switch module, and then the host agent module can cache the training data.

[0047] The host agent module 12 is a software tool or software module that can process data, and it processes data (such as gradient data) with the help of the hardware device in the computer device and the operating environment provided by the operating system. In one example, the host agent module 12 can run on a processor (CPU) connected to the memory.

[0048] In one embodiment, the host agent module 12 includes one or more physical or logical blocks of computer instructions organized as objects, programs, or functions. However, the executable files of the modules do not have to be physically located together, but may include different commands stored in different locations, which, when logically connected together, can achieve the specified goals of the host agent module 12. In one example, the host agent module 12 includes C++ code.

[0049] The host agent module 12 is used to obtain the distributed training data corresponding to a training task from the storage service module 10 to enable the computing node 2 to perform gradient calculation, and send the gradient data obtained by the computing node 2 to the network; and receive aggregated data from the network and send it to each computing node.

[0050] In one embodiment, each computing node 2 directly communicates with the host agent module 12 to perform data interaction. For example, the computing node 2 receives aggregate data sent by the switch module 13 from the host agent module 12; for another example, the computing node 2 sends gradient data to the host agent module 12 so that the host agent module 12 sends the gradient data to the switch module for aggregate calculation; for another example, the computing node obtains the training data corresponding to a training task obtained from the storage service module 10 and cached by the host agent module 12.

[0051] In order to cache the gradient data and aggregated data, the host agent module 12 includes a training data buffer and a gradient data buffer, wherein the training data buffer is used to buffer the training data obtained from the storage service module, and the gradient data buffer is used to buffer the gradient data obtained from the computing node. In one example, the training data buffer and the gradient data buffer are constructed using DPDK to minimize the software stack reaching the network interface card, thereby reducing the fluctuation of software delay. For example, the host agent module continuously buffers the training data to the training data buffer when its training data buffer is idle and during the idle period of bandwidth (for example, when the host agent module does not receive gradient data and does not send gradient data).

[0052] In distributed training, a computing node can only start the next round of iterative training after receiving the aggregated data calculated by the switch module. The completion time of the aggregate calculation is determined by the computing node that sends the gradient data last. Therefore, if their gradient data are sent at the same time, we can achieve the shortest waiting time. Furthermore, if one computing node sends gradient block 1 first, and another computing node sends gradient block 2 first. They are sent in different orders, and the two gradient blocks will need to use different aggregators on the switch and consume twice the registers. For example, gradient block 1 occupies aggregator 1, and gradient block 2 occupies aggregator 2. Among them, each aggregator includes multiple registers. The gradient blocks will be described in detail later.

[0053] To this end, first, in an iteration process, the computing node divides a gradient obtained by executing a training task according to a preset block rule to obtain a gradient block. In other words, all computing nodes divide the gradient into blocks according to the preset block rule. Among them, the preset block rule is an equal-length block rule or an unequal-length block rule. The equal-length block rule means that the data length of each gradient block after division is equal. For example, a preset number of bytes are divided into a gradient block during division, such as 1MB bytes are divided into a gradient block. The unequal-length block rule means that the data length of each gradient block after division is not completely equal. For example, different preset numbers of bytes are divided into different gradient blocks during division.

[0054] Since the neural network model of each computing node is consistent, the array form of the gradient obtained by each computing node when executing a training task in one iteration is consistent (for example, the gradient calculated by each computing node is a one-dimensional array of 256 numbers), and the data length of the gradient obtained by each computing node is consistent under the same digital representation method in the CPU. In other words, the data size of each gradient is also consistent. For example, the gradient calculated by each computing node includes 100MB bytes. Therefore, the gradient blocks obtained by all computing nodes after dividing the gradient into blocks are also consistent.

[0055] Furthermore, each gradient block needs to be provided with a marker to identify the position of the gradient block in the gradient. Figure 3a and Figure 3b , respectively show schematic diagrams of gradient blocks obtained after dividing the gradient in different embodiments of the present application. As shown in the figure, gradient block 1 represents the starting area of ​​the gradient, gradient block 2 represents the middle area of ​​the gradient, and gradient block 3 represents the ending area of ​​the gradient.

[0056] After the computing node calculates the gradient block, the computing node generates gradient data sent to the host agent module. The gradient data includes an ID of a training task and a gradient block, and further, the gradient data also includes an identifier of the gradient block.

[0057] It should be noted that when a computing node includes multiple GPUs, the computing node will also locally aggregate the gradients obtained by each GPU to obtain a gradient corresponding to the training task, and then divide the gradient blocks. Furthermore, when the computing node divides the gradient, it can divide the gradient after a training task is fully calculated, or it can divide only part of the gradient after a partial calculation is completed. After the division, the calculated gradient blocks are sent to the host agent module. For example, the gradient of a training task includes 100MB, and a computing node has only completed the calculation of the partial gradient of the first 10MB. In this case, the first 10MB will be divided into blocks and sent to the host agent module.

[0058] In order to determine that the same gradient block (e.g. gradient block 1) of each computing node is ready on the host agent module, in other words, to ensure that the same gradient block of each computing node is already in the gradient data buffer of the host agent module. The host agent module is also used to generate a bitmap of each computing node based on the gradient data received from each computing node and send it to the switch module. The bitmap uses a value of 0 or a value of 1 at a data position to indicate whether a gradient block exists. For example, a value of 1 indicates that the gradient block corresponding to the data position is received, and a value of 0 indicates that the gradient block corresponding to the data position is not received. In one example, please continue to refer to Figure 3a and Figure 3b As shown in the figure, each computing node divides the gradient into gradient block 1, gradient block 2, and gradient block 3. The bitmap of a computing node generated by the host agent module is (1, 0, 0), which means that the host agent module has received the gradient block 1 sent by the computing node, but has not received the gradient block 2 and the gradient block 3.

[0059] Furthermore, the switch module is further used to generate a global bitmap based on the received bitmaps of each computing node and send it to the host agent module. The global bitmap uses a value of 0 or a value of 1 at a data position to indicate whether the host agent module has received all the gradient blocks at the data position. For example, a value of 1 indicates that the host agent module has received all the gradient blocks corresponding to the data position sent by each computing node, and a value of 0 indicates that the host agent module has not received all the gradient blocks corresponding to the data position sent by each computing node.

[0060] In an example, see Figure 3a and Figure 3b As shown in the figure, each computing node divides the gradient into gradient block 1, gradient block 2, and gradient block 3. The host agent module generates bitmaps of the three computing nodes, namely (1, 1, 0), (1, 0, 0), and (1, 0, 0) and sends them to the switch module. The switch module generates a global bitmap (1, 0, 0), where 1 indicates that the host agent module has received all the gradient blocks 1 sent by each computing node, and the two-digit number 0 indicates that the host agent module has not received all the gradient blocks 2 and 3 sent by each computing node.

[0061] After the host agent module sends the generated global bitmap to the host agent module, the host agent module is further used to send the gradient data of each computing node corresponding to the data position with a value of 1 in the global bitmap to the switch module.

[0062] In one embodiment, the host agent module sends a gradient data as a data packet to be aggregated to the switch module, so that the switch module can allocate an aggregator to perform aggregation calculation for the data packets to be aggregated corresponding to the same gradient data. For example, when the data length of the gradient data itself is less than the number of registers in the aggregator, the host agent module can send a gradient data as a data packet to be aggregated to the switch module. Figure 3a and Figure 3b As shown, the host agent module sends the three gradient data corresponding to the gradient block 1 of the three computing nodes in the cache as data packets to be aggregated to the switch module at the same time so that the switch module allocates the same aggregator to perform aggregation calculation on the three data packets to be aggregated.

[0063] In another embodiment, when the gradient data of each computing node corresponding to the data position with a value of 1 in the global bitmap is sent to the switch module at the same time, the host agent module first divides the gradient block in each gradient data according to the number of registers in the aggregator, specifically, divides the gradient block using the number of registers in the aggregator as the division length so that the divided data is less than or equal to the number of registers in an aggregator. Further, after the host agent module completes the division, it generates a data packet to be aggregated according to the divided data, the position of the divided data, and the ID.

[0064] For example, Figure 3a The gradient block 1 generates data packet 1 to be aggregated, data packet 2 to be aggregated and data packet 3 to be aggregated, the data in data packet 1 to be aggregated is located from the start position to the 92nd position of the gradient block 1, the data in data packet 2 to be aggregated is located from the 93rd position to the 184th position of the gradient block 1, and the data in data packet 3 to be aggregated is located from the 185th position to the 276th position of the gradient block 1. The host agent module sends multiple data packets to be aggregated corresponding to each gradient data to the switch module at the same time, so that the switch module allocates the same aggregator to the same data packet to be aggregated corresponding to different gradient data for aggregation calculation.

[0065] It should be noted that a sending rule may be preset in the host agent module, and the sending rule specifies the sending order of the gradient blocks. Then, when there are multiple 1s in the global bitmap, the sending order of the gradient blocks may be coordinated according to the sending rule. For example, the sending rule is to first send the gradient blocks close to the starting position of the gradient. For another example, the sending rule is to first send the data close to the starting position of the gradient block.

[0066] In this way, the shortest waiting time during aggregation can be achieved by sending the same gradient blocks to different computing nodes at the same time, thereby improving the computational efficiency of distributed training and the overhead of the aggregator.

[0067] In order to further improve synchronization, the host agent module is also used to start all computing nodes in the network at the same time when the last gradient data is received during an iteration process so that each computing node starts the next iteration. Specifically, when all gradient blocks corresponding to each training task of an iterative training process are received, the host agent module starts the computing nodes at the same time to start the next iteration. For example, the host agent module uses the last gradient data as a barrier signal. When the last gradient data arrives, the host agent module immediately releases the waiting computing nodes to start the next round of iteration.

[0068] Since computing nodes may fail during training, the training efficiency of deep neural networks will be greatly reduced if the failures cannot be monitored and handled in a timely and effective manner. However, if a computing node misses a single acknowledgment (ACK) during the gradient data transmission process, this does not mean that the computing node has failed. It is likely due to factors such as network instability.

[0069] To this end, when a computing node misses a single ACK, the computing node will retransmit the gradient data to the host agent module. When a computing node retransmits the gradient data to the host agent module multiple times within a preset time, the host agent module will generate a fault report of the computing node and send the fault report to the naming control module. The preset time is 3000 milliseconds for example. For further information, please refer to Figure 2 , which is a schematic diagram of data transmission in a network when a computing node fails in one embodiment of the present application. After receiving the fault report sent by the host agent module 12, the naming control module 11 determines the number of computing nodes that can work normally in the network according to the acquired fault report and sends it to the storage service module 10, so that the storage service module 10 reallocates training data according to the number of computing nodes that can work normally.

[0070] In one embodiment, the storage service module 10 redistributes all training data corresponding to the number of iterations when the computing node fails according to the number of computing nodes that can work normally. For example, during one iteration, the storage service module distributes 100 images to 5 computing nodes, but after one of the computing nodes fails, the storage service module distributes the 100 images to the remaining 4 computing nodes for calculation.

[0071] In this way, when a computing node fails, subsequent iterative calculations will not be affected, thereby improving the success rate of distributed training.

[0072] The switch module 13 is a software tool or software module that can process data, and is used to describe the corresponding processing and operation of the data plane of the programmable switch on the data packet. For example, the switch module describes the parsing, processing, modification and forwarding logic of the data packet. In an actual embodiment, the programmable switch is exemplified as a 64×100 Gbps programmable switch.

[0073] In one embodiment, the switch module 13 includes one or more physical or logical blocks of computer instructions organized as objects, programs, or functions. However, the executable files of the modules do not have to be physically located together, but may include different commands stored in different locations, which, when logically connected together, can achieve the specified goals of the switch module 13. In one example, the switch module 13 includes P4 code. P4 language is a programming language for describing data plane behavior.

[0074] The switch module is used to perform aggregation calculation based on the gradient data corresponding to each training task obtained from the network to obtain the aggregated data, and send the aggregated data to each computing node in the network through the network.

[0075] In one embodiment, when sending the gradient data, the host agent module sends one gradient data as one data packet to be aggregated. Then the switch module allocates the same aggregator for the same gradient data of different computing nodes for aggregation calculation. Among them, the aggregator used for aggregation in the switch is composed of multiple registers, and the number of registers in the aggregator is the same as the size of a data packet. For example, if a data packet processed by the switch includes 92 bits, the aggregator includes 92 registers. An aggregator is used to aggregate the gradient data corresponding to the same gradient block of different computing nodes. For example, an aggregator is used to aggregate the gradient data corresponding to the gradient block 1 of different computing nodes. When the switch module aggregates all the gradient data sent by all computing nodes in one iteration, it generates an aggregate data, and sends the aggregate data to each computing node in the network through the network. For example, the switch module sends the aggregate data to each computing node through the host agent module, so that each computing node can perform the next iteration according to the aggregate data.

[0076] In another embodiment, when the gradient data of each computing node corresponding to the data position with a value of 1 in the global bitmap is sent to the switch module at the same time, the host agent module sends multiple data packets to be aggregated corresponding to each gradient data to the switch module at the same time; then the switch module provides an aggregator for the multiple data packets to be aggregated corresponding to each gradient data to perform aggregation calculation. Specifically, the switch module assigns the same aggregator to the same data packet to be aggregated corresponding to different gradient data for aggregation calculation. The same data packet to be aggregated refers to the data in the data packet to be aggregated being located at the same position of the gradient block. For example, data packets 1 to be aggregated are all located from the starting position to the 92nd position of different gradient blocks, then data packets 1 to be aggregated of different gradient data are assigned the same aggregator for calculation.

[0077] At present, the size of the data packet that the switch can aggregate at one time is related to the number of registers. Therefore, the switch module of the present application allows the data packet to be aggregated to access the register of the switch at the egress processing stage of the pipeline. Specifically, the present application allows a data packet to be aggregated to access the register at the egress processing stage of the pipeline to increase the number of available registers for aggregation in a single pipeline. This increases the payload size of the data packet processed by the switch pipeline, which leads to a higher goodput under the same number of data packets.

[0078] Furthermore, in order to ensure that the data packets to be aggregated received by the switching module are data packets that are allowed to access the register of the switch, the switch module is also used to perform an initial check on the acquired data packets to be aggregated to determine whether the data packets to be aggregated are allowed to access the aggregator of the switch for aggregation calculation. Specifically, when the switch module receives the data packets to be aggregated, it performs an initial check on the data packets to be aggregated. Data packets to be aggregated that pass the initial check can be allowed to access the register of the switch in the pipeline for aggregation calculation. For data packets that fail the initial check, the switch module will forward the data packets to be aggregated normally, that is, forward them according to the destination address of the data packets to be aggregated.

[0079] In one embodiment, during the initial check, the switch module determines whether the ID corresponding to the data packet to be aggregated is a data packet registered by the naming control module, whether it is a data packet that will cause aggregation to overflow, and whether it is a data packet with a parameter server as the destination address. If the ID corresponding to the data packet to be aggregated has been registered by the naming control module, the data packet to be aggregated will not cause aggregation to overflow, and the data packet to be aggregated is not a data packet with a parameter server as the destination address, then the data packet to be aggregated can pass the initial check; otherwise, the data packet to be aggregated fails the initial check.

[0080] After aggregation, the switch module will also determine whether the gradient data of all training tasks have been aggregated. If the gradient data of all training tasks in one iteration are not aggregated, the aggregated data obtained by aggregation will be discarded. If it is determined that the gradient data of all training tasks in one iteration are aggregated, the switch module is also used to recycle the aggregated data after aggregation to read the aggregated data. For example, after aggregation, the switch module determines whether the gradient data of all training tasks have been aggregated by determining whether the data packet to be aggregated is the last data packet in one iteration. If it is not the last data packet to be aggregated, the aggregation is not completed, otherwise the aggregation is completed. Further, when the aggregated data is recirculated, the switch module does not need to perform an initial check on the aggregated data, and only needs to re-pipeline the aggregated data. It should be noted that re-pipelining the aggregated data is only for reading all the bits of the aggregated data. In other words, 2n bits of aggregated data can be written in a single memory access, but only n bits can be read, so it is necessary to re-circulate once to read 2n bits of aggregated data.

[0081] In order to improve the computing efficiency of the switch module and reduce the communication volume, the switch module will not back up the aggregated data. When a computing node does not receive the aggregated data, the computing node will retransmit the gradient data, and then the host agent module will retransmit the data packet to be aggregated corresponding to the gradient data. The switch module forwards the retransmitted data packet to be aggregated to the parameter server module so that the parameter server module can send the aggregated data to the switch module, and then the switch module can send the aggregated data to the computing node that has not received the aggregated data. It should be noted that the destination address of the retransmitted data packet to be aggregated is the parameter server module. The parameter server module will back up the aggregated data of each iteration, and the parameter server module is configured on any computing node in the network. The parameter server module is a software tool or software module that can process data, which is used to describe the corresponding processing and operation of the computing node on the data.

[0082] The present application does not set up a separate parameter server, but uses a computing node as a backup parameter server, which is enabled only when data is lost, thereby reducing the communication volume in the network during normal aggregation.

[0083] See also Figure 4 , which is a comparison chart showing the time taken by different distributed training systems to perform one iteration of calculation on different models. As shown in the figure, the calculation time of the distributed training system of the present application in one iteration is Figure 4All deep neural network models shown have shorter computation time in one iteration than the ATP system, SwitchML-1 system, and SwitchML-4 system.

[0084] Under the same workload, higher GPU and network utilization will result in better performance. See Figure 5 , showing a comparison chart of GPU and network utilization of different distributed training systems. As shown in the figure, we executed 400 different training task combinations in three distributed training systems (ATP system, SwitchML-1 system and the multi-job distributed training system of this application corresponding to the red line) to compare the utilization of GPU and network. The GPU utilization of this application is improved by 100% and 50% respectively relative to that of the ATP system and SwitchML-1 system. For the network, this application achieves 3.2 times and 2.0 times the utilization of the ATP system and SwitchML-1 system.

[0085] See also Figure 6 , which shows a comparison of the training time of different distributed training systems at different numbers of iterations when a fault is introduced at the 70th iteration. As shown in the figure, we run 120 VGG16 iterations using three computing nodes in three distributed training systems (ATP system, SwitchML-system 1, and the multi-job distributed training system of the present application corresponding to the red line). At the 70th iteration, we introduced a fault by shutting down one computing node. Then, the computing node was restarted at the 100th iteration. As shown in the figure, after the fault is injected, all computing nodes in the ATP system and the SwitchML-1 system stop running and cannot be recovered. In contrast, the multi-job distributed training system of the present application automatically retries the current iteration after a few seconds of recovery time, and then continues the next iteration using the remaining two computing nodes. We observe that each iteration from the 71st to the 100th iteration takes longer because there are only two computing nodes instead of three computing nodes. We can see that the training time is immediately restored after the computing node is restarted at the 100th iteration.

[0086] In summary, the present application discloses a multi-job distributed training system, which utilizes a switch module to aggregate gradient data and sends the aggregated data to each computing node in the network, and utilizes a storage service module to manage the training data. In this way, the communication overhead of the parameter server can be reduced and the utilization rate of the network can be improved, thereby improving the efficiency and scalability of distributed training.

[0087] In addition, the present application also achieves the shortest waiting time during aggregation by simultaneously sending the same gradient blocks to different computing nodes, thereby improving the computational efficiency of distributed training and the overhead of the aggregator; when the host agent module receives the last gradient data during an iteration, it simultaneously starts all the computing nodes in the network so that each computing node starts the next iteration to further improve synchronization. Moreover, the present application uses a computing node as a backup parameter server, which is only enabled when data is lost, reducing the amount of communication in the network during normal aggregation. And when a computing node fails during training, the computing tasks on the computing node can be transferred to other computing nodes, thereby not affecting subsequent distributed training.

[0088] In one embodiment, see Figure 7 , which is a flowchart of a multi-job distributed training method in one embodiment of the present application. As shown in the figure, the multi-job distributed training method includes steps S10, S11, S12 and step S13. The multi-job distributed training method can be executed by each device in the network for distributed training of a deep neural network as described above (for example, a storage server configured with the storage service module, a computer device configured with a naming control module, a computer device configured with a host agent module, and a programmable switch configured with a switch module).

[0089] In step S10, a computer device equipped with a naming control module assigns an ID to each training task to locate the training data of each training task. Each training task is completed by a computing node; the training data is managed by a storage service module, and the storage service module is used to distribute the training data to each training task according to the ID of each training task and its iteration number.

[0090] In step S11, a computer device equipped with a host agent module obtains training data of a computing task to enable the computing node to perform gradient calculation to obtain gradient data and send the gradient data to the network.

[0091] In step S12, the programmable switch configured with the switch module performs aggregation calculation on the gradient data corresponding to each training task obtained from the network to obtain aggregated data and broadcasts it to the network through the network.

[0092] In step S13, the computing node receives aggregated data from the network to perform training for the next iteration.

[0093] Here, the working mode of each step in the multi-job distributed training method of the present application is the same as or similar to the working mode of each module in the above-mentioned multi-job distributed training system, and will not be repeated here.

[0094] In some embodiments, the present application further provides a storage server, which is applied to a network for distributed training of a deep neural network, wherein the network includes a plurality of computing nodes for participating in the distributed training. The storage server includes a memory and a server. Further, the storage server also includes a network port for connecting to a network.

[0095] In some embodiments, the memory may include a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store training data or store the software program corresponding to the storage service module described above, and the processor executes the software program. In one example, the memory is exemplified by an SSD.

[0096] In some embodiments, the processor includes an integrated circuit chip with signal processing capabilities; or a general-purpose processor, for example, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a discrete gate or transistor logic device, a discrete hardware component, which can implement or execute the functions of the disclosed methods and modules in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0097] The memory is used to store training data for distributed training of a deep neural network.

[0098] The processor is used to manage the stored training data and distribute the training data to each training task according to the ID of each training task and its iteration number. The ID of each training task is used to locate the training data of each training task, and each training task is completed by a computing node. The computing node performs gradient calculation based on the distributed training data corresponding to a training task to obtain gradient data and sends it to the network and obtains aggregate data from the network. The aggregate data is obtained by performing aggregate calculation through a switch.

[0099] The way in which the processor manages the training data and distributes the training data to each training task is the same or similar to the working mode of the storage service module in the above-mentioned multi-job distributed training system, and will not be repeated here.

[0100] In some embodiments, the present application further provides a computer device configured to perform the corresponding functions of the host agent module in the above-mentioned multi-job distributed training system, and the computer device is applied to a network for distributed training of a deep neural network, and the network includes multiple computing nodes for participating in the distributed training. The computer device includes a memory and a server. Further, the computer device also includes a network port for connecting to a network.

[0101] In some embodiments, the memory of the computer device may include a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory of the computer device is used to store the software program corresponding to the host agent module described above, and the processor of the computer device executes the software program.

[0102] In some embodiments, the processor of the computer device includes an integrated circuit chip with signal processing capabilities; or a general-purpose processor, for example, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a discrete gate or transistor logic device, a discrete hardware component, which can implement or execute the functions of the disclosed methods and modules in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0103] The processor of the computer device is used to obtain the distributed training data corresponding to a training task so that the computing node can perform gradient calculation, and send the gradient data obtained by the computing node to the network; and receive aggregated data from the network and send it to each computing node. Each training task is completed by a computing node, and the training data corresponding to a training task is distributed to each training task by a storage server according to the ID of each training task and its iteration number. The ID of each training task is used to locate the training data of each training task, and the aggregated data is obtained by performing aggregate calculation through a switch.

[0104] In this embodiment, the working mode of the processor of the computer device is the same as or similar to the working mode of the host agent module in the above-mentioned multi-job distributed training system, and will not be repeated here.

[0105] In some other embodiments, the present application also provides a computer device configured to perform the corresponding functions of the naming control module in the above-mentioned multi-job distributed training system, and the computer device is applied to a network for distributed training of a deep neural network, and the network includes multiple computing nodes for participating in the distributed training. The computer device includes a memory and a server. Further, the computer device also includes a network port for connecting to a network.

[0106] In some embodiments, the memory of the computer device may include a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory of the computer device is used to store the software program corresponding to the naming control module described above, and the processor of the computer device executes the software program.

[0107] In some embodiments, the processor of the computer device includes an integrated circuit chip with signal processing capabilities; or a general-purpose processor, for example, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a discrete gate or transistor logic device, a discrete hardware component, which can implement or execute the functions of the disclosed methods and modules in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0108] The processor of the computer device is used to assign an ID to each training task to locate the training data of each training task, wherein each training task is completed by a computing node, and the training data corresponding to a training task is distributed to each training task through a storage server based on the ID of each training task and its number of iterations. The computing node performs gradient calculation based on the distributed training data corresponding to a training task to obtain gradient data and sends it to the network and obtains aggregated data from the network, and the aggregated data is obtained by performing aggregation calculation through a switch.

[0109] In this embodiment, the working mode of the processor of the computer device is the same as or similar to the working mode of the naming control module in the above-mentioned multi-job distributed training system, and will not be repeated here.

[0110] In one embodiment, the present application also provides a programmable switch, which is applied to a network for distributed training of a deep neural network, wherein the network includes a plurality of computing nodes for participating in the distributed training. The switch includes a network port and an aggregation module. In an actual embodiment, the programmable switch is exemplified as a 64×100Gbps programmable switch. In one example, the programmable switch can also be exemplified as a tofino switch.

[0111] In one embodiment, the network port is used to connect the switch to the network, and the network port includes an Ethernet interface, a Fibre Channel interface, etc.

[0112] The processor of the programmable switch includes an integrated circuit chip with signal processing capabilities; or a general-purpose processor, for example, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a discrete gate or transistor logic device, a discrete hardware component, which can implement or execute the functions of the disclosed method steps and modules in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor of the programmable switch also includes registers for aggregate calculations. In one example, the processor of the programmable switch includes a Barefoot Tofino chip.

[0113] Furthermore, the programmable switch also includes a memory, and the switch module mentioned above can be configured in the memory of the programmable switch.

[0114] See also Figure 8 , which is a schematic diagram of the structure of a computer device in an embodiment of the present application, wherein the computer device 3 includes a storage device 30 and a processing device 31 connected to the storage device 30. Furthermore, the computer device also includes an interface device 32.

[0115] In some embodiments, the storage device 30 is used to store at least one program, and the at least one program can be executed by the processing device 31 to coordinate the storage device 30 to execute and implement the above Figure 7The multi-job distributed training method including steps S10-S13. Here, the storage device 30 includes but is not limited to: a read-only memory, a random access memory, and a non-volatile memory. For example, the storage device 30 includes a flash memory device or other non-volatile solid-state storage device. In some embodiments, the storage device 30 may also include a memory remote from one or more processing devices 31, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network, a wide area network, a storage area network, etc., or a suitable combination thereof. The memory controller may control access to the memory by other components of the device such as a CPU and a peripheral interface.

[0116] In some embodiments, the processing device 31 includes one or more processors. The processing device 31 can be operable to perform data read and write operations with the storage device 30. The processing device 31 includes one or more general-purpose microprocessors, one or more special-purpose processors, one or more digital signal processors, one or more field programmable logic arrays, or any combination thereof.

[0117] In some embodiments, the interface device 32 includes at least one interface unit, each of which is used to output a visual interface, receive a human-computer interaction event generated according to the operation of a technician, etc. For example, the interface device 32 includes but is not limited to: a serial interface such as an HDMI interface or a USB interface, or a parallel interface, etc. In one embodiment, the interface device 32 also includes a network communication unit, which is a device for data transmission using a wired or wireless network, examples of which include but are not limited to: an integrated circuit including a network card, a local area network module such as a WiFi module or a Bluetooth module, a wide area network module such as a mobile network, etc.

[0118] The present application also provides a computer-readable storage medium storing at least one program, wherein when the at least one program is called and executed by a processor of a computer, the at least one program implements the above-mentioned Figure 7 The multi-job distributed training method includes steps S10-S13.

[0119] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the above-mentioned related steps to implement the above-mentioned Figure 7 The multi-job distributed training method includes steps S10-S13.

[0120] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device equipped with the storage medium to execute all or part of the steps of the method described in each embodiment of the present application.

[0121] In the embodiments provided in the present application, the computer storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0122] In one or more exemplary aspects, the functions described in the multi-job distributed training method described in the present application can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored or transmitted as one or more instructions or codes to a computer-readable medium. The steps of the method or algorithm disclosed in the present application can be embodied in a processor-executable software module, where the processor-executable software module can be located on a tangible, non-temporary computer storage medium. A tangible, non-temporary computer storage medium can be any available medium that a computer can access.

[0123] The flowchart and block diagram in the accompanying drawings described in the present application illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. Based on this, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be realized by a dedicated hardware-based system that performs a specified function or operation, or can be realized by a combination of dedicated hardware and computer instructions.

[0124] In summary, the present application discloses a multi-job distributed training system, a multi-job distributed training method, a storage server, a computer device, a programmable switch, a computer-readable storage medium, and a computer program product, which utilize a switch module to aggregate gradient data and send the aggregated data to each computing node in the network, and utilize a storage service module to manage the training data. In this way, the communication overhead of the parameter server can be reduced and the utilization rate of the network can be improved, thereby improving the efficiency and scalability of distributed training.

[0125] Furthermore, by simultaneously sending the same gradient blocks to different computing nodes, the shortest waiting time during aggregation is achieved, thereby improving the computational efficiency of distributed training and the overhead of the aggregator; when the host agent module receives the last gradient data during an iteration, all computing nodes in the network are started at the same time so that each computing node starts the next iteration to further improve synchronization. Moreover, the present application uses a computing node as a backup parameter server, which is enabled only when data is lost, reducing the amount of communication in the network during normal aggregation. And when a computing node fails during training, the computing tasks on the computing node can be transferred to other computing nodes, thereby not affecting subsequent distributed training.

[0126] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A multi-job distributed training system, applied to a network for distributed training of a deep neural network, wherein the network comprises a plurality of computing nodes for participating in the distributed training, characterized in that: The multi-job distributed training system comprises: A storage service module, used to manage the stored training data and distribute the training data to each training task according to the ID of each training task and its iteration number; A naming control module, used to assign an ID to each training task to locate the training data of each training task, wherein each training task is completed by a computing node; A host agent module, configured to obtain the distributed training data corresponding to a training task from the storage service module to enable the computing node to perform gradient calculation, and send the gradient data obtained by the computing node to the network; and receive aggregated data from the network and send it to each computing node; The switch module is used to perform aggregation calculation based on the gradient data corresponding to each training task obtained from the network to obtain the aggregated data, and send the aggregated data to each computing node in the network through the network.

2. The multi-task distributed training system according to claim 1, characterized in that: The naming control module is also used to determine the corresponding relationship between the ID of each training task and its iteration number and the training data.

3. The multi-task distributed training system according to claim 2, characterized in that: The naming control module also sends the corresponding relationship to the storage service module.

4. The multi-task distributed training system according to claim 3, characterized in that: The storage service module distributes training data to each training task according to the ID of each training task and its iteration number, including: distributing training data to the training task based on the corresponding relationship and the request data of a training task generated by the host agent module; The request data includes the ID of the training task and its iteration number.

5. The multi-task distributed training system according to claim 1, characterized in that: The host agent module is also used to send a fault report of a failed computing node to the naming control module.

6. The multi-task distributed training system according to claim 5, characterized in that: The naming control module is also used to determine the number of computing nodes that can operate normally in the network based on the fault reports of the acquired computing nodes and send them to the storage service module so that the storage service module can reallocate training data based on the number of computing nodes that can operate normally.

7. The multi-task distributed training system according to claim 6, characterized in that: The storage service module reallocates the received training data according to the number of computing nodes that can work normally, including: reallocating all training data corresponding to the number of iterations when the computing node fails according to the number of computing nodes that can work normally.

8. The multi-task distributed training system according to claim 5, characterized in that: When a computing node retransmits gradient data to the host agent module multiple times within a preset time, the host agent module generates a fault report of the computing node.

9. The multi-task distributed training system according to claim 1, characterized in that: The gradient data includes an ID of a training task and a gradient block; wherein the gradient block is obtained by dividing the gradient obtained by a computing node executing the training task according to a preset block rule.

10. The multi-task distributed training system according to claim 1 or 9, characterized in that: The host agent module is also used to generate a bitmap of each computing node based on the gradient data sent by each computing node and send it to the switch module; wherein the bitmap of each computing node is represented by a value 0 or a value 1 at a data position; a value 1 indicates that the gradient block corresponding to the data position is received; and a value 0 indicates that the gradient block corresponding to the data position is not received.

11. The multi-task distributed training system according to claim 10, characterized in that: The switch module is also used to generate a global bitmap based on the bitmaps received from each computing node and send it to the host agent module; wherein the global bitmap is represented by a value of 0 or a value of 1 at a data position, a value of 1 indicates that the host agent module has received all the gradient blocks corresponding to the data position sent by each computing node; a value of 0 indicates that the host agent module has not received all the gradient blocks corresponding to the data position sent by each computing node.

12. The multi-task distributed training system according to claim 11, characterized in that: The step of sending the gradient data obtained by the computing node to the network by the host agent module includes: sending the gradient data of each computing node corresponding to the data position with a value of 1 in the global bitmap to the switch module.

13. The multi-task distributed training system according to claim 12, characterized in that: The host agent module is also used to start all computing nodes in the network simultaneously when receiving the last gradient data during an iteration so that each computing node starts the next iteration.

14. The multi-task distributed training system according to claim 9, characterized in that: The switch module performs aggregation calculations based on the gradient data corresponding to each training task obtained from the network, including: providing an aggregator for one or more data packets to be aggregated corresponding to each gradient data to perform aggregation calculations; wherein the data packets to be aggregated are obtained by the host agent module based on the number of registers in the aggregator of the switch.

15. The multi-task distributed training system according to claim 14, characterized in that: The switch module is also used for recirculating the aggregated data to read the aggregated data after determining that the gradient data of all training tasks in one iteration process are aggregated.

16. The multi-task distributed training system according to claim 14, characterized in that: The switch module is further configured to forward the retransmitted data packets to be aggregated to the parameter server module, so that the parameter server module sends the aggregated data to the switch module.

17. The multi-task distributed training system according to claim 16, characterized in that: The parameter server module is configured on any computing node in the network.

18. A multi-job distributed training method, applied to a network for distributed training of a deep neural network, wherein the network comprises a plurality of computing nodes for participating in the distributed training, characterized in that: The multi-job distributed training method comprises the following steps: Assigning an ID to each training task to locate the training data of each training task; wherein each training task is completed by a computing node; the training data is managed by a storage service module, and the storage service module is used to distribute the training data to each training task according to the ID of each training task and its iteration number; Acquire training data of the computing task to enable the computing node to perform gradient calculation to obtain gradient data and send it to the network; Aggregate the gradient data corresponding to each training task obtained from the network to obtain aggregate data and broadcast it to the network through the network; The computing node receives aggregated data from the network to perform training for the next iteration.

19. A storage server, applied to a network for distributed training of a deep neural network, wherein the network comprises a plurality of computing nodes for participating in the distributed training, characterized in that: The storage server comprises: A memory for storing training data for distributed training of a deep neural network; A processor, configured to manage the stored training data and distribute the training data to each training task according to the ID of each training task and its iteration number; Among them, the ID of each training task is used to locate the training data of each training task, and each training task is completed by a computing node; the computing node performs gradient calculation based on the distributed training data corresponding to a training task to obtain gradient data and sends it to the network and obtains aggregate data from the network; the aggregate data is obtained by performing aggregate calculation through a programmable switch.

20. A computer device, used in a network for distributed training of a deep neural network, wherein the network comprises a plurality of computing nodes for participating in the distributed training, characterized in that: The computer device comprises: Memory; A processor, configured to obtain distributed training data corresponding to a training task so as to enable the computing node to perform gradient calculation, and send the gradient data obtained by the computing node to the network; and receive aggregated data from the network and send it to each computing node; wherein each training task is completed by one computing node; Among them, the training data corresponding to a training task is distributed to each training task through a storage server according to the ID of each training task and its number of iterations; the ID of each training task is used to locate the training data of each training task; the aggregated data is obtained by aggregation calculation through a programmable switch.

21. A computer device, used in a network for distributed training of a deep neural network, wherein the network comprises a plurality of computing nodes for participating in the distributed training, characterized in that: The computer device comprises: Memory; A processor, configured to assign an ID to each training task to locate training data of each training task, wherein each training task is completed by one computing node; Among them, the training data corresponding to a training task is distributed to each training task through a storage server according to the ID of each training task and its number of iterations; the computing node performs gradient calculation based on the distributed training data corresponding to a training task to obtain gradient data and sends it to the network and obtains aggregate data from the network; the aggregate data is obtained by performing aggregate calculation through a programmable switch.

22. A programmable switch, applied to a network for distributed training of a deep neural network, wherein the network comprises a plurality of computing nodes for participating in the distributed training, characterized in that: The programmable switch comprises: Network ports; A processor, configured to perform aggregation calculation based on the gradient data corresponding to each training task obtained from the network port to obtain the aggregate data, and send the aggregate data to each computing node in the network through the network port; wherein each training task is completed by one computing node; Among them, the training data corresponding to a training task is distributed to each training task through a storage server according to the ID of each training task and its number of iterations; the ID of each training task is used to locate the training data of each training task; the gradient data is obtained by each computing node performing gradient calculation based on the allocated training data corresponding to a training task.

23. A computer device, characterized in that: include: A storage device for storing at least one program; A processing device is connected to the storage device, and is used to call the at least one program from the storage device and implement the multi-job distributed training method as described in claim 18 when executing it.

24. A computer-readable storage medium, characterized in that: At least one program is stored, and when the at least one program is called and executed by a processor of a computer, the multi-job distributed training method as described in claim 18 is implemented.

25. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is enabled to execute the multi-job distributed training method according to claim 18 .

Images