Computation result aggregation processing system and setting method

Abstract

Provided is a processing system that executes a process flow in which computation results are aggregated after data is divided and processed. This processing system comprises: a computer cluster 10 that is composed of a plurality of computation units which each process divided data; a DPU cluster 20 that is composed of at least one DPU which aggregates computation results; and a cross-connect switch 30 that connects devices. A setting device enumerates process flows which can be established by the processing system while changing the number l of interfaces of a computation unit to be used and the number m of DPUs to be used, combines the process flows such that data communication time in the processing system is minimized, and sets the cross-connect switch 30 such that a combination of the process flows can be executed.

Description

Calculation result aggregation processing system and setting method

[0001] The present disclosure relates to a processing system and configuration method.

[0002] In distributed training of machine learning models and distributed high-performance computing (HPC) applications in scientific computing, a collective communication technique called Allreduce is used to share data among multiple computing units.

[0003] Methods for realizing Allreduce collective communication can be broadly divided into methods (1) and (2) below. Method (1) consists of one-to-one communication between computational units and data aggregation operations at the computational units. Method (2) aims to speed up Allreduce collective communication by delegating data aggregation operations to data processing units (DPUs) within the network, thereby reducing the total amount of data flowing through the network and the number of communications. The technology of method (2) is called in-network computing or in-network aggregation. In method (2), computational units send data to DPUs, which aggregate the data, and only the necessary data is sent to the computational units. Examples of DPU devices that are used include network switches and dedicated servers (Non-Patent Document 1).

[0004] Jarek Kazmierczak and Nikita Namjoshi, "Optimize training performance with Reduction Server on Vertex AI," August 26, 2021, Google Cloud Blog,〈 URL: https: / / cloud.google.com / blog / topics / developers-practitioners / optimize-training-performance-reduction-server-vertex-ai?hl=en〉

[0005] In method (2), the amount of data that the computation unit sends and receives is at the theoretical lower limit, while in method (1), the amount of data that needs to be sent and received is approximately twice as much. Therefore, method (2) is theoretically superior to method (1) in terms of data transfer time.

[0006] However, if the aggregation bandwidth or number of interfaces of the DPU cluster is insufficient, method (1), which is inferior in terms of data transfer time, must be selected. For example, in method (2), if the data aggregation bandwidth of the DPU cluster is lower than the data transfer bandwidth, congestion can occur, significantly impairing performance. Furthermore, if all computing units cannot establish connections with the DPU cluster, the data aggregation protocol of conventional method (2) cannot be executed. In addition, even if all computing units can connect to the DPU cluster, if the number of interfaces in the DPU cluster is insufficient, the execution time of method (2) may be shorter than that of method (1). For example, if each computing unit has two interfaces and only one of them can connect to the DPU cluster, the communication bandwidth of the computing unit cannot be fully utilized.

[0007] The present disclosure has been made in view of the above, and aims to realize a data transfer time that is suited to the system configuration.

[0008] A processing system of one embodiment of the present disclosure is a processing system that executes a processing flow that divides and processes data and then aggregates the calculation results, and is equipped with a plurality of computing devices that each process the divided data, one or more data processing devices that aggregate the calculation results of the plurality of computing devices, and a cross-connect switch that connects the devices.The processing system lists processing flows that can be constructed in the processing system by changing the number of interfaces of the computing devices used and the number of data processing devices used, combines the processing flows so as to minimize the data communication time within the processing system, and configures the cross-connect switch so that the combination of processing flows can be executed.

[0009] According to the present disclosure, it is possible to achieve a data transfer time that is appropriate for the system configuration.

[0010] FIG. 1 is a diagram illustrating an example of the configuration of a processing system according to this embodiment. FIG. 2 is a flowchart illustrating an example of a process flow for constructing a process flow. FIG. 3 is a flowchart illustrating an example of a process flow for enumerating process flows. FIG. 4 is a diagram illustrating an example of a data flow for aggregating data in a DPU. FIG. 5 is a diagram illustrating an example of a process flow obtained from the data flow of FIG. 4. FIG. 6 is a flowchart illustrating an example of a process flow for searching for a combination of process flows. FIG. 7 is a diagram illustrating an example in which a wavelength division multiplexing transceiver is connected to a computation unit. FIG. 8 is a flowchart illustrating an example of a process flow for constructing a process flow. FIG. 9 is a diagram illustrating an example in which multiple GPUs are used as a single computation unit. FIG. 10 is a diagram illustrating an example of DPU processing. FIG. 11 is a diagram illustrating an example of DPU processing. FIG. 12 is a diagram illustrating an example of the hardware configuration of a setting device.

[0011] [Configuration of Processing System] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0012] FIG. 1 is a diagram illustrating an example of the configuration of a processing system according to this embodiment. The processing system illustrated in the figure includes a computer cluster 10 having multiple computing units, a DPU cluster 20 having one or more DPUs, and a cross-connect switch 30. In the processing system, data is divided and assigned to each of the computing units. Each of the computing units processes the assigned divided data in parallel to obtain partial computation results. The computation results of each computing unit are transmitted and received within the processing system using Allreduce collective communication and aggregated (e.g., addition or multiplication), and the final computation result is shared by all of the computing units.

[0013] The computer cluster 10 is configured with dedicated parallel arithmetic processors such as graphics processing units (GPUs) as computing units, or general-purpose central processing units (CPUs) as computing units.

[0014] The DPU cluster 20 is composed of devices capable of stream processing, such as a Field-Programmable Gate Array (FPGA), a Smart Network Interface Card (SmartNIC), or a network switch, and performs data aggregation operations.

[0015] The cross-connect switch 30 is configured using an Ethernet switch, an L1 switch, an optical circuit switch, etc. The cross-connect switch 30 dynamically sets paths between ports internally to connect ports, thereby connecting computation units, DPUs, and computation units and DPUs.

[0016] In the example of Figure 1, the computer cluster 10 is composed of four computing units, each with two input / output interfaces. Note that Figure 1 is a diagram that shows the flow of data, and computing units with the same name represent the same machine. The input / output interfaces of each computing unit and the input / output interface of the DPU are connected to the cross-connect switch 30. The dashed lines within the cross-connect switch 30 represent the flow of data between ports, with data flowing in the direction of the arrows.

[0017] In the example of Figure 1, a processing flow is constructed that combines the above-mentioned methods (1) and (2). The processing flow of method (1) is indicated by a thick arrow, and the processing flow of method (2) is indicated by a thin arrow. The processing flow is a process in which the calculation results of each calculation unit are transferred and aggregated, and the aggregated calculation results are shared. The processing flow of method (1) is a processing flow in which adjacent calculation units are directly connected without going through the DPU cluster 20, and the calculation results are aggregated in the calculation unit. The processing flow of method (2) is a processing flow in which the calculation results sent by each calculation unit are input to the DPU, the calculation results are aggregated in the DPU, and the aggregated calculation results are then sent to each calculation unit.

[0018] Table 1 below shows the performance of Ring-Allreduce (method (1)) and Allreduce with in-network computing (method (2)) when the number of computational units is N = 4. k is the order of the computational unit (the number of available interfaces), S is the data size of Allreduce, and β is the inverse of the link bandwidth.

[0019]

[0020] The Ring-Allreduce algorithm provides the shortest data transfer time among algorithms that only involve communication between computational units, and in-network computing technology can further reduce the data transfer time. However, if the number of DPUs is smaller than the order of the computational units, or if the order of the DPUs is smaller than the number of computational units, the Allreduce with in-network computing processing flow cannot be executed due to the lack of interfaces between the DPUs.

[0021] This processing system connects the computer cluster 10 and the DPU cluster 20 with a cross-connect switch 30, and by combining the processing flow of method (1) with the processing flow of method (2), it is possible to configure a processing flow that achieves a data transfer time that is intermediate between the data transfer time of method (1) and the data transfer time of method (2), even if the data aggregation operation bandwidth or number of interfaces of the DPU group is insufficient to execute method (2).

[0022] In the example in Figure 1, the number of computational units is N=4, the degree of the computational units is k=2, and the number of DPUs is M=1. When constructing only the Ring-Allreduce processing flow, if two computational unit interfaces are used, the data transfer time is (3 / 4)Sβ. The amount of data sent and received with Ring-Allreduce is calculated as (2(N-1) / N)S. When constructing only the Allreduce with in-network computing processing flow, only one computational unit interface can be used due to DPU limitations, so the transfer time is Sβ. With Allreduce with in-network computing, only half of the communication bandwidth of the computational units is used.

[0023] In the example shown in Figure 1, a Ring-Allreduce processing flow and an Allreduce with in-network computing processing flow are constructed, and data is divided into (2 / 5)S and (3 / 5)S, with Allreduce collective communication performed in each processing flow. As a result, the data transfer time for Ring-Allreduce is (3 / 2)*(2 / 5)Sβ, and the data transfer time for Allreduce with in-network computing is (3 / 5)Sβ. In other words, the data transfer time for the entire processing system is (3 / 5)Sβ, and this processing system can reduce the data transfer time compared to Ring-Allreduce alone and Allreduce with in-network computing alone.

[0024] [Processing Flow Construction Process] Next, an example of the processing flow for constructing a processing flow will be described with reference to the flowchart in Figure 2. A setting device (not shown) configures the processing system. Specifically, the setting device determines the processing flow based on the number of computing units N, the degree of the computing units L, the total number of DPUs M, and the data size number S, and then reconfigures the cross-connect switch 30, writes the processing flow to the computer cluster 10 and the DPU cluster 20, and sets up communication between devices.

[0025] In step S11, the setting device inputs the number of calculation units N, the total number of DPUs M, and the data size of Allreduce S. The order L of the calculation units is a fixed value that is set in advance.

[0026] In step S12, the setting device enumerates the processing flows of Allreduce while changing the number m of DPUs to be used and the order 1 of the calculation units to be used. Details of the enumeration of the processing flows will be described later.

[0027] In step S13, the setting device combines the process flows listed in step S12 so as to minimize the communication time within the processing system. Specifically, the setting device calculates the communication time for the process flow combination by adjusting the amount of data transferred by each process flow included in the process flow combination, and selects the process flow combination that minimizes the communication time. Details of the process flow combination search will be described later.

[0028] In step S14, the setting device reconfigures the cross-connect switch 30 so that the processing flow selected in step S13 can be executed.

[0029] In step S15, the setting device writes the processing flow to the computing units and DPU. For example, the setting device sets the computing units and DPU so that the data of the computing units is divided according to the proportion of the data amount to be transferred in each processing flow calculated in step S13 and processed in each processing flow. The setting device sets the computing units and DPU so that the Ring-Allreduce processing flow is executed for data communicated using Ring-Allreduce. The setting device sets and establishes connections between the computing units and DPUs using the cross-connect switch 30 so that the processing flow can be executed.

[0030] In step S16, the setting device sets up communication between the devices.

[0031] Through the above process, a processing flow is established in the processing system. The data to be processed is divided and assigned to each computation unit, and each computation unit processes the assigned divided data in parallel to obtain partial computation results. Each computation result is sent, received, and aggregated within the processing system according to the processing flow, and the final computation result is shared by all computation units.

[0032] [Enumerating Process Flows] Next, an example of processing for enumerating process flows will be described with reference to the flowchart of FIG.

[0033] In step S21, the setting device initializes the number of DPUs to be used m to 0, the order l of the calculation unit to 1, and the set F of processing flows to be empty.

[0034] In step S22, the setting device determines whether the number m of DPUs to be used is equal to or less than the total number M of DPUs. If m is greater than M, the process of listing the processing flow ends.

[0035] In step S23, the setting device searches for a processing flow that satisfies the conditions of the number of DPMs m to be used and the order l of the calculation unit to be used, and if a processing flow is found, the process proceeds to step S24, but if no processing is found, the process proceeds to step S25.

[0036] In the search for a processing flow, for example, the connection relationships between devices and the order of data flow between devices are formulated as a combinatorial optimization problem, and the problem is solved. If a solution cannot be found within the time limit, or if the search proves that there is no solution to the problem, the setting device is deemed to have failed to find a processing flow. An example of the search for a processing flow is described below.

[0037] When the number of DPUs used, m, is 0, the processing flow of Ring-Allreduce and the processing flow based on Tree topology are listed.

[0038] If the number of DPUs to be used, m, is greater than 0, one DPU is selected and a data flow that aggregates data to the selected DPU is searched for. If necessary, connections between computational units are also permitted. For example, in the example of Figure 4, DPU3 is selected and a data flow that aggregates data to DPU3 is searched for. Note that the cross-connect switch 30 is omitted in Figure 4.

[0039] If a data flow that aggregates data to the selected DPU can be found, the data flow is inverted and a data flow that broadcasts data from the selected DPU is constructed, thereby obtaining the Allreduce processing flow. Figure 5 shows the processing flow obtained by inverting the data flow in Figure 4.

[0040] If a processing flow is found, in step S24, the setting device adds a pair (m, l) of the number of DPUs m used and the order l of the computing units used to the set F of processing flows. If there are multiple processing flows that can be realized with (m, l), only the processing flow with the smallest transfer time T is considered. Therefore, the setting device may store information about the processing flow with the smallest transfer time T among the processing flows found in the search in a key-value store with (m, l) as the key and the processing flow as the value.

[0041] After searching for the processing flow for the number m of DPUs to be used and the order l of the calculation unit to be used, in step S25, the setting device determines whether the order l of the calculation unit to be used is the order L of the calculation unit.

[0042] If the order l of the calculation unit to be used is the order L of the calculation unit, in step S26, the setting device increments the number m of DPUs to be used, sets the order l of the calculation unit to be used to 1, and proceeds to step S22.

[0043] If the degree l of the calculation unit to be used is not the degree L of the calculation unit, in step S27, the setting device increments the degree l of the calculation unit to be used, and proceeds to step S23 to search for a processing flow with new conditions.

[0044] [Searching for a Combination of Process Flows] Next, an example of a process for searching for a combination of process flows will be described with reference to the flowchart of FIG.

[0045] In step S31, the setting device selects several processing flows from the set F of processing flows and calculates the data volume that minimizes the communication time for the selected combination of processing flows. At this time, the total number of DPUs m in the combination of processing flows must be less than or equal to the total number of DPUs M, the total number of degrees l of the calculation units must be less than or equal to the order L of the calculation units, and the total amount of data s(m, l) flowing in each processing flow must be the data size S.

[0046] This will be explained in detail using the processing system shown in Figure 1. The processing system in Figure 1 has the number of computational units N=4, the total number of DPUs M=1, and the degree of the computational units L=2. In the process of enumerating the processing flows, three processing flows are obtained: the number of DPUs m=0 and the degree of the computational units l=1, the number of DPUs m=0 and the degree of the computational units l=2, and the number of DPUs m=1 and the degree of the computational units l=1. In other words, the set F={(0,1), (0,2), (1,1)} is obtained. Note that the processing flow Allreduce with in-network computing cannot be executed for the case where the number of DPUs m=1 and the degree of the computational units l=2, i.e., (1,2), because the number of DPUs m is smaller than the degree of the computational units l.

[0047] The combinations for selecting one or more processing flows (m,l) from the set F are (0,1), (0,2), (1,1), (0,1) and (0,1), (0,1) and (0,2), (0,1) and (1,1), (0,2) and (0,2), (0,2) and (1,1), (1,1) and (1,1), and (0,1) and (0,2) and (1,1). However, there are five combinations of processing flows that satisfy the conditions for the total number of DPUs m and the total degree of the computational units l: (0,1), (0,2), (1,1), (0,1) and (0,1), and (0,1) and (1,1).

[0048] Of the above combinations, (0,1), (0,2), and (1,1) have one processing flow, so the data volume s(m,l) = S. Also, (0,1) and (0,1) are equivalent to the processing flow of (0,2).

[0049] The setting device determines the data volume s(m,l) for the processing flow of (0,1) and the processing flow of (1,1) for the combination of (0,1) and (1,1) so that the communication time is minimized. The processing flow of (0,1) is Ring-Allreduce. From Table 1, when the number of computing units N = 4, the communication time is T(s(0,1); 0,1) = (3 / 2)s(0,1)β. The processing flow of (1,1) is Allreduce with in-network computing. From Table 1, the communication time is T(s(1,1); 1,1) = s(1,1)β. The sum of s(0,1) and s(1,1) must be the data size S. If data is sent to each processing flow so that s(0,1)=(2 / 5)S and s(1,1)=(3 / 5)S, the communication time T(s(0,1); 0,1)=(3 / 2)*(2 / 5)Sβ=(3 / 5)Sβ and the communication time T(s(1,1); 1,1)=(3 / 5)Sβ, which is the shortest communication time. Incidentally, if data is sent so that s(0,1)=(1 / 2)S and s(1,1)=(1 / 2)S, the communication time T(s(0,1); 0,1)=(3 / 2)*(1 / 2)Sβ=(3 / 4)Sβ and the communication time T(s(1,1); 1,1)=(1 / 2)Sβ, which is the longer communication time of the entire processing system, (3 / 4)Sβ, which is slower than (3 / 5)Sβ. The setting device determines the distribution of data amounts that minimizes the overall communication time for each combination of process flows while changing the amount of data sent to each process flow.

[0050] In step S32, the setting device selects the combination of process flows selected in step S31 that minimizes the communication time for the determined data volume.

[0051] In the example of the processing system in Figure 1, four combinations of (0,1), (0,2), (1,1), (0,1) and (1,1) are extracted in step S31, and the respective communication times are (4 / 3)Sβ, (2 / 3)Sβ, Sβ, and (3 / 5)Sβ. Therefore, the setting device obtains the combination of (0,1) and (1,1) with the shortest communication time as the processing flow to be constructed in the processing system.

[0052] [Modifications] Next, several modifications of the processing system will be described.

[0053] As shown in Figure 7, a wavelength division multiplexing (WDM) transceiver is connected to a computing unit (which may be a DPU), and a combined signal is separated into different wavelength signals by a demultiplexer (Demux), and the different wavelength signals are combined into a combined signal by a multiplexer (Mux). This allows the number of virtual interfaces of the computing unit or DPU to be increased without changing the number of physical interfaces. However, since multiple signals of the same wavelength cannot be input to the Mux of the same transceiver, it is determined whether wavelengths can be assigned to each of the links of the processing flow combination obtained in step S13.

[0054] An example of a process flow for constructing a process flow for wavelength division multiplexing communication will be described with reference to the flowchart of FIG.

[0055] In step S11, the setting device inputs the number of calculation units N, the total number of DPUs M, and the data size S of Allreduce. The order L of the calculation units and the wavelength multiplexing degree W are assumed to be fixed values ​​and set in advance. The order L of the calculation units is the number of virtual interfaces.

[0056] In step S12, the setting device lists the processing flow of Allreduce while changing the number m of DPUs to be used and the order l of the calculation unit to be used.

[0057] In step S13, the setting device combines the process flows listed in step S12 so as to minimize the communication time.

[0058] In step S17, the setting device determines whether or not W wavelengths can be assigned to each of the links of the combination of processing flows obtained in step S13.

[0059] If wavelength allocation is not possible, the setting device removes the process flow combination from the candidates, returns to step S13, and obtains another process flow combination.

[0060] If the wavelength can be allocated, in step S14 the setting device reconfigures the cross-connect switch 30 so that the processing flow selected in step S13 can be executed.

[0061] In step S15, the setting device writes the processing flow to the calculation unit and the DPU.

[0062] In step S16, the setting device sets up communication between the devices.

[0063] Next, an embodiment in which two-stage Allreduce is performed will be described with reference to FIG.

[0064] As shown in Fig. 9, accelerators such as GPUs 51 used as computing units are generally installed in multiple locations within the same server (node) 50. Within the server 50, the GPUs 51 are connected via an inter-GPU communication bus 52 such as NVLINK, which has a higher bandwidth. Each GPU 51 also has a network interface 53 such as a host channel adapter (HCA).

[0065] At the beginning of Allreduce execution, Allreduce is executed among the GPUs 51 in the server 50, and local aggregate values ​​are shared among all the GPUs 51. This allows the server 50 to be regarded as a computation unit with four interfaces. After that, Allreduce is executed across the entire processing system.

[0066] Next, an example of the processing of the DPU will be described with reference to FIGS.

[0067] 10 and 11 are diagrams showing an example of processing performed by a DPU. The upper port is the input port, and the lower port is the output port. The Reduction unit receives data from the input port and aggregates it. The Broadcast unit duplicates the data aggregated by the Reduction unit and outputs it from the output port.

[0068] The DPU in Fig. 10 aggregates data received from four input ports, replicates the aggregated data, and outputs it from four output ports. This corresponds to the processing of the DPU in Fig. 1.

[0069] The DPU in Figure 11 performs the following processes: aggregating data received from three input ports and outputting the aggregated data from one output port; and outputting data received from one input port from three output ports. This corresponds to the processes of DPU1 and DPU2 in Figure 5. For example, DPU1 in Figure 5 receives data from computation units 1 to 3, aggregates it, and outputs the aggregated data to DPU3, and outputs data received from DPU3 to computation units 1 to 3.

[0070] As described above, the processing system of this embodiment includes a computer cluster 10 consisting of multiple computing units that each process divided data, a DPU cluster 20 consisting of one or more DPUs that aggregate the computation results, and a cross-connect switch 30 that connects the devices. The configuration device lists processing flows that can be constructed in the processing system while changing the number of interfaces l of the computing units used and the number m of DPUs used, combines the processing flows to minimize the data communication time within the processing system, and configures the cross-connect switch 30 so that the combination of processing flows can be executed. This allows for data transfer times appropriate to the configuration of the processing system to be achieved, even if the data aggregation bandwidth or number of interfaces of the DPU cluster 20 is insufficient to execute AllReduce collective communication using conventional in-network computing.

[0071] The setting device described above can be, for example, a general-purpose computer system including a CPU 901, memory 902, storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 12. In this computer system, the setting device is realized by the CPU 901 executing a predetermined program loaded onto the memory 902. This program can be recorded on a computer-readable non-transitory recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be distributed via a network.

[0072] 10 Computer cluster 20 DPU cluster 30 Cross-connect switch

Claims

1. A processing system that executes a processing flow that divides and processes data and then aggregates the calculation results, comprising: a plurality of computing devices that each process the divided data; one or more data processing devices that aggregate the calculation results of the plurality of computing devices; and a cross-connect switch that connects the devices; the processing system lists processing flows that can be constructed in the processing system while changing the number of interfaces of the computing devices used and the number of data processing devices used; combines the processing flows so as to minimize the data communication time within the processing system; and configures the cross-connect switch so that the combination of processing flows can be executed.

2. A method for configuring a processing system that executes a processing flow that divides and processes data and then aggregates the calculation results, the processing system comprising a plurality of computing devices that each process the divided data, one or more data processing devices that aggregate the calculation results of the plurality of computing devices, and a cross-connect switch that connects the devices, the method comprising: a computer enumerating processing flows that can be constructed in the processing system while changing the number of interfaces of the computing devices used and the number of data processing devices used, combining the processing flows so as to minimize the data communication time within the processing system, and configuring the cross-connect switch so that the combination of processing flows can be executed.

3. A setting method according to claim 2, wherein the combination of processing flows includes a processing flow that connects the computing devices, transfers calculation results between the computing devices, and aggregates the calculation results in the computing devices, and a processing flow that connects the computing devices and the data processing devices, transfers calculation results to the data processing devices, and aggregates the calculation results in the data processing devices.

4. A setting method according to claim 2 or 3, comprising: calculating the communication time for the combination of processing flows by adjusting the amount of data transferred in each of the processing flows included in the combination of processing flows; and selecting the combination of processing flows that minimizes the communication time.

Images