Multi-plane network set communication method

Abstract

The invention relates to the field of computers, in particular to a multi-plane network set communication method, which comprises the following steps of: S101, dividing initial data needing to be subjected to set communication operation into different network planes; each processor is divided to obtain data needing to be subjected to set communication operation; s102, operating data blocks needing set communication operation, and performing in-node set communication operation on each processor on each node; s103, performing corresponding set communication operation among the processors of each network plane, and attributing calculation result data to a target node; and S104, according to different operation types of the set communication, finishing convergence in the target node. The method has the advantages that in the whole process, the parameter plane and the data plane participate in parameter transmission of set communication at the same time, the data size of parameter transmission of a single network plane is reduced, bandwidth can be utilized to the maximum extent, and the transmission efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of computers, and more particularly to a multi-plane network aggregate communication method. Background Technology

[0002] With the development of artificial intelligence technology, deep learning has been successfully applied in various fields such as computer vision, pattern recognition, and natural language processing. When training deep learning models, graphics processing units (GPUs) are generally used for acceleration. However, when training more complex models and larger datasets, single-GPU training is time-consuming, necessitating the use of multiple GPUs for parallel training. Modern training servers typically utilize multiple GPUs to enhance computing power. The GPUs on each node are interconnected via corresponding network interface cards (NICs) and switches, forming a specific network topology. Since each GPU requires a dedicated NIC, network setup costs are high. Therefore, a multi-plane network is employed, where each NIC corresponds to a separate network plane, and these planes are not interconnected, significantly reducing network costs. The GPUs within the server are interconnected via NVLink, utilizing NVLink's high bandwidth for data forwarding, enabling communication between the multi-plane networks.

[0003] One key consideration when employing multi-GPU parallel training is the communication performance between GPUs. In data-parallel mode, GPUs need to perform ensemble communication operations, which places high demands on GPU communication. Current technologies using NCCL (Non-Concurrent Communication Array) do not specifically adapt or improve for multi-plane networks, resulting in low bandwidth utilization, especially with small data volumes. In multi-plane network applications, the parameter plane formed by GPU interconnects is typically used for ensemble computation, while the data plane formed by CPU interconnects is used for data storage. However, with the increasing bandwidth of the data plane, improving ensemble communication in multi-GPU parallel training, providing an efficient ensemble communication method between multi-plane networks, better utilizing data plane bandwidth to improve overall bandwidth utilization, enhance network performance, reduce congestion, and make communication between multi-plane networks more efficient are problems that need to be solved. Summary of the Invention

[0004] To overcome the above-mentioned shortcomings, the present invention aims to provide a multi-plane network ensemble communication method for ensemble communication operations in parallel training.

[0005] This invention achieves the above objective through the following scheme: a multi-plane network ensemble communication method for ensemble communication operations in parallel model training, comprising the following steps:

[0006] S101: Divide the initial data that needs to be used for aggregate communication operations into different network planes; each processor separately divides the data that needs to be used for aggregate communication operations.

[0007] S102: Divide the data that requires aggregate communication operations into blocks and perform intra-node aggregate communication operations on each processor on each node;

[0008] S103: Perform corresponding aggregate communication operations between processors in each network plane, and the computation result data is attributed to the target node;

[0009] S104: Depending on the type of aggregated communication operation, complete the aggregation within the target node.

[0010] Preferably, steps S102-S104 divide the data to be operated in each step into blocks and perform pipeline-style operations, that is, divide the data to be operated into small data blocks. For each block, the above steps are executed in the order of S102-S104, and from the overall perspective, they are performed simultaneously.

[0011] Preferably, in step S101, the set communication operation includes: reduction, full reduction, broadcast, and splitting.

[0012] Preferably, the multi-plane network comprises several nodes, which perform collective communication operations with each other, wherein each node is equipped with a processor, including a CPU and a GPU.

[0013] Preferably, the multi-plane network includes several parameter planes. A network plane composed of several GPUs is called a parameter plane. In step S101, the network plane is the parameter plane by default.

[0014] The multi-plane network also includes a data plane. A network plane composed of several CPUs is called a data plane. In step S101, the data plane can also be selected to participate in the aggregated communication operation.

[0015] Preferably, in step S101, the initial data partitioning is based on the bandwidth ratio of the network plane, where the bandwidth ratio of the network plane is the bandwidth of the network plane / the total bandwidth of the multi-plane network.

[0016] Preferably, GPUs or CPUs within the same network plane are interconnected via network interface cards (NICs) and access data using RDMA communication via NICs.

[0017] Preferably, data is transmitted between GPUs or CPUs in different network planes via NVlink.

[0018] An apparatus includes a network interface card (NIC), a processor, and a memory; wherein the memory stores a computer program executable on the processor; and the processor, when executing the computer program, is able to work with the NIC to perform the methods for transmitting data and / or receiving data as described in any of the above embodiments.

[0019] The beneficial effects of this invention are as follows: the data that needs to be aggregated for communication operations is divided into different network planes, aggregated for communication operations are first performed inside the processor, then between network planes, and finally converged on the target node processor. The whole process reduces the amount of data transmitted by a single network plane by allowing parameter transmission in both the parameter plane and the data plane to participate in aggregated communication simultaneously, thereby maximizing bandwidth utilization and improving transmission efficiency. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the steps of the method of the present invention;

[0021] Figure 2 This is a schematic diagram of the connections between nodes according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the initial data in an embodiment of the present invention. Detailed Implementation

[0023] The data set communication synchronization method provided in this application embodiment can be applied to scenarios where computing clusters execute distributed computing tasks, including but not limited to scenarios where distributed training tasks are executed in the field of artificial intelligence deep learning, scenarios where physical experimental data is analyzed, scenarios where meteorological data is analyzed, or other parallel computing scenarios.

[0024] The present invention will be further described below with reference to specific implementation examples, but the scope of protection of the present invention is not limited thereto:

[0025] Example: Figure 1As shown, a multi-plane network ensemble communication method is applied to a scenario of deep learning model training in a multi-plane network topology formed by interconnected multiple nodes. The computer cluster involved in this embodiment includes multiple nodes, where each node refers to a computer server. These computing nodes communicate through a configured network system to jointly complete the model training task. Deep learning primarily involves vectorized matrix operations. As a graphics accelerator, the GPU provides a large number of computing cores for rendering; therefore, GPUs are used for model training. Each node computer server includes multiple processors, including GPUs and CPUs. Each node also has multiple network interface cards (NICs) connected to each GPU and CPU, respectively. These NICs support RDMA communication. In the entire computing cluster, the GPUs and CPUs of the node computer servers are connected to each other via NICs. The NICs on different nodes can form independent network planes through the network topology, called a multi-plane network. Multi-plane networks allow data to be transmitted on different network paths, thereby improving network bandwidth, reliability, and flexibility.

[0026] Different network interface cards (NICs) on the same node can reside in the same or different network planes, depending on requirements. Correspondingly, GPUs can also reside in the same or different network planes. The specific connection method between NICs depends on the on-site multi-plane network configuration and is not restricted here. For a multi-plane network structure, NICs not belonging to the same plane cannot directly access each other. For the same node, GPUs located in different network planes on the same node can synchronize data through NVLink. For example, a GPU write kernel can be used to directly write data to the GPU's video memory, or a copy engine can be used to copy data to the GPU's video memory. Communication between the GPU and CPU can be achieved through NVLink, PCIe, or other methods.

[0027] A multi-plane network consists of several parameter planes and several data planes. Multiple network planes, composed of multiple GPUs connected through network cards, are called parameter planes, also known as computation planes. These planes are typically used for calculating, updating, transferring, synchronizing gradients, or other computational results of model parameters.

[0028] The network plane interconnecting CPUs is called the data plane. It typically handles data storage, processing, and transmission, such as storing, processing, and transmitting training, validation, and test datasets, but does not transmit model parameters. The ensemble communication operations can be of the types of reduction, all-reduce, broadcast, and scatter. Generally, ensemble communication operations occur in the parameter plane.

[0029] In the fields of deep learning and machine learning, parallel and distributed computing are often used to improve computational efficiency during model training, which requires the use of ensemble communication. This allows a group of processors (or nodes) to collaborate over a network to complete a task. In this mode, processors need to exchange data to ensure that all processors can synchronize and execute computations correctly.

[0030] In parallel computing scenarios, the distributed computing tasks executed by the entire computing system can be divided into several subtasks. Each node and each GPU in the computing system is used to execute one or more subtasks, completing a portion of the computation. After each node executes its assigned subtask, it obtains the result of that subtask, which needs to be aggregated from all nodes in the computing system. In this scenario, by executing the aggregated communication method provided in this application embodiment, the results obtained by each node can be efficiently synchronized to other GPUs on other nodes in the computing system, thereby efficiently aggregating the results of the entire computing system's task execution.

[0031] Figure 1 This is a physical diagram of the first embodiment of the present invention. This embodiment provides a method for ensemble communication in a multi-plane network for ensemble communication operations in parallel model training. It includes several nodes, each equipped with a processor, including a GPU and a CPU. Several GPUs form several parameter planes in the multi-plane network, and several CPUs form several data planes in the multi-plane network.

[0032] The specific methods and steps include:

[0033] S101: Divide the initial data that needs to be used for group communication operations into different network planes; each processor separately divides the data that needs to be used for group communication operations.

[0034] The network plane is the parameter plane by default, but the data plane can also be selected. The initial data on each node needs to be partitioned across different network planes. This partitioning is a logical calculation of network affiliation, not a physical transfer. The partitioning ratio is based on the bandwidth ratio of the respective network plane. The network plane bandwidth ratio is the bandwidth of the current network plane divided by the total bandwidth of the multi-plane network. Assume each node has N GPUs and M CPUs. The N GPUs form the N parameter planes of the multi-plane network, each with a bandwidth of J. The CPUs form the M data planes of the multi-plane network, each with a bandwidth of K. The total bandwidth of the multi-plane network is N*J + M*K. Therefore, the initial data allocation for each GPU on that node is J / (N*J + M*K), and the initial data allocation for each CPU on that node is K / (N*J + M*K). After the initial data partitioning, each GPU and CPU equally distributes the data required for aggregate communication operations.

[0035] S102: Divide the data that requires set communication operations into blocks and perform set communication operations on each processor on each node.

[0036] Taking reduction as an example, each GPU on a single node uses NVLink and the ring reduce algorithm (or other reduce algorithms) to perform a reduction operation on the data segments it has partitioned. For data segments belonging to the CPU data plane, the GPU performs a device-to-host (D2H) operation to send them to the CPU (for machines equipped with CPU-GPU NVLink, NVLink is used to send data; other machines perform D2H operations based on traditional PCIe). Subsequently, the CPU performs the most basic reduce operation on the received data to obtain the reduced result.

[0037] S103: Perform corresponding aggregate communication operations between processors in each network plane, and the computation result data is attributed to the target node.

[0038] Within the computing cluster, GPUs on different nodes of each parameter plane perform ensemble operations. Each parameter plane GPU completes an ensemble communication operation on the total data volume J / (N*J+M*K) and reduces the result to the GPU of the target node. The data volume transferred by the parameter plane is the total J / (N*J+M*K), and each GPU of the target node stores the total data volume J / (N*J+M*K). Simultaneously, CPUs on different nodes of each data plane perform ensemble communication operations. Each data plane completes a reduction calculation on the total data volume K / (N*J+M*K) and reduces the result to the CPU of the target node. The data volume transferred by the data plane is the total K / (N*J+M*K), and each CPU of the target node stores the total data volume K / (N*J+M*K). In a multiplane network, different nodes, such as GPUs and CPUs, correspond to their own planes. The GPUs (or CPUs) within a plane are interconnected through network cards. Their communication method uses RDMA (Remote Data Access) to access each other. This technology supports direct memory copying of data between the two systems over the network, enabling high-throughput and low-latency network communication.

[0039] In one specific embodiment, for the aggregation communication algorithm within the same plane, such as the reduce algorithm in this case, any reduce algorithm can be used, such as Rabenseifner's algorithm, Ring Reduce, Binary Tree algorithm, etc. This invention primarily aims at accelerating multi-plane networks, and the communication algorithm within the same plane is not limited. For example, Recursive Doubling, Bruck, Binary Tree, and Double Binary Tree algorithms used for allreduce can all be used here for cross-node aggregation communication within the plane.

[0040] S104: Depending on the type of aggregated communication operation, complete the aggregation within the target node.

[0041] Different aggregation communication operations present data differently within the target node. For example, broadcast, reduce, and allreduce operations require data aggregation within the node, while this step is unnecessary for splitting because the data has already been distributed in previous steps. For reduction, one node is the target node of the reduction algorithm, and this target node has a GPU or CPU as the final reduction target. This requires finally aggregating the data from each processor on the target node to the target processor.

[0042] For scatter, all processors on the target node have the result data and there is no need for data aggregation within the node.

[0043] The aggregation operation is performed on the GPUs within the target node. The CPU and GPU of the current target node have already obtained part of the results of the first step of partitioning. Now they need to be aggregated to the target GPU via NVLINK or PCIe.

[0044] For the same node, GPUs located on different network planes can synchronize data using NVlink. For example, the GPU write kernel can be used to write data directly to the GPU's video memory, or the copyengine can be used to copy data to the GPU's video memory. The GPU and CPU can communicate via NVlink, PCIe, or other methods.

[0045] It is important to note that S102, S103, and S104 are all pipelined operations performed through block division. Block division involves dividing the data to be processed into smaller blocks, and then performing S102, S103, and S104 operations on these blocks. Each step can be pipelined through block division; that is, the above steps are performed sequentially for each block, but overall, they occur simultaneously.

[0046] The types of operations in the set communication can be reduce, all-reduce, broadcast, or scatter.

[0047] The target node is the node to which the result is attributed after the data completes the set communication operation.

[0048] In a specific embodiment, a reduction calculation is performed using two nodes as an example, such as... Figure 2 As shown, the network includes nodes 1 and 2. Node 1 has two CPUs and eight GPUs, namely CPU1-1, CPU1-2, GPU1-1, GPU1-2, GPU1-3, GPU1-4, GPU1-5, GPU1-6, GPU1-7, and GPU1-8. Node 2 has two CPUs and eight GPUs, namely CPU2-1, CPU2-2, GPU2-1, GPU2-2, GPU2-3, GPU2-4, GPU2-5, GPU2-6, GPU2-7, and GPU2-8. It is important to note that the above quantities and naming are for ease of understanding and to distinguish different entities, and are not intended to limit the number or function of entities in this invention. CPU1-1 and CPU2-1 form parameter plane 1 in the multi-plane network, CPU1-2 and CPU2-2 form parameter plane 2 in the multi-plane network, GPU1-1 and GPU2-1 form data plane 1 in the multi-plane network, and the remaining data planes 2 through 8 follow the same pattern. Node 2 is the target node, and GPU1 in node 2 is the target GPU. Assuming the parameter plane bandwidth is 400Gbps and the data plane bandwidth is 200Gbps, after the initial data is divided, each GPU can get 1 / 9 of the initial data on that node, and each data plane can get 1 / 18 of the initial data.

[0049] like Figure 3As shown, ABCD, as the initial data for the aggregated communication operation in this embodiment, belong to Node 1 and Node 2 respectively. According to the aforementioned network plane bandwidth ratio, data segment a1 is 1 / 18 of data A, data segment b1 is 1 / 18 of data B, and a1 and b1 are assigned to CPU1-1 of Node 1; data segment a2 is 1 / 18 of data A, data segment b2 is 1 / 18 of data B, and a2 and b2 are assigned to CPU1-2 of Node 1; data segment a3 is 1 / 9 of data A, data segment b3 is 1 / 9 of data B, and a3 and b3 are assigned to GPU1-1 of Node 1; data segment a4 is 1 / 9 of data A, data segment b4 is 1 / 9 of data B, and a4 and b4 are assigned to GPU1-2 of Node 1, and so on. Data C and D are also assigned to the corresponding processors of Node 2 according to the aforementioned network plane bandwidth ratio.

[0050] Then, for node 1, CPU1-1 performs reduction calculations on data segments a1 and b1, obtaining a1+b1 and storing the reduction result on CPU1-1. CPU1-2 performs reduction calculations on data segments a2 and b2, obtaining a2+b2 and storing the reduction result on CPU1-2. GPU1-1 performs reduction calculations on data segments a3 and b3, obtaining a3+b3 and storing the reduction result on GPU1-1. The remaining data segments on A and B are processed in the same way. Similarly, reduction calculations are performed on node 2. CPU2-1 performs reduction calculations on data segments c1 and d1, obtaining c1+d1 and storing the reduction result on CPU2-1. GPU2-1 performs reduction calculations on data segments c3 and d3, obtaining c3+d3 and storing the reduction result on GPU2-1. The remaining data segments on C and D are processed in the same way. In this way, each GPU on each node performs a reduction calculation of 1 / 9 of the node's data size, and each CPU performs a reduction calculation of 1 / 18 of the node's data size. This method can effectively reduce the amount of computation per GPU and CPU, and achieve parallel computing to effectively improve computing efficiency.

[0051] On data plane 1, the reduction calculations for a1+b1 and c1+d1 are performed, and the reduction result a1+b1+c1+d1 is stored on CPU2-1 of node 2. On data plane 2, the reduction calculations for a2+b2 and c2+d2 are performed, and the reduction result a2+b2+c2+d2 is stored on CPU2-2 of node 2. On parameter plane 1, the reduction calculations for a3+b3 and c3+d3 are performed, and the reduction result a3+b3+c3+d3 is stored on GPU2-1 of node 2. The remaining parameter planes follow the same pattern. Each data plane CPU completes 1 / 18 of the overall data reduction calculations, and the data transfer volume is 1 / 18 of the total data volume. The amount of data on the target node CPU is 1 / 18 of the total data volume. Each parameter plane completes 1 / 9 of the overall data reduction calculations, and the data transfer volume is 1 / 9 of the total data volume. The amount of data on the target node GPU2-1 is 1 / 9 of the total data volume.

[0052] This method can effectively distribute the reduction computation between nodes across various network planes, reducing network congestion and improving computational efficiency.

[0053] Finally, the data between the GPU and CPU on node 2 is reduced to obtain the reduction result a1+b1+c1+d1+a2+b2+c2+d4+……+a9+b9+c9+d9+a10+b10+c10+d10, which is stored on GPU2-1 of node 2. Obtaining the reduction results for all data completes the reduction calculation.

[0054] The data set communication synchronization method provided in this application embodiment can be applied to scenarios where computing clusters execute distributed computing tasks, including but not limited to scenarios where distributed training tasks are executed in the field of artificial intelligence deep learning, scenarios where physical experimental data is analyzed, scenarios where meteorological data is analyzed, or other parallel computing scenarios.

[0055] Some preferred embodiments of the present invention provide an apparatus comprising: a network interface card (NIC), a processor, and a memory; wherein the memory stores a computer program executable on the processor; and when the processor executes the computer program, it is able to work with the NIC to execute the data transmission method and / or data reception method described in any of the above embodiments.

[0056] Some preferred embodiments of the present invention provide a computer network system comprising: a plurality of (not less than two) devices described in any of the above embodiments; wherein these devices are interconnected via a network; wherein these devices are used either for transmitting data (i.e., as source host ends), or for receiving data (i.e., as destination host ends), or for both transmitting data and receiving data from other devices.

[0057] Some preferred embodiments of the present invention provide a computer-readable storage medium storing code for performing tag data transmission, including: instructions for performing the data transmission method operation described in any of the above embodiments, and / or, instructions for the data reception method operation described in any of the above embodiments, or instructions for the collective communication data transmission method operation described in any of the above embodiments.

[0058] Furthermore, it should be emphasized that the memory mentioned in the above embodiments includes, but is not limited to, memory in the narrow sense (i.e., the memory provided by the memory card in the computer system), as well as the memory of the graphics card (usually referred to as video memory), etc.

[0059] The above description describes specific embodiments of the present invention and the technical principles employed. Any changes made in accordance with the concept of the present invention that do not exceed the spirit of the specification and drawings should still fall within the protection scope of the present invention.

Claims

1. A multi-plane network ensemble communication method for ensemble communication operations in parallel model training, characterized in that... Includes the following steps: S101: Divide the initial data that needs to be used for aggregate communication operations into different network planes; each processor separately divides the data that needs to be used for aggregate communication operations. S102: Divide the data that requires aggregate communication operations into blocks and perform intra-node aggregate communication operations on each processor on each node; S103: Perform corresponding aggregate communication operations between processors in each network plane, and the computation result data is attributed to the target node; S104: Depending on the type of aggregated communication operation, complete the aggregation within the target node.

2. The multi-plane network aggregated communication method according to claim 1, characterized in that, The steps S102-S104 involve dividing the data to be operated in each step into blocks and performing pipeline-style operations. That is, the data to be operated is divided into small data blocks. For each block, the above steps are executed in the order of S102-S104, and from the overall perspective, they are performed simultaneously.

3. A multi-plane network aggregated communication method according to claims 1-2, characterized in that, In step S101, the set communication operations include: reduction, full reduction, broadcast, and splitting.

4. A multi-plane network aggregated communication method according to claims 1-3, characterized in that, The multiplane network comprises several nodes, which perform collective communication operations. Each node is equipped with a processor, which includes a CPU and a GPU.

5. A multi-plane network aggregated communication method according to claims 1-4, characterized in that, The multi-plane network includes several parameter planes. A network plane composed of several GPUs is called a parameter plane. In step S101, the network plane is the parameter plane by default.

6. A multi-plane network aggregated communication method according to claims 1-5, characterized in that, The multi-plane network also includes a data plane. A network plane composed of several CPUs is called a data plane. In step S101, the data plane can also be selected to participate in the aggregated communication operation.

7. A multi-plane network aggregated communication method according to claims 1-6, characterized in that, In step S101, the initial data partitioning is based on the bandwidth ratio of the network plane, where the bandwidth ratio of the network plane is the bandwidth of the network plane / the total bandwidth of the multi-plane network.

8. The multi-plane network aggregated communication method according to claim 1, characterized in that, GPUs or CPUs within the same network plane are interconnected via network interface cards (NICs) and access data using RDMA communication.

9. The multi-plane network aggregated communication method according to claim 1, characterized in that, Data is transmitted between GPUs or CPUs in different network planes via NVlink.

10. An apparatus, characterized in that... It includes a network interface card (NIC), a processor, and a memory; wherein the memory stores a computer program that can run on the processor; and when the processor executes the computer program, it can work with the NIC to execute the data transmission method and / or data reception method described in any of the above embodiments.

Images