An efficient parameter synchronization method based on the AllReduce architecture

Abstract

This invention discloses a high-efficiency parameter synchronization method based on the AllReduce architecture. The method first groups and decomposes nodes according to the underlying physical link bandwidth, matching the decomposed link bandwidth with the communication volume. Then, it divides the data to be communicated into blocks based on the communication volume of different communication stages after decomposition. Finally, each data block is synchronized in parallel on different types of links using a grouped decomposition AllReduce approach. This invention is used for efficient parameter synchronization in complex network environments, overcoming the impact of bandwidth heterogeneity. It can achieve efficient parallelism in multi-NIC scenarios and utilize idle links to improve parallelism in single-NIC scenarios, achieving excellent performance in both homogeneous and heterogeneous network environments.

Description

Technical Field

[0001] This invention belongs to the field of computer distributed communication, and in particular relates to an efficient parameter synchronization method based on the AllReduce architecture. Background Technology

[0002] With the development of deep learning and large-scale model technology, the scale of neural network models used in practice is becoming increasingly large. In most scenarios, the memory of a single node is insufficient to accommodate the complete neural network model, and the computing power of a single node cannot complete the training or inference of the model within a reasonable time. Therefore, various distributed parallel training techniques have been developed to achieve efficient training and inference of large-scale neural network models. The basic parallel modes in distributed parallel training techniques include data parallelism and model parallelism. However, regardless of the parallel method, frequent communication between nodes is required to synchronize parameters. For data parallelism, the parameters to be synchronized are gradients or model parameters, while for model parallelism, the parameters to be synchronized are activation values ​​of intermediate layers of the model. Frequent inter-node communication limits the performance of model training, and thus limits the scalability of distributed training systems. Therefore, designing efficient inter-node parameter synchronization methods is crucial. To solve this problem, the AllReduce algorithm from the MPI ensemble communication library has been introduced into this field. AllReduce is a class of algorithms that achieve efficient parameter synchronization through data block partitioning and ensemble communication scheduling. Some classic algorithms, such as Ring AllReduce, have been integrated into popular deep learning frameworks.

[0003] However, most of the AllReduce algorithms in the MPI ensemble communication library are designed for homogeneous scenarios, while the GPU or NPU clusters required for training large-scale deep learning models often exhibit varying degrees of heterogeneity. Existing AllReduce algorithms perform poorly in deep learning scenarios. Therefore, many recent works have proposed efficient AllReduce algorithms for distributed training systems in deep learning.

[0004] For deep learning scenarios, the current mainstream AllReduce algorithms are as follows:

[0005] 1. Ring AllReduce is a typical AllReduce algorithm widely used in High Performance Computing (HPC) and has been proven to be bandwidth-optimal. In 2017, Baidu introduced Ring AllReduce into the field of deep learning and open-sourced its TensorFlow-based implementation. In Ring AllReduce, all nodes form a directed ring topology, with each node connected to only two adjacent nodes. Ring AllReduce first divides the data into blocks according to the number of nodes before communication. The algorithm consists of two parts: scatter-reduce and all-gather. In scatter-reduce, each node collects a single complete data block from different locations, and in all-gather, each node aggregates the data blocks from different locations to obtain the complete data. Ring AllReduce often achieves good results, but its communication latency is proportional to the number of nodes. Performance is limited when the number of nodes is large, and slow links in heterogeneous networks can become communication bottlenecks, reducing communication efficiency. A current multi-machine training hierarchical Ring AllReduce algorithm consists of three steps: intra-group Ring AllReduce, inter-group Ring AllReduce, and intra-group broadcast. This hierarchical topology fully utilizes the high bandwidth within groups while reducing the high communication overhead caused by the low bandwidth between groups. However, this algorithm limits the underlying physical topology, and the communication cost of intra-group broadcast is relatively high.

[0006] 2. 2D-Torus: This architecture is also a hierarchical architecture. It arranges GPUs into a two-dimensional logical network, with each GPU belonging to a group in both the vertical and horizontal directions. The algorithm steps are as follows: scatter-reduce operations are performed between nodes in the same group in the horizontal direction; ring all-reduce operations are performed between nodes in the same group in the vertical direction; and all-gather operations are performed between nodes in the same group in the horizontal direction. HiPS is an improved algorithm based on the same principle as 2D-Torus, extending it to the k-dimensional case and adding link parallel operations in addition to grouping decomposition. Taking the vertical-horizontal 2D grouping of GPUs as an example, when one process synchronizes using the "horizontal-vertical-vertical-horizontal" sequence, another process synchronizes using the "vertical-horizontal-horizontal-vertical" sequence, ensuring that the vertical-horizontal links are utilized at every moment. The link parallelism of HiPS can theoretically achieve a k-fold speedup in server-centric server physical topologies (such as Bcube and Torus). The above two algorithms are optimized based on commonly used hierarchical physical topologies, but they have certain limitations on the structure and dimensions of the physical topology.

[0007] 3. HD (recursive halving and doubling) AllReduce is a classic tree reduction algorithm. HD AllReduce also consists of two steps: scatter-reduce and all-gather. First, in scatter-reduce, half of the gradient from the previous step is swapped with the node whose distance has doubled, for a total of log₂n steps. All-gather is the reverse of scatter-reduce, swapping the full gradient from the previous step with the node whose distance has halved, also for a total of log₂n steps, ultimately completing the full reduction. This algorithm has a fixed flow and is only applicable to specific physical topologies.

[0008] 4. BlueConnect, a communication topology based on AllReduce decomposition, primarily involves decomposing the total number of nodes n into integers, and then dividing the decomposed nodes into p... i The algorithm decomposes AllReduce into parallelizable multi-stage scatter-reduce and all-gather operations. Through appropriate decomposition, the reduction process can be mapped to the underlying physical topology, thus adapting to heterogeneous network layers. This algorithm proposes a more general framework, and in fact, it can encompass algorithms such as RingAllReduce, 2D-Torus, and HD AllReduce through different forms of decomposition. However, although this algorithm is adaptable to different types of heterogeneous networks, insufficient link utilization in heterogeneous network environments limits its performance. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies by proposing an efficient parameter synchronization method based on the AllReduce architecture.

[0010] The objective of this invention is achieved through the following technical solution: a high-efficiency parameter synchronization method based on the AllReduce architecture, which includes the following steps:

[0011] The nodes are grouped and decomposed according to the underlying physical topology of the GPU / NPU cluster system. The communication volume during the data synchronization phase is calculated based on the node grouping results, so that the decomposed link bandwidth matches the communication volume during the data synchronization phase.

[0012] The data to be synchronized is divided into data blocks based on the communication volume of different communication stages after decomposition and the bandwidth of each link.

[0013] The data blocks obtained from the splitting are synchronized in parallel. The data synchronization communication methods include scatter-reduce and all-gather. The number of stages of each data synchronization communication method is the same as the number of groups of nodes. After the communication of each stage is completed, the synchronization operation between the communication processes of different data blocks is performed.

[0014] Ultimately, this allows each node to have a complete data block that aggregates data from all nodes, thus completing the parameter synchronization process.

[0015] Furthermore, the step of grouping and decomposing nodes based on the underlying physical topology of the GPU / NPU cluster system includes: grouping the nodes into integer decomposition forms as follows:

[0016]

[0017] Where n is the total number of nodes participating in synchronization, k is the number of groups, which is the same as the number of communication stages in scatter-reduce and all-gather in data synchronization, and p i The number of nodes in each communication group during each stage of data synchronization.

[0018] Furthermore, the communication volume during the data synchronization phase specifically includes:

[0019] The amount of communication data between scatter-reduce stage i and the i-th reciprocal stage of all-gather is The amount of communication data decreases as i increases, where m is the size of the data to be synchronized.

[0020] Furthermore, the decomposition method that matches the decomposed link bandwidth with the communication volume during the data synchronization phase includes:

[0021] For a multidimensional network system of dimension d, let the size of each dimension be (a1, a2, ... a...). d The network bandwidth corresponding to each dimension is (q1, q2, ... q). d ), has the number of nodes The decomposition method is to make p i Numerically, it corresponds to the size of each dimension of the system, i.e., p i =a i (i = 1, 2, ..., d), k = d.

[0022] Furthermore, the decomposition method that matches the decomposed link bandwidth with the communication volume during the data synchronization phase includes:

[0023] For a multidimensional network system of dimension d, let the size of each dimension be (a1, a2, ... a...). d The network bandwidth corresponding to each dimension is (q1, q2, ... q).d ), has the number of nodes The decomposition method is as follows:

[0024] Let g be the amount of communication data in stage i. i The minimum bandwidth on the i-th communication link is q. j The correspondence between i and j is as follows: minimize Thus, k and the optimal decomposition form are determined. The solution method is selected based on the size of the system.

[0025] Furthermore, the specific steps of dividing the data to be synchronized are as follows: The first data block adopts the default synchronization method that matches the data communication volume and link bandwidth at each stage of the synchronization process; the synchronization stage i of the r-th data block corresponds to the link of the (i+r-1)%k-th stage of the default synchronization method, and the size of the r-th data block is... in b i The bandwidth of the link is represented by m, the size of the data to be synchronized is m, and the total number of data blocks after segmentation is k, which is equal to the number of communication stages.

[0026] Furthermore, the data synchronization specifically includes:

[0027] A 2k-stage synchronization operation is performed on a single data block. The first k stages perform scatter-reduce operations, and the i-th stage contains a total of Several communication groups communicate in parallel on the same type of link, and each communication group has p i There are k nodes, and each communication group communicates using the scatter-reduce method in Ring Allreduce; the last k stages perform an all-gather operation, and there are a total of k nodes in the i-th stage from the end. Several communication groups communicate in parallel on the same type of link, and each communication group has p i Each node communicates with each communication group in an all-gather manner as in Ring Allreduce, and performs synchronization operations between different data block communication processes after the communication of each stage is completed.

[0028] Furthermore, the scatter-reduce operation specifically involves: further dividing a single data block into n sub-blocks based on the number of nodes; during communication, the nodes within the communication group form a ring, with each node sending corresponding data to the next node in the communication group; in stage i, there are a total of There are 1 communication group, and each communication group contains p. i Each node communicates once. A series of consecutive data sub-blocks, communicating together (p i -1) times, create an auxiliary variable array For the r-th data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k], which ensures that different data blocks communicate in parallel on different links, and the synchronization process of each data block is independent of each other; after each node receives a data sub-block, it sums it with the data sub-block at the corresponding position.

[0029] Furthermore, the all-gather operation specifically involves: passing the complete data sub-blocks collected in the scatter-reduce stage to each node; during communication, the nodes within the communication group form a ring, with each node sending corresponding data to the next node in the communication group; in the i-th to last stage, there are a total of There are 1 communication group, and each communication group contains p. i Each node communicates once. A series of consecutive data sub-blocks, communicating together (p i -1) times, create an auxiliary variable array For the r-th data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k]; after each node receives the data sub-block, it directly replaces the data sub-block at the corresponding position.

[0030] After all all-gather operations are completed in the last k stages, each node has aggregated all complete data sub-blocks, and finally each node has a complete data block that aggregates the data from all nodes.

[0031] On the other hand, this invention also provides a high-efficiency parameter synchronization device based on the AllReduce architecture, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the high-efficiency parameter synchronization method based on the AllReduce architecture.

[0032] The beneficial effects of this invention are:

[0033] 1. The method of this invention is adapted to the underlying physical links of different GPU / NPU cluster systems in deep learning scenarios, and can overcome the impact of bandwidth heterogeneity. By introducing additional data splitting and link parallel operation in the BlueConnect algorithm, the link utilization efficiency of the algorithm on the network hardware system is improved, thereby improving the execution speed of the algorithm. It can achieve excellent performance in both homogeneous and heterogeneous network environments.

[0034] 2. Due to the link parallelism of the method of this invention, the proposed algorithm can be efficiently parallelized in multi-NIC scenarios and can also improve the parallelism by utilizing idle links in single-NIC scenarios. It is compatible with various network hardware systems and is suitable for efficient parameter synchronization under complex and ever-changing network conditions. Attached Figure Description

[0035] Figure 1 A flowchart of an efficient parameter synchronization method based on the AllReduce architecture provided for embodiments of the present invention;

[0036] Figure 2 This is a schematic diagram of the physical topology of the GPU within the server used in an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of the overall process of the parameter synchronization method for 2×2×2 decomposition provided in an embodiment of the present invention;

[0038] Figure 4 A schematic diagram illustrating the specific process of the scatter-reduce method for 2×2×2 decomposition parameter synchronization provided in an embodiment of the present invention;

[0039] Figure 5 A schematic diagram illustrating the specific process of the all-gather parameter synchronization method for 2×2×2 decomposition provided in an embodiment of the present invention;

[0040] Figure 6 This is a performance comparison curve of the AllReduce algorithm in a homogeneous network environment provided in an embodiment of the present invention;

[0041] Figure 7 This is a performance comparison curve of the AllReduce algorithm in a heterogeneous network environment provided in an embodiment of the present invention;

[0042] Figure 8 This is a schematic diagram of a high-efficiency parameter synchronization device based on the AllReduce architecture, provided as an embodiment of the present invention. Detailed Implementation

[0043] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0044] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0045] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0046] For distributed learning tasks, especially distributed collaboration in deep learning scenarios, the number of parameters to be synchronized between nodes has increased dramatically in recent years as model size has continued to grow. This necessitates efficient AllReduce algorithms for parameter synchronization to ensure both algorithm performance and training speed. Simultaneously, distributed training faces fundamental challenges such as heterogeneous inter-node communication. In real-world scenarios with heterogeneous bandwidth, many existing AllReduce algorithms suffer from performance degradation due to low-bandwidth link limitations, failing to achieve efficient communication. Therefore, this invention introduces a high-efficiency parameter synchronization method based on the AllReduce architecture: Parallel-BlueConnect. This algorithm incorporates the link parallelism of the HIPS algorithm on top of the BlueConnect algorithm, overcoming the insufficient link utilization problem of the BlueConnect algorithm in heterogeneous network environments. The goal of this method is to achieve efficient parameter synchronization in complex network environments, overcome the impact of bandwidth heterogeneity, and simultaneously improve the parallelism of parameter synchronization.

[0047] The specific content of this invention embodiment is as follows: A high-efficiency parameter synchronization method based on the AllReduce architecture, the flowchart of which is shown below. Figure 1 As shown, the method includes the following steps:

[0048] S1, group and decompose the nodes according to the underlying physical topology of the GPU / NPU cluster system used for deep learning model training or inference, decomposing the total number of nodes n into the form of k integers multiplied together: n = p0p1p2…p k This ensures that the decomposed link bandwidth matches the communication volume;

[0049] S2, based on the communication volume of different communication stages after decomposition and the bandwidth of each link, the data to be synchronized x is... i (Model gradient, parameters, or intermediate layer activation values) are divided into k blocks x i =(x i1 ,x i2 ,…,x ik );

[0050] S3 synchronizes the k data blocks from S2 in parallel using a grouped AllReduce approach. The synchronization communication is divided into 2k stages, in which the k data blocks communicate in parallel on different types of links within each stage. After the communication is completed, each node aggregates the k data blocks to obtain the synchronized parameters. Complete the parameter synchronization process.

[0051] Furthermore, the node grouping and decomposition method in S1 is specifically as follows:

[0052] Group the nodes according to the following integer factorization form:

[0053]

[0054] Where n is the total number of nodes participating in synchronization, k is the total number of factors, and p i Let be the decomposition factor for stage i. After the above decomposition, the synchronous communication is divided into 2k stages. The first k stages perform scatter-reduce operations, and the i-th stage has a total of Several communication groups communicate in parallel on the same type of link, and each communication group has p i There are k nodes, and each communication group communicates using the scatter-reduce method in Ring Allreduce; the last k stages perform an all-gather operation, and there are a total of k nodes in the i-th stage from the end. Several communication groups communicate in parallel on the same type of link, and each communication group has p i Each node communicates in an all-gather manner, similar to the Ring Allreduce protocol.

[0055] The amount of communication data between scatter-reduce stage i and the i-th reciprocal stage of all-gather is The amount of communication data decreases as i increases, where m is the size of the data to be synchronized.

[0056] Grouping and decomposition require that the bandwidth of each type of link be matched as closely as possible to the amount of communication data in the scatter-reduce stage i. For a multidimensional network system of dimension d, let the size of each dimension be (a1, a2, ... a...). d The network bandwidth corresponding to each dimension is (q1, q2, ... q). d ), has the number of nodes A simple decomposition method is p i Numerically, it corresponds to the size of each dimension of the system, i.e., p i =a i (i = 1, 2, ..., d), k = d. Since the bandwidth decreases in each dimension in a real system, this decomposition method can guarantee a rough match between the amount of communication data and the link bandwidth, but it cannot guarantee optimality. The optimal decomposition form is obtained by solving the following problem: Let g be the amount of communication data in stage i. i The minimum bandwidth on the i-th communication link is q. j (Correspondence between i and j:) minimize Thus, k and the optimal decomposition form are determined. For systems of different sizes, exhaustive search, pruning search, and heuristic search methods can be used to solve the problem.

[0057] Furthermore, the data segmentation method in step S2 is specifically as follows:

[0058] Assume that for a group of nodes to be synchronized, its grouping decomposition form has been obtained based on its underlying physical topology, and that the bandwidth b of the link corresponding to stage i is... i At this point, data segmentation is performed so that different data blocks can be synchronized in parallel on different links. The first data block uses the default synchronization method, meaning that the data communication volume and link bandwidth are matched at each stage of its synchronization process. The synchronization stage i of the r-th (r = 2…k) data block corresponds to the link of the (i+r-1)%k-th stage of the default synchronization method, so that all types of links can synchronize in parallel at each synchronization stage. However, for the r-th (r = 2…k) data block, the data communication volume and link bandwidth are not perfectly matched. Therefore, when segmenting data, it is necessary to ensure that the i-th synchronization stage of any data block is not slower than the i-th synchronization stage of the first data block. Let the communication volume per unit data size in stage i be c. i The size of the r-th data block is m r Then the time taken for the i-th synchronization phase of the first data block is The time consumed by the i-th synchronization phase of the r-th data block is For any i, it is necessary to ensure t ir ≤t i1 ,Right now Note that the above inequality is also satisfied when r is 1.

[0059] Based on the above, the size of each data block is finally calculated through the following steps:

[0060] (1) For the r-th data block, calculate the relative size of the data block.

[0061] (2) Assume the size of the data to be synchronized is m, and for the r-th data block, its size is...

[0062] Furthermore, the parameter synchronization method in step S3 is specifically as follows:

[0063] For a single data block, the synchronous communication is divided into 2k stages. The first k stages perform scatter-reduce operations, and the i-th stage contains a total of Several communication groups communicate in parallel on the same type of link, and each communication group has p iThere are k nodes, and each communication group communicates using the scatter-reduce method in Ring Allreduce; the last k stages perform an all-gather operation, and there are a total of k nodes in the i-th stage from the end. Several communication groups communicate in parallel on the same type of link, and each communication group has p i Each node communicates in an all-gather manner, similar to the Ring Allreduce approach. Different data blocks are processed at each stage.

[0064] Asynchronous and parallel communication across different types of links, with k data blocks communicating simultaneously on each node, requires a multi-process approach. To prevent link congestion caused by misalignment of communication phases between different data blocks, synchronization operations between the communication processes of different data blocks are needed after each phase of communication is completed.

[0065] The scatter-reduce operation specifically involves further dividing a single data block into n sub-blocks based on the number of nodes. During communication, the nodes within the communication group form a ring, with each node sending its corresponding data to the next node in the group. In stage i, there are a total of... There are 1 communication group, and each communication group contains p. i Each node communicates once. A series of consecutive data sub-blocks, communicating together (p i -1) times, create an auxiliary variable array For the r-th data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k]. This ensures that different data blocks communicate in parallel on different links, and the synchronization process of each data block is independent of each other. After receiving a data sub-block, each node sums it with the data sub-block at the corresponding position.

[0066] After each scatter-reduce operation, each node collects a specific data sub-block from all nodes in the communication group. As the stage number i increases, the range of nodes collected for each node's corresponding data sub-block gradually increases. After all scatter-reduce operations in the first k stages are completed, each node has a sub-block that aggregates the data from all nodes.

[0067] The all-gather operation specifically involves passing the complete data sub-blocks collected in the scatter-reduce phase to each node. During communication, the nodes within the communication group form a ring, with each node sending its corresponding data to the next node in the group. In the i-th to last phase, there are a total of... There are 1 communication group, and each communication group contains p. i Each node communicates once. A series of consecutive data sub-blocks, communicating together (p i -1) times, create an auxiliary variable array For the r-th data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k]. Each node directly replaces the corresponding data block after receiving the data sub-block.

[0068] After each all-gather operation, each node aggregates the complete data sub-blocks corresponding to all nodes in the communication group. As the stage number i increases, the number of complete data sub-blocks aggregated by each node also increases. After all all-gather operations in the last k stages are completed, each node has aggregated all complete data sub-blocks, and finally each node has a complete data block that aggregates the data from all nodes.

[0069] I. System Basic Settings and Underlying Physical Topology

[0070] This embodiment is implemented on a single server with eight 2080Ti graphics cards, each with 11GB of video memory. In this embodiment, each GPU acts as a node. A process is started on each GPU using the `torch.distributed` interface within the PyTorch framework in Python, and point-to-point communication between GPU nodes is achieved using the `broadcast` function in `torch.distributed`. Since the algorithm does not affect the model update process in distributed learning, for simplicity, the parameters that need to be synchronized are directly generated using the `torch.rand` interface.

[0071] The physical topology of the GPU within the server is as follows: Figure 2 As shown, there are three types of links connecting GPUs within a single machine. The first type is communication across a single PCIe Bridge (Switch), where two GPU cards are mounted on the same PCIe Bridge, equivalent to GPU0 to GPU1 in the diagram. The second type is communication across multiple PCIe Bridges (Switches), equivalent to GPU0 to GPU2 in the diagram. The third type is GPU communication across NUMA nodes via QPI / UPI, where each NUMA node consists of one CPU; this link is equivalent to GPU0 to GPU4 in the diagram. To ensure efficient communication between GPUs within a single machine, the bandwidth of these three different links is not significantly different in practice, and they can be considered essentially homogeneous networks. Therefore, this embodiment constructs heterogeneous bandwidth for different links through software settings to simulate the heterogeneous bandwidth environment commonly found in inter-server communication or cross-data center communication.

[0072] GPUs 0, 1, 2, 3 and 4, 5, 6, 7 are mounted on different CPUs. In this embodiment, the bandwidth bottleneck of communication between GPU sub-clusters mounted on different CPUs is simulated by limiting the bandwidth of the third type of link mentioned above through software. At the same time, this situation can also simulate the bandwidth bottleneck between server clusters connected to different switches.

[0073] II. Specific methods for node decomposition

[0074] Let the bandwidth capacities of the three links introduced in the previous section be W1, W2, and W3, respectively. When W3 becomes the communication bottleneck (i.e., W3 is significantly smaller than W1 and W2), the amount of communication data on the link corresponding to W3 needs to be minimized. Based on the communication volume in stage i... Take the maximum value n, (p i -1) When the minimum value of 1 is taken, the amount of communication data in this stage is minimal. Therefore, the communication in the final stage should be carried out on the link corresponding to W3 through group decomposition. At the same time, the final stage p i The value is 2. For a single machine with 8 GPU nodes, the optimal decomposition is 4×2 or 2×2×2. The 2×2×2 decomposition method, after data partitioning in the method proposed in this invention, can achieve higher parallelism and better acceleration. Therefore, this embodiment uses the 2×2×2 decomposition as the specific decomposition method in the implementation.

[0075] III. Specific Methods of Data Partitioning

[0076] In the 2×2×2 decomposition form, k is 3, and the data to be synchronized is divided into 3 blocks. In this embodiment, bandwidth is limited for the cross-CPU GPU communication link within the server, with limit_coe representing the degree of bandwidth limitation. The limited bandwidth is the original bandwidth. That is According to step S2, the calculation is obtained This gives us the size ratio of the three data blocks as 1 + limit_coe:1:1. We can then divide the original data blocks into smaller blocks according to this ratio.

[0077] IV. Specific Process of Parameter Synchronization

[0078] The overall process of the parameter synchronization method for 2×2×2 decomposition in this embodiment is as follows: Figure 3 As shown in the diagram, synchronization is divided into three stages, with three sets of data communicating in parallel within each stage. In this embodiment, the distance_list is [1,2,4], and the solid line represents data block x. i1 The communication link, with the dashed line representing data block x. i2 The communication link, with dotted lines representing data blocks x. i3The communication link for these three data blocks is independent of each other. To prevent link congestion caused by misalignment of communication stages between different data blocks, synchronization between the communication processes of different data blocks is required after the communication of each stage is completed.

[0079] The specific process of parameter synchronization for a single data block is described below, taking data block x as an example. i1 For example, the specific processes of scatter-reduce and all-gather in parameter synchronization are as follows: Figure 4 and Figure 5 As shown. In the scatter-reduce stage i, the node p with a node index difference of distance_list[i] is... i Each node communicates with the others in a group. After each communication phase, the data sub-block at a specific location within the node collects the data from the corresponding data sub-blocks of all nodes in the communication group. After all scatter-reduce phases are completed, each node has a data sub-block that aggregates the data from all nodes. For example... Figure 5 As shown, in the i-th stage from the end of all-gather, the node p with a node index difference of distance_list[i] is... i Each node communicates with the others in a group. After each communication phase, the node summarizes the complete data sub-blocks of all nodes in the communication group at their corresponding positions. After all all-gather operations are completed, each node has summarized all complete data sub-blocks, and finally each node has a complete data block that aggregates the data from all nodes.

[0080] For the second and third data blocks, the synchronization process is basically the same as that for the first data block, except that the difference between the node indexes in the communication group of the scatter-reduce stage i and the i-th-last stage of all-gather is distance_list[(i+r)%k] (r=2,3), which corresponds to the links in the scatter-reduce stage (i+r)%k and the i-th-last stage of all-gather for the first data block, respectively.

[0081] V. Numerical Verification Experiment

[0082] This invention validates the effectiveness of the proposed parameter synchronization method through numerical experiments, which were conducted using Python. The experiments were designed from the following aspects:

[0083] Example 1: This experiment compares the Mesh AllReduce algorithm, Ring AllReduce algorithm, BlueConnect algorithm, and the parameter synchronization method Parallel-BlueConnect (hereinafter referred to as Mesh, Ring, Blue, and P-Blue) of this invention. Both the Blue and P-Blue algorithms employ a 2×2×2 decomposition corresponding to the physical topology within a single machine. Random parameters to be synchronized are generated, and the above four methods are used for parameter synchronization. The time consumed by the entire synchronization process is measured, and the results are plotted as follows: Figure 6 As shown.

[0084] Figure 6 The horizontal axis represents the data size of a single node, ranging from 1KB to 1GB, and the vertical axis represents the global synchronization time. Both axes use an exponential coordinate system. The graph shows that when the amount of communication data is substantial, the synchronization times of Ring and Blue are essentially the same, while the synchronization times of Mesh and P-Blue are essentially the same and shorter than those of Ring and Blue. This aligns with theoretical results. In homogeneous networks, Ring and Blue have the same amount of communication data, the same number of communication links, and similar link utilization. Mesh and P-Blue, however, have higher link utilization, higher equivalent communication bandwidth, and shorter global synchronization times compared to Ring and Blue. When the amount of communication data is small (<1MB), the global synchronization time is basically consistent with the algorithm's latency term. The method in this invention, due to the additional data segmentation and synchronization operations between different data block groups, incurs a relatively larger time overhead when the amount of communication data is small. The global synchronization time is similar to that of Ring, but in this case, the global synchronization time is in the millisecond range and has virtually no impact on the overall training process.

[0085] Example 2: This experiment constructed a network environment with heterogeneous bandwidth using software settings. In this scenario, the Mesh AllReduce algorithm, Ring AllReduce algorithm, BlueConnect algorithm, and the method of this invention, Parallel-BlueConnect (hereinafter referred to as Mesh, Ring, Blue, and P-Blue), were compared. This experiment limited the bandwidth of the GPU communication link across CPUs within the server, using limit_coe to represent the degree of bandwidth limitation. The bandwidth after limitation was 1 / (1+limit_coe) of the original. In software implementation, bandwidth limitation for specific links was achieved by sending additional invalid data. The data volume of a single node was 100MB. The global synchronization time of each algorithm in the heterogeneous bandwidth scenario is as follows: Figure 7 As shown.

[0086] Depend on Figure 7It can be seen that Ring and Mesh algorithms are greatly affected by heterogeneous networks. The bandwidth limitation of local links significantly increases the global synchronization time. Specifically, the Ring algorithm's synchronization time in heterogeneous network environments increases by nearly a factor of limit_coe, with communication time essentially depending on the link with the smallest bandwidth. In contrast, Blue and P-Blue, through node sequence mapping and packet decomposition, minimize the data transmission volume on communication-limited links, thus mitigating the impact of these links on synchronization time. The graph shows that the global synchronization time curves of Blue and P-Blue with increasing limit_coe have a smaller slope, adapting better to heterogeneous bandwidth environments. Compared to the Blue algorithm, P-Blue has a significantly shorter synchronization time when limit_coe is small (i.e., less heterogeneous network bandwidth), but gradually becomes the same as the Blue algorithm as network bandwidth heterogeneity increases. This is because if the bandwidth of a bandwidth-limited link is significantly smaller than other links, the global synchronization time is mainly dominated by the communication time on that link, and the parallelism of other links cannot accelerate the synchronization process.

[0087] As can be seen from the above embodiments, the Parallel-BlueConnect method of the present invention significantly outperforms the Mesh AllReduce and Ring AllReduce algorithms in heterogeneous networks, and outperforms this method in homogeneous and low-heterogeneity environments. This method can flexibly decompose the underlying physical topology, and its multi-link parallel characteristics enable it to maximize the utilization of different types of links, achieving optimal performance in both homogeneous and heterogeneous network environments.

[0088] Corresponding to the aforementioned embodiment of an efficient parameter synchronization method based on the ALLREDUCE architecture, the present invention also provides an embodiment of an efficient parameter synchronization device based on the ALLREDUCE architecture.

[0089] See Figure 8 The present invention provides an efficient parameter synchronization device based on the ALLREDUCE architecture, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an efficient parameter synchronization method based on the ALLREDUCE architecture in the above embodiment.

[0090] The embodiment of the high-efficiency parameter synchronization device based on the ALLREDUCE architecture provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 8 The diagram shown illustrates a hardware structure of any data processing-capable device, including the high-efficiency parameter synchronization device based on the ALLREDUCE architecture provided by this invention. (Except for...) Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0091] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0092] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0093] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements an efficient parameter synchronization method based on the ALLREDUCE architecture described in the above embodiments.

[0094] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0095] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned efficient parameter synchronization method based on the ALLREDUCE architecture.

[0096] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0097] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A high-efficiency parameter synchronization method based on the AllReduce architecture, characterized in that, The method includes the following steps: The nodes are grouped and decomposed according to the underlying physical topology of the GPU / NPU cluster system. The communication volume during the data synchronization phase is calculated based on the node grouping results, so that the decomposed link bandwidth matches the communication volume during the data synchronization phase. The data to be synchronized is divided into data blocks based on the communication volume of different communication stages after decomposition and the bandwidth of each link. The specific steps of dividing the data to be synchronized are as follows: The first data block adopts the default synchronization method that matches the data communication volume and link bandwidth at each stage of the synchronization process; the synchronization stage i of the r-th data block corresponds to the i-th stage of the default synchronization method. The link consists of several stages, and the size of the r-th data block is... ;in , This is expressed as the bandwidth of the link; m is the size of the data to be synchronized, and k is the total number of data blocks after segmentation, which is equal to the number of communication stages; The data blocks obtained from the splitting are synchronized in parallel. The data synchronization communication methods include scatter-reduce and all-gather. The number of stages of each data synchronization communication method is the same as the number of groups of nodes. After the communication of each stage is completed, the synchronization operation between the communication processes of different data blocks is performed. The data synchronization specifically refers to: A 2k-stage synchronization operation is performed on a single data block. The first k stages perform scatter-reduce operations, and the i-th stage contains a total of Several communication groups communicate in parallel on the same type of link, where n is the total number of nodes participating in synchronization, and each communication group has... There are k nodes, and each communication group communicates using the scatter-reduce method in Ring Allreduce; the last k stages perform an all-gather operation, and there are a total of k nodes in the i-th stage from the end. Several communication groups communicate in parallel on the same type of link, and each communication group has Each node and each communication group communicates in an all-gather manner as in Ring Allreduce, and performs synchronization operations between different data block communication processes after the communication of each stage is completed; Ultimately, this allows each node to have a complete data block that aggregates data from all nodes, thus completing the parameter synchronization process.

2. The efficient parameter synchronization method based on the AllReduce architecture according to claim 1, characterized in that, The step of grouping and decomposing nodes based on the underlying physical topology of the GPU / NPU cluster system includes: grouping the nodes into integer decomposition forms as follows: ； Where k is the number of groups, which is the same as the number of communication stages in scatter-reduce and all-gather during data synchronization. The number of nodes in each communication group during each stage of data synchronization.

3. The efficient parameter synchronization method based on the AllReduce architecture according to claim 2, characterized in that, The communication volume during the data synchronization phase is specifically as follows: The amount of communication data between scatter-reduce stage i and the i-th reciprocal stage of all-gather is The amount of communication data decreases as i increases, where m is the size of the data to be synchronized.

4. The efficient parameter synchronization method based on the AllReduce architecture according to claim 3, characterized in that, The decomposition methods that match the decomposed link bandwidth with the communication volume during the data synchronization phase include: For a multidimensional network system of dimension d, let the size of each dimension be ( The network bandwidth corresponding to each dimension is ( ). ), has the number of nodes The decomposition method is to make Numerically, it corresponds to the size of each dimension of the system, that is... , .

5. The efficient parameter synchronization method based on the AllReduce architecture according to claim 3, characterized in that, The decomposition methods that match the decomposed link bandwidth with the communication volume during the data synchronization phase include: For a multidimensional network system of dimension d, let the size of each dimension be ( The network bandwidth corresponding to each dimension is ( ). ), has the number of nodes The decomposition method is as follows: Let the communication data volume of stage i be denoted as The minimum bandwidth on the i-th stage communication link is The correspondence between i and j is as follows: ;minimize Thus, k and the optimal decomposition form are determined. The solution method is selected based on the size of the system.

6. The efficient parameter synchronization method based on the AllReduce architecture according to claim 1, characterized in that, The scatter-reduce operation specifically involves: further dividing a single data block into n sub-blocks based on the number of nodes; during communication, the nodes within the communication group form a ring, with each node sending corresponding data to the next node in the communication group; In stage i, there are a total of There are 1 communication group, and each communication group contains 12 communication groups. Each node communicates once. A series of consecutive data sub-blocks, communicating together ( ( ) times, create an auxiliary variable array distance_list = [1, , … For the r-th data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i +r)%k]. This ensures that different data blocks communicate in parallel on different links, and the synchronization process of each data block is independent of each other. After receiving the data sub-block, each node sums it with the data sub-block at the corresponding position.

7. The efficient parameter synchronization method based on the AllReduce architecture according to claim 1, characterized in that, The all-gather operation specifically involves: passing the complete data sub-blocks collected in the scatter-reduce phase to each node; during communication, the nodes in the communication group form a ring, with each node sending corresponding data to the next node in the communication group; in the i-th to last phase, there are a total of There are 1 communication group, and each communication group contains 12 communication groups. Each node communicates once. A series of consecutive data sub-blocks, communicating together ( ( ) times, create an auxiliary variable array distance_list = [1, , … For the r-th data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i +r)%k]; after each node receives a data sub-block, it directly replaces the data sub-block at the corresponding position; After all all-gather operations are completed in the last k stages, each node has aggregated all complete data sub-blocks, and finally each node has a complete data block that aggregates the data from all nodes.

8. A high-efficiency parameter synchronization device based on the AllReduce architecture, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an efficient parameter synchronization method based on the AllReduce architecture as described in any one of claims 1-7.

Images