Efficient parameter synchronization method based on AllReduce architecture

Abstract

The invention discloses an efficient parameter synchronization method based on an AllReduce architecture, which comprises the following steps of: firstly, performing grouping decomposition on nodes according to a bottom layer physical link bandwidth to enable the decomposed link bandwidth to be matched with communication traffic, and then partitioning data to be communicated according to the communication traffic of different communication stages after decomposition, and finally, carrying out parallel parameter synchronization on each data block on different types of links in a packet decomposition AllReduce mode. The method is used for efficient parameter synchronization in a complex network environment, the influence of bandwidth heterogeneity can be overcome, efficient parallelism can be achieved in a multi-network-card scene, the parallelism degree can be improved by using an idle link in a single-network-card scene, and excellent performance can be achieved in both isomorphic and heterogeneous network environments.

Description

Technical Field

[0001] The present invention belongs to the field of computer distributed communication, and in particular relates to an efficient parameter synchronization method based on AllReduce architecture. Background Art

[0002] With the development of deep learning and large model technology, the scale of neural network models used in practice is getting larger and larger. In most scenarios, the memory of a single node can no longer accommodate a complete neural network model. At the same time, the computing power of a single node cannot complete the training or reasoning of the model within a reasonable time. Therefore, various distributed parallel training technologies have been developed to achieve efficient training and reasoning of large-scale neural network models. The basic parallel modes in distributed parallel training technology include data parallelism and model parallelism. However, no matter which parallel mode is used, frequent communication between nodes is required to synchronize parameters. For data parallelism, the parameters to be synchronized are gradients or model parameters. For model parallelism, the parameters to be synchronized are activation values ​​of the model's middle layer. Frequent communication between nodes limits the performance of model training, and thus limits the scalability of distributed training systems. Therefore, it is very important to design an efficient method for synchronizing parameters between nodes. In order to solve this problem, the AllReduce algorithm in the MPI collective communication library was introduced into this field. AllReduce is a type of algorithm that performs efficient parameter synchronization through data block segmentation and collective communication scheduling. Some classic algorithms such as Ring AllReduce have been integrated into popular deep learning frameworks.

[0003] However, most of the AllReduce algorithms in the MPI collective communication library are designed in homogeneous scenarios, while the GPU or NPU clusters required for training large-scale deep learning models often have varying degrees of heterogeneity, and the existing AllReduce algorithms perform poorly in deep learning scenarios. Therefore, in recent years, many works have proposed efficient AllReduce algorithms for deep learning distributed training systems.

[0004] For deep learning scenarios, the existing mainstream AllReduce algorithms are as follows:

[0005] 1. Ring AllReduce is a typical AllReduce algorithm. It is widely used in the field of High Performance Computing (HPC). It has been proven to be the bandwidth-optimal algorithm. In 2017, Baidu introduced Ring AllReduce into the field of deep learning and open-sourced the implementation code based on TensorFlow. In RingAllReduce, all nodes form a directed ring topology, and each node is only connected to two adjacent nodes. RingAllReduce first divides the data into blocks according to the number of nodes before communicating. The algorithm process is divided into two parts: scatter-reduce and all-gather. In scatter-reduce, each node collects a single complete data block at different locations. In all-gather, each node aggregates data blocks at different locations to obtain complete data. Ring AllReduce often achieves relatively good results, but its communication delay is proportional to the number of nodes. When the number of nodes is large, the performance will be restricted. At the same time, slow links in heterogeneous networks will become communication bottlenecks, reducing communication efficiency. In the prior art, a hierarchical Ring AllReduce for multi-machine training has the following three steps: intra-group Ring AllReduce, inter-group Ring AllReduce, and intra-group broadcast. Such a hierarchical topology fully utilizes the high bandwidth within the group while reducing the high communication overhead caused by the low bandwidth between groups. However, the algorithm limits the underlying physical topology, and the communication cost of intra-group broadcast is relatively high.

[0006] 2. 2D-Torus, this architecture is also a hierarchical architecture. It arranges GPUs into a two-dimensional logical network. Each GPU belongs to a group in the vertical and horizontal directions. The algorithm steps are as follows: scatter-reduce operations are performed between nodes in the same group in the horizontal direction; ring all-reduce operations are performed between nodes in the same group in the vertical direction; all-gather operations are performed between nodes in the same group in the horizontal direction. HiPS is an improved algorithm based on the same principle as 2D-Torus. It extends 2D-Torus to the k-dimensional case. In addition to the group decomposition, it adds link parallel operations. Taking the vertical-horizontal 2D grouping of GPUs as an example, when one process uses the "horizontal-vertical-vertical-horizontal" sequence for synchronization, another process is used to synchronize in the "vertical-horizontal-horizontal-vertical" sequence, so that the vertical-horizontal links at each moment are utilized. The link parallel characteristics of HiPS can theoretically achieve k times acceleration in server-centric server physical topologies (such as Bcube and Torus). The above two algorithms are optimized based on the commonly used hierarchical physical topology, but there are certain restrictions on the structure and dimension of the physical topology.

[0007] 3. HD (recursive halving and doubling) AllReduce is a classic tree reduction algorithm. HD AllReduce is also divided into two steps: scatter-reduce and all-gather. The first is scatter-reduce, which exchanges half of the gradients in the previous step with the nodes whose distances are doubled each time, and executes log2n steps in total. All-gather is the reverse process of scatter-reduce, which exchanges the full gradient segments in the previous step with the nodes whose distances are halved each time, and also executes log2n steps to complete the full reduction. The algorithm process is fixed and is only applicable to specific physical topologies.

[0008] 4. BlueConnect, a communication topology based on AllReduce decomposition, is a topology that uses integer decomposition of the total number of nodes n to calculate the number of nodes p. i Grouping, decomposing AllReduce into parallelizable multi-stage scatter-reduce and all-gather operations. Through appropriate decomposition, the reduction process can be mapped to the underlying physical topology, thus adapting to heterogeneous network levels. The algorithm proposes a more general framework, and in fact can cover algorithms such as RingAllReduce, 2D-Torus, and HD AllReduce through different forms of decomposition. However, although the algorithm is adaptable to different types of heterogeneous networks, the link utilization is insufficient when communicating in a heterogeneous network environment, which limits the performance of the algorithm. Summary of the invention

[0009] The purpose of the present invention is to address the deficiencies of the prior art and to propose an efficient parameter synchronization method based on the AllReduce architecture.

[0010] The object of the present invention is achieved through the following technical solution: an efficient parameter synchronization method based on AllReduce architecture, the method comprising the following steps:

[0011] The nodes are grouped and decomposed according to the underlying physical topology of the GPU / NPU cluster system, and the communication volume in the data synchronization phase is calculated based on the node grouping results, so that the link bandwidth after decomposition matches the communication volume in the data synchronization phase;

[0012] The data to be synchronized is divided into data blocks according to the communication volume in different communication stages after decomposition and the bandwidth size of each link;

[0013] The data blocks obtained by segmentation are synchronized in parallel. The data synchronization communication modes include scatter-reduce and all-gather. The number of stages of each data synchronization communication mode is the same as the number of node groupings. After the communication of each stage is completed, the synchronization operation between the communication processes of different data blocks is performed.

[0014] Eventually, each node has a complete data block that aggregates the data of all nodes, completing the parameter synchronization process.

[0015] Further, grouping and decomposing the nodes according to the underlying physical topology of the GPU / NPU cluster system includes: grouping the nodes in the following integer decomposition form:

[0016]

[0017] Where n is the total number of nodes participating in synchronization, k is the number of groups, which is the same as the number of communication phases of scatter-reduce and all-gather in data synchronization, and p i The number of nodes in each communication group in each phase of data synchronization.

[0018] Furthermore, the communication volume in the data synchronization phase is specifically:

[0019] The amount of communication data between the scatter-reduce stage i and the penultimate i-th stage of all-gather is The amount of communication data decays as i increases, where m is the size of the data to be synchronized.

[0020] Furthermore, the decomposition method that makes the decomposed link bandwidth match the communication volume in the data synchronization phase includes:

[0021] For a multidimensional network system with dimension d, let the size of each dimension be (a1, a2, ... a d ), the network bandwidth corresponding to each dimension is (q1,q2,…q d ), with the number of nodes The decomposition method is to make p i The value corresponds to the size of each dimension of the system, that is, p i =a i (i=1,2……d), k=d.

[0022] Furthermore, the decomposition method for matching the decomposed link bandwidth with the communication volume in the data synchronization phase includes:

[0023] For a multidimensional network system with dimension d, let the size of each dimension be (a1, a2, ... a d ), the network bandwidth corresponding to each dimension is (q1,q2,…qd ), with the number of nodes The decomposition method is:

[0024] Let the communication data volume in stage i be g i , the minimum bandwidth on the communication link in stage i is q j , where the corresponding relationship between i and j is: minimize So as to determine k and the optimal decomposition form The solution method is selected according to the size of the system.

[0025] Furthermore, the data to be synchronized is divided into: the first data block adopts the default synchronization mode in which the data traffic and link bandwidth of each stage in the synchronization process match; the synchronization stage i of the rth data block corresponds to the link of the (i+r-1)%kth stage of the default synchronization mode, and the size of the rth data block is in b i It is expressed as the bandwidth of the link; m is the size of the data to be synchronized, and k is the total number of data blocks after segmentation, which is equal to the number of communication stages.

[0026] Furthermore, the data synchronization is specifically as follows:

[0027] A 2k-stage synchronization operation is performed on a single data block. The first k stages perform scatter-reduce operations. There are a total of communication groups communicate in parallel on the same type of links, and there are p communication groups in each communication group. i nodes, each communication group communicates in the scatter-reduce mode in Ring Allreduce; the last k stages perform all-gather operations, and there are a total of communication groups communicate in parallel on the same type of links, and there are p communication groups in each communication group. i Each communication group communicates in the all-gather mode in Ring Allreduce, and performs synchronization operations between communication processes of different data blocks after the communication in each stage is completed.

[0028] Furthermore, the scatter-reduce operation is specifically as follows: a single data block is further divided according to the number of nodes, that is, divided into n sub-blocks; during communication, the nodes in the communication group are constructed into a ring, and each node sends corresponding data to the next node in the communication group; in stage i, there are communication groups, each of which has p i nodes, each node communicates each time Continuous data sub-blocks, a total of (p i -1) times, create an auxiliary variable array For the rth data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k]. This ensures that different data blocks communicate in parallel on different links, and the synchronization process of each data block is independent of each other; after each node receives a data sub-block, it sums it with the data sub-block at the corresponding position.

[0029] Furthermore, the all-gather operation is specifically as follows: the complete data sub-blocks collected in the scatter-reduce stage are delivered to each node; during communication, the nodes in the communication group are constructed into a ring, and each node sends corresponding data to the next node in the communication group; in the penultimate i-th stage, there are communication groups, each of which has p i nodes, each node communicates each time Continuous data sub-blocks, a total of (p i -1) times, create an auxiliary variable array For the rth data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k]; after each node receives a data sub-block, it directly replaces the data sub-block at the corresponding position;

[0030] After all the all-gather operations in the last k stages are completed, each node summarizes all the complete data sub-blocks, and finally each node has a complete data block that aggregates the data of all nodes.

[0031] On the other hand, the specification of the present invention also provides an efficient parameter synchronization device based on the AllReduce architecture, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, an efficient parameter synchronization method based on the AllReduce architecture is implemented.

[0032] Beneficial effects of the present invention:

[0033] 1. The method of the present invention adapts to the underlying physical links of different GPU / NPU cluster systems in deep learning scenarios, can overcome the impact of bandwidth heterogeneity, and improves the link utilization efficiency of the algorithm for the network hardware system by introducing additional data segmentation and link parallel operation in the BlueConnect algorithm, thereby improving the execution speed of the algorithm, and can achieve excellent performance in both homogeneous and heterogeneous network environments;

[0034] 2. Due to the link parallel characteristics of the method of the present invention, the proposed algorithm can be efficiently parallelized in multi-network card scenarios, and can also use idle links to improve parallelism in single network card scenarios. It is adaptable to various network hardware systems and is suitable for efficient parameter synchronization under complex and changeable network conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of an efficient parameter synchronization method based on the AllReduce architecture provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the physical topology of a GPU in a server used in an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of the overall flow of a 2×2×2 decomposed parameter synchronization method provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of a specific process of a scatter-reduce parameter synchronization method for a 2×2×2 decomposition provided by an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of a specific process of the all-gather parameter synchronization method of a 2×2×2 decomposition provided by an embodiment of the present invention;

[0040] Figure 6 A performance comparison curve of the AllReduce algorithm in a homogeneous network environment provided by an embodiment of the present invention;

[0041] Figure 7 A performance comparison curve of the AllReduce algorithm in a heterogeneous network environment provided by an embodiment of the present invention;

[0042] Figure 8 A schematic diagram of an efficient parameter synchronization device based on AllReduce architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0044] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0045] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0046] For distributed learning tasks, especially distributed collaboration in deep learning scenarios, with the continuous increase of model scale in recent years, the amount of synchronized parameters between nodes has increased dramatically, and an efficient AllReduce algorithm is needed to synchronize parameters to ensure the performance and training speed of the algorithm. At the same time, distributed training has basic problems such as heterogeneous communication between nodes. In actual scenarios such as bandwidth heterogeneity, many existing AllReduce algorithms are limited by low-bandwidth links, and performance drops sharply, making it impossible to achieve efficient communication. Therefore, the present invention introduces an efficient parameter synchronization method Parallel-BlueConnect based on the AllReduce architecture. The algorithm introduces the link parallelism of the HIPS algorithm on the basis of the BlueConnect algorithm, overcoming the problem of insufficient link utilization of the BlueConnect algorithm in a heterogeneous network environment. The goal of this method is to achieve efficient parameter synchronization in a complex network environment, overcome the influence of bandwidth heterogeneity, and improve the parallelism of parameter synchronization.

[0047] The specific content of the embodiment of the present invention is as follows: An efficient parameter synchronization method based on AllReduce architecture, the method flow chart is as follows Figure 1 As shown, the method comprises the following steps:

[0048] S1, group and decompose the nodes according to the underlying physical topology of the GPU / NPU cluster system used for deep learning model training or reasoning, and decompose the total number of nodes n into the form of k integer multiplication n = p0p1p2…p k , so that the decomposed link bandwidth matches the communication volume;

[0049] S2, according to the communication volume of different communication stages after decomposition and the bandwidth size of each link, the data to be synchronized x i (Model gradient, parameter or intermediate layer activation value) is divided into k blocks x i =(x i1 ,x i2 ,…,x ik );

[0050] S3 synchronizes the k data blocks in S2 in parallel by grouping AllReduce. The synchronous communication is divided into 2k stages. In each stage, k data blocks communicate in parallel on different types of links. After the communication is completed, each node aggregates k data blocks to obtain the synchronized parameters. Complete the parameter synchronization process.

[0051] Furthermore, the node grouping decomposition method in S1 is specifically as follows:

[0052] The nodes are grouped in the following integer decomposition form:

[0053]

[0054] Where n is the total number of nodes participating in synchronization, k is the total number of factors, and p i is the decomposition factor of the i-th stage. After the above decomposition, the synchronous communication is divided into 2k stages. The first k stages perform scatter-reduce operations, and there are communication groups communicate in parallel on the same type of links, and there are p communication groups in each communication group. i nodes, each communication group communicates in the scatter-reduce mode in Ring Allreduce; the last k stages perform all-gather operations, and there are a total of communication groups communicate in parallel on the same type of links, and there are p communication groups in each communication group. i Nodes, each communication group communicates in the all-gather manner in Ring Allreduce.

[0055] The amount of communication data between the scatter-reduce stage i and the penultimate i-th stage of all-gather is The amount of communication data decreases as i increases, where m is the size of the data to be synchronized.

[0056] The packet decomposition needs to make the bandwidth of each type of link match and be close to the communication data volume of the scatter-reduce stage i as much as possible. For a multi-dimensional network system with dimension d, let the size of each dimension be (a1, a2, ... a d ), the network bandwidth corresponding to each dimension is (q1,q2,…q d ), with the number of nodes A simple decomposition is p i The value corresponds to the size of each dimension of the system, that is, p i =a i (i=1,2……d), k=d. Since the bandwidth in each dimension of the actual system is decreasing, this decomposition method can ensure the approximate match between the communication data volume and the link bandwidth, but it cannot guarantee the optimality. For the optimal decomposition form, we can solve the following problem: Let the communication data volume in stage i be g i , the minimum bandwidth on the communication link in stage i is q j (The corresponding relationship between i and j: minimize So as to determine k and the optimal decomposition form For systems of different sizes, exhaustive search, pruning search, and heuristic search methods can be used to solve the problem.

[0057] Furthermore, the data segmentation method in step S2 is specifically as follows:

[0058] Assume that for a group of nodes to be synchronized, its group decomposition form has been obtained according to its underlying physical topology, and there is a bandwidth b of the link corresponding to stage i i . At this time, data is divided into blocks so that different data blocks can be synchronized in parallel on different links. The first data block adopts the default synchronization method, that is, the data communication volume and link bandwidth in each stage of its synchronization process are matched. The synchronization stage i of the rth (r=2…k) data block corresponds to the (i+r-1)%kth stage link of the default synchronization method, so that all types of links can be synchronized in parallel in each synchronization stage. However, at this time, for the rth (r=2…k) data block, the data communication volume and link bandwidth are not completely matched. Therefore, when dividing the data, it is necessary to ensure that the i-th synchronization stage of any data block is not slower than the i-th synchronization stage of the first data block. Let the communication volume per unit data size in stage i be c i , the size of the rth data block is m r , then the time consumed in the i-th synchronization phase of the first data block is The i-th synchronization phase of the r-th data block takes time For any i, we need to ensure that t ir ≤t i1 ,Right now Note that the above inequality is also satisfied when r is 1.

[0059] Based on the above, the size of each data block is finally calculated through the following steps:

[0060] (1) For the rth data block, calculate the relative size of the data block

[0061] (2) Assuming that the size of the data to be synchronized is m, the size of the rth data block is

[0062] Furthermore, the parameter synchronization method in step S3 is specifically:

[0063] For a single data block, synchronous communication is divided into 2k stages. The first k stages perform scatter-reduce operations, and the i-th stage has a total of communication groups communicate in parallel on the same type of links, and there are p communication groups in each communication group. inodes, each communication group communicates in the scatter-reduce mode in Ring Allreduce; the last k stages perform all-gather operations, and there are a total of communication groups communicate in parallel on the same type of links, and there are p communication groups in each communication group. i Each communication group communicates in the all-gather mode in Ring Allreduce.

[0064] Asynchronous and parallel communication is performed on different types of links. Each node has k data blocks communicating at the same time, which needs to be implemented through multiple processes. In order to prevent link congestion caused by the misalignment of communication stages of different data blocks, synchronization operations between communication processes of different data blocks need to be performed after the communication of each stage is completed.

[0065] The scatter-reduce operation is as follows: a single data block is further divided according to the number of nodes, that is, it is divided into n sub-blocks. During communication, the nodes in the communication group are constructed into a ring, and each node sends the corresponding data to the next node in the communication group. In stage i, there are communication groups, each of which has p i nodes, each node communicates each time Continuous data sub-blocks, a total of (p i -1) times, create an auxiliary variable array For the rth data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k], which ensures that different data blocks communicate in parallel on different links, and the synchronization process of each data block is independent of each other. After each node receives a data sub-block, it sums it with the corresponding position data sub-block.

[0066] Each time a scatter-reduce operation is completed, each node collects a specific location data sub-block of all nodes in the communication group. As the number of stages i increases, the range of nodes collected by each node's corresponding location data sub-block gradually increases. After all scatter-reduce operations in the first k stages are completed, each node has a sub-block that aggregates the data of all nodes.

[0067] The all-gather operation is as follows: the complete data sub-blocks collected in the scatter-reduce phase are delivered to each node. During communication, the nodes in the communication group are constructed into a ring, and each node sends the corresponding data to the next node in the communication group. communication groups, each of which has p i nodes, each node communicates each time Continuous data sub-blocks, a total of (p i -1) times, create an auxiliary variable array For the rth data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k]. After each node receives a data sub-block, it directly replaces the data sub-block at the corresponding position.

[0068] Each time an all-gather operation is completed, each node summarizes the complete data sub-blocks of the corresponding positions of all nodes in the communication group. As the number of stages i increases, the number of complete data sub-blocks summarized by each node increases. After all all-gather operations in the last k stages are completed, each node summarizes all complete data sub-blocks, and finally each node has a complete data block that aggregates the data of all nodes.

[0069] 1. Basic system settings and underlying physical topology

[0070] This example is carried out on a single server with 8 2080Ti graphics cards in total. Each GPU has 11G video memory. In this example, each GPU is used as a node. The torch.distributed distributed communication interface under the Pytorch framework in Python is used to start a process on each GPU. The broadcast communication interface function in torch.distributed is used to implement point-to-point communication between GPU nodes. Since the algorithm does not affect the model update process in distributed learning, for simplicity, the parameters that need to be synchronized are directly generated through the torch.rand interface.

[0071] The physical topology of the GPU in the server is as follows: Figure 2 As shown. There are three types of links between GPUs in a single machine. The first link is communication across a single PCIe Bridge (Switch), that is, two GPU cards are mounted on the same PCIe Bridge, which is equivalent to GPU0 to GPU1 in the figure; the second link is communication across multiple PCIe Bridges (Switches), which is equivalent to GPU0 to GPU2 in the figure; the third link is cross-NUMAnode GPU communication through QPI / UPI, where a NUMA node consists of a CPU, and this link is equivalent to GPU0 to GPU4 in the figure. In order to ensure efficient communication of GPUs in a single machine, in practice there is no significant difference in the bandwidth of these three different links, and they can basically be regarded as isomorphic networks. Therefore, this embodiment constructs bandwidth heterogeneity of different links through software settings to simulate heterogeneous bandwidth environments commonly seen in communications between servers or across data centers.

[0072] GPUs 0, 1, 2, 3 and 4, 5, 6, 7 are mounted on different CPUs respectively. This embodiment simulates the bandwidth bottleneck of communication between GPU subclusters mounted on different CPUs by limiting the bandwidth of the third link by software. This situation can also simulate the bandwidth bottleneck between server clusters connected to different switches.

[0073] 2. Specific methods of node decomposition

[0074] Let the bandwidth capacities of the three links introduced in the previous section be W1, W2, and W3 respectively. When W3 becomes a communication bottleneck (i.e., W3 is significantly smaller than W1 and W2), the amount of communication data on the link corresponding to W3 needs to be as small as possible. Take the maximum value n, (p i -1) takes the minimum value of 1, the amount of communication data in this stage is the smallest, so the communication in the last stage should be carried out on the link corresponding to W3 through packet decomposition. i =2. For a single machine with 8 GPU nodes, the optimal decomposition is 4×2 or 2×2×2. The 2×2×2 decomposition can achieve higher parallelism and better acceleration effect through data block partitioning in the method proposed by the present invention. Therefore, this embodiment uses the 2×2×2 decomposition as the specific decomposition in the method implementation.

[0075] 3. Specific methods of data segmentation

[0076] In the 2×2×2 decomposition form, k is 3, and the data to be synchronized is divided into 3 blocks. In this embodiment, the bandwidth of the GPU communication link across CPUs in the server is limited, and limit_coe is used to represent the degree of bandwidth limitation. The bandwidth after limitation is 1 / 2 of the original bandwidth. That is According to step S2, it is calculated that The size ratio of the three data blocks is 1+limit_coe:1:1. The original data block can be divided into blocks according to this ratio.

[0077] 4. Parameter synchronization specific process

[0078] The overall process of the parameter synchronization method of 2×2×2 decomposition in this embodiment is as follows: Figure 3 As shown in the figure, synchronization is divided into three stages, and three groups of data are communicated in parallel in each stage. In this embodiment, distance_list is [1,2,4], and the solid line represents the data block x i1 The communication link, the dotted line represents the data block x i2 The communication link is the dotted line, which represents the data block x i3The communication process of these three data blocks is independent of each other. In order to prevent link congestion caused by the misalignment of the communication stages of different data blocks, it is necessary to perform synchronization operations between the communication processes of different data blocks after the communication of each stage is completed.

[0079] For the specific process of parameter synchronization of a single data block, take data block x i1 For example, the specific processes of scatter-reduce and all-gather in parameter synchronization are as follows: Figure 4 and Figure 5 As shown. In the scatter-reduce stage i, the node p whose node index difference is distance_list[i] i Each group communicates with each other. After each communication phase, the data sub-block at a specific position in the node collects the data of the data sub-block at the corresponding position of all nodes in the communication group. After all scatter-reduce phases are completed, each node has a data sub-block that aggregates the data of all nodes. Figure 5 As shown, the node p whose node index difference is distance_list[i] in the penultimate i-th stage of all-gather i Each group communicates with each other. After each communication phase is completed, the node summarizes the complete data sub-blocks at the corresponding positions of all nodes in the communication group. After all all-gather operations are completed, each node summarizes all complete data sub-blocks, and finally each node has a complete data block that aggregates the data of all nodes.

[0080] For the second and third data blocks, the synchronization process is basically the same as the first data block, except that the difference in the node index in the communication group between the scatter-reduce stage i and the i-th last stage of all-gather is distance_list[(i+r)%k](r=2,3), which corresponds to the links in the scatter-reduce stage (i+r)%k of the first data block and the (i+r)%k-th last stage of all-gather, respectively.

[0081] 5. Numerical Verification Experiment

[0082] The present invention verifies the effect of the proposed parameter synchronization method through numerical experiments, and the numerical experiments are completed by Python. The present invention designs experiments from the following aspects:

[0083] Example 1: This experiment compares the Mesh AllReduce algorithm, the Ring AllReduce algorithm, the BlueConnect algorithm, and the parameter synchronization method Parallel-BlueConnect of the present invention (hereinafter referred to as Mesh, Ring, Blue, and P-Blue), where both the Blue and P-Blue algorithms use a 2×2×2 decomposition corresponding to the physical topology within a single machine. The experiment generates random parameters to be synchronized, uses the above four methods to synchronize parameters, measures the time consumed by the entire synchronization process, and draws a curve graph as shown in the figure. Figure 6 shown.

[0084] Figure 6 The horizontal axis is the data size of a single node, ranging from 1KB to 1GB, and the vertical axis is the global synchronization time. Both the horizontal axis and the vertical axis are expressed by exponential coordinate axes. It can be seen from the figure that when the communication data volume has a certain scale, the synchronization time of Ring and Blue is basically the same, and the synchronization time of Mesh and P-Blue is basically the same and less than the synchronization time of Ring and Blue. This is consistent with the theoretical results. Under the condition of a homogeneous network, Ring and Blue have the same communication data volume, the same number of communication links, and similar communication link utilization. Mesh and P-Blue have higher link utilization than Ring and Blue, higher equivalent communication bandwidth, and shorter global synchronization time. When the communication data volume is small (<1MB), the global synchronization time is basically consistent with the delay term of the algorithm. Due to the additional data segmentation operation and the synchronization operation between different data block groups, the method of the present invention brings a relatively large time overhead when the communication data volume is small. The global synchronization time is similar to Ring, but in this case, the global synchronization time is at the millisecond level, which basically does not affect the overall training process.

[0085] Example 2: This experiment constructed a bandwidth heterogeneous network environment through software settings, and compared the Mesh AllReduce algorithm, Ring AllReduce algorithm, BlueConnect algorithm and the method Parallel-BlueConnect of the present invention (hereinafter referred to as Mesh, Ring, Blue and P-Blue) in this scenario. This experiment limits the bandwidth of the GPU communication link across CPUs in the server, and uses limit_coe to represent the degree of bandwidth limitation. The bandwidth after limitation is 1 / (1+limit_coe) of the original. In software implementation, the bandwidth limitation of a specific link is achieved by sending additional invalid data. The data volume of a single node is 100MB. The global synchronization time of each algorithm in the bandwidth heterogeneous scenario is as follows: Figure 7 shown.

[0086] Depend on Figure 7It can be seen that Ring and Mesh are greatly affected by heterogeneous networks. The bandwidth limitation of local links will greatly increase the global synchronization time. The synchronization time of the Ring algorithm in a heterogeneous network environment increases by nearly limit_coe times, and the communication time basically depends on the link with the smallest bandwidth. However, after node sequence mapping and group decomposition, Blue and P-Blue make the data transmission volume on the communication-restricted link as small as possible, thereby weakening the impact of the communication-restricted link on the synchronization time. From the figure, it can be seen that the slope of the growth curve of the global synchronization time of Blue and P-Blue with limit coe is small, which better adapts to the heterogeneous bandwidth environment. Among them, compared with the Blue algorithm, the synchronization time of P-Blue is significantly smaller when limit_coe is small, that is, when the network bandwidth heterogeneity is small, but as the network bandwidth heterogeneity increases, it gradually becomes the same as the Blue algorithm. This is because if the bandwidth of the bandwidth-restricted link is significantly smaller than that of other links, the global synchronization time is mainly dominated by the communication time on the link, and the parallelization of other links cannot accelerate the synchronization process.

[0087] From the above examples, it can be seen that the method Parallel-BlueConnect of the present invention performs significantly better than the Mesh AllReduce algorithm and the Ring AllReduce algorithm in heterogeneous networks, and is better than the present method in homogeneous and low heterogeneous environments. The present method can flexibly decompose the underlying physical topology, and its multi-link parallel characteristics enable the present method to maximize the use of different types of links, achieving optimal performance in both homogeneous and heterogeneous network environments.

[0088] Corresponding to the aforementioned embodiment of an efficient parameter synchronization method based on the ALLREDUCE architecture, the present invention also provides an embodiment of an efficient parameter synchronization device based on the ALLREDUCE architecture.

[0089] See also Figure 8 An embodiment of the present invention provides an efficient parameter synchronization device based on the ALLREDUCE architecture, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement an efficient parameter synchronization method based on the ALLREDUCE architecture in the above embodiment.

[0090] The embodiment of the efficient parameter synchronization device based on the ALLREDUCE architecture provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 8 As shown, it is a hardware structure diagram of any device with data processing capability where an efficient parameter synchronization device based on ALLREDUCE architecture provided by the present invention is located. Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0091] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0092] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0093] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, an efficient parameter synchronization method based on the ALLREDUCE architecture in the above embodiment is implemented.

[0094] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0095] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the computer program implements the efficient parameter synchronization method based on the ALLREDUCE architecture.

[0096] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0097] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. An efficient parameter synchronization method based on AllReduce architecture, characterized in that: The method comprises the following steps: The nodes are grouped and decomposed according to the underlying physical topology of the GPU / NPU cluster system, and the communication volume in the data synchronization phase is calculated based on the node grouping results, so that the link bandwidth after decomposition matches the communication volume in the data synchronization phase; The data to be synchronized is divided into data blocks according to the communication volume in different communication stages after decomposition and the bandwidth size of each link; The data blocks obtained by segmentation are synchronized in parallel. The data synchronization communication modes include scatter-reduce and all-gather. The number of stages of each data synchronization communication mode is the same as the number of node groupings. After the communication of each stage is completed, the synchronization operation between the communication processes of different data blocks is performed. Eventually, each node has a complete data block that aggregates the data of all nodes, completing the parameter synchronization process.

2. According to claim 1, an efficient parameter synchronization method based on AllReduce architecture is characterized in that: The grouping and decomposing nodes according to the underlying physical topology of the GPU / NPU cluster system includes: grouping the nodes in the following integer decomposition form: Where n is the total number of nodes participating in synchronization, k is the number of groups, which is the same as the number of communication phases of scatter-reduce and all-gather in data synchronization, and p i The number of nodes in each communication group in each phase of data synchronization.

3. According to claim 2, an efficient parameter synchronization method based on AllReduce architecture is characterized in that: The communication volume in the data synchronization phase is specifically: The amount of communication data between the scatter-reduce stage i and the penultimate i-th stage of all-gather is The amount of communication data decays as i increases, where m is the size of the data to be synchronized.

4. According to claim 3, an efficient parameter synchronization method based on AllReduce architecture is characterized in that: The decomposition method that makes the decomposed link bandwidth match the communication volume in the data synchronization stage includes: For a multidimensional network system with dimension d, let the size of each dimension be (a1, a2, ... a d ), the network bandwidth corresponding to each dimension is (q1,q2,…q d ), with the number of nodes The decomposition method is to make p i The value corresponds to the size of each dimension of the system, that is, p i =a i (i=1,2……d), k=d.

5. The efficient parameter synchronization method based on AllReduce architecture according to claim 3 is characterized in that: The decomposition method that makes the decomposed link bandwidth match the communication volume in the data synchronization stage includes: For a multidimensional network system with dimension d, let the size of each dimension be (a1, a2, ... a d ), the network bandwidth corresponding to each dimension is (q1,q2,…q d ), with the number of nodes The decomposition method is: Let the communication data volume in stage i be g i , the minimum bandwidth on the communication link in stage i is q j , where the corresponding relationship between i and j is: minimize So as to determine k and the optimal decomposition form The solution method is selected according to the size of the system.

6. The efficient parameter synchronization method based on AllReduce architecture according to claim 1, characterized in that: The specific method of dividing the data to be synchronized is as follows: the first data block adopts the default synchronization mode that matches the data communication volume and link bandwidth at each stage in the synchronization process; the synchronization stage i of the rth data block corresponds to the link of the (i+r-1)%kth stage of the default synchronization mode, and the size of the rth data block is in b i It is expressed as the bandwidth of the link; m is the size of the data to be synchronized, and k is the total number of data blocks after segmentation, which is equal to the number of communication stages.

7. The efficient parameter synchronization method based on AllReduce architecture according to claim 1, characterized in that: The data synchronization is specifically as follows: A 2k-stage synchronization operation is performed on a single data block. The first k stages perform scatter-reduce operations. There are a total of communication groups communicate in parallel on the same type of links, and there are p communication groups in each communication group. i nodes, each communication group communicates in the scatter-reduce mode in Ring Allreduce; the last k stages perform all-gather operations, and there are a total of communication groups communicate in parallel on the same type of links, and there are p communication groups in each communication group. i Each communication group communicates in the all-gather mode in Ring Allreduce, and performs synchronization operations between communication processes of different data blocks after the communication in each stage is completed.

8. The efficient parameter synchronization method based on AllReduce architecture according to claim 1, characterized in that: The scatter-reduce operation is specifically as follows: a single data block is further divided according to the number of nodes, that is, divided into n sub-blocks; during communication, the nodes in the communication group are constructed into a ring, and each node sends corresponding data to the next node in the communication group; In stage i, there are communication groups, each of which has p i nodes, each node communicates each time Continuous data sub-blocks, a total of (p i -1) times, create an auxiliary variable array For the rth data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k]. This ensures that different data blocks communicate in parallel on different links, and the synchronization process of each data block is independent of each other; after each node receives a data sub-block, it sums it with the data sub-block at the corresponding position.

9. The efficient parameter synchronization method based on AllReduce architecture according to claim 1, characterized in that: The all-gather operation is specifically as follows: the complete data sub-blocks collected in the scatter-reduce stage are delivered to each node; during communication, the nodes in the communication group are constructed into a ring, and each node sends the corresponding data to the next node in the communication group; in the penultimate i-th stage, there are communication groups, each of which has p i nodes, each node communicates each time Continuous data sub-blocks, a total of (p i -1) times, create an auxiliary variable array For the rth data block, the index difference between two adjacent nodes in the same communication group is distance_list[(i+r)%k]; after each node receives a data sub-block, it directly replaces the data sub-block at the corresponding position; After all the all-gather operations in the last k stages are completed, each node summarizes all the complete data sub-blocks, and finally each node has a complete data block that aggregates the data of all nodes.

10. An efficient parameter synchronization device based on AllReduce architecture, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, an efficient parameter synchronization method based on the AllReduce architecture as described in any one of claims 1 to 9 is implemented.

Images