Multi-transmit and multi-collect communication method and system based on butterfly network

Abstract

The present invention discloses a multi-transmitter, multi-collector combined communication method and system based on a butterfly network, belonging to the field of distributed computing technology. The method comprises: obtaining a single-round communication startup delay, a single data element calculation time, and a bandwidth of the butterfly network; determining a data scale critical point of a single computing node based on the single-round communication startup delay, the single data element calculation time, and the bandwidth; obtaining a data scale; if the data scale is greater than the data scale critical point, performing a global reduction operation between multiple computing nodes of the butterfly network through a first multi-transmitter, multi-collector communication paradigm; otherwise, performing a global reduction operation between multiple computing nodes of the butterfly network through a second multi-transmitter, multi-collector communication paradigm; and enabling adaptive communication for distributed data transmission scenarios of different scales, thereby solving the problem that existing communication algorithms cannot balance communication delay and bandwidth utilization.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a multi-transmitter and multi-collector combined communication method and system based on a butterfly network. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the continuous expansion of deep learning models and the continued growth of training data, distributed training has become an inevitable choice for large-scale machine learning. In distributed training, multiple computing nodes need to frequently exchange model parameters and gradient information, making the efficiency of collective communication directly impact overall training performance. In particular, in algorithms such as synchronized stochastic gradient descent, each iteration requires a global reduction operation to aggregate gradient information from all computing nodes. Therefore, efficient collective communication is a key factor in improving distributed training performance.

[0004] In large-scale distributed training scenarios, communication overhead can consume a significant portion of the total training time. Furthermore, synchronous stochastic gradient descent requires waiting for all compute nodes to complete gradient calculation and communication before each iteration. Load imbalances between compute nodes and network fluctuations can also lead to reduced training efficiency. As cluster size increases, the complexity of the communication topology increases significantly, placing higher demands on the algorithm's scalability.

[0005] Currently, a variety of global reduction algorithms have been proposed and widely used in the field of distributed collective communication. Among them, the ring-based global reduction algorithm organizes all participating nodes into a logical ring and completes data aggregation in two phases. In the distributed reduction phase, each node divides the data into N parts, and the reduction of the partial data is completed through N-1 rounds of communication. In the global collection phase, the reduction results are broadcast to all nodes through N-1 rounds of communication. Ring-based global reduction algorithms have high bandwidth utilization and are particularly suitable for large-scale data transmission scenarios. However, the number of communication rounds increases linearly with the number of nodes, resulting in significant communication latency.

[0006] The butterfly global reduction algorithm is based on the divide-and-conquer principle and employs a tree-like communication model for data aggregation. During the reduction phase, the algorithm pairs nodes in pairs for reduction, completing data reduction in log(N) rounds. During the broadcast phase, the algorithm broadcasts data using the same tree structure, also requiring log(N) rounds. Compared to the ring-based global reduction algorithm, the butterfly global reduction algorithm has fewer communication rounds and lower latency, but its bandwidth utilization is relatively low, making it more suitable for small data transmission scenarios.

[0007] The 2D-Ring Global Reduction Algorithm organizes nodes into a 2D grid structure. The entire process is performed in three phases: first, a circular global reduction is performed within the rows, then a reduction operation is performed along the columns, and finally, a circular global collection is performed along the rows. This 2D-Ring Global Reduction Algorithm strikes a good balance between communication overhead and is particularly well-suited for grid topologies. However, its implementation is complex and places high demands on the network topology.

[0008] To adapt to heterogeneous network environments, researchers have also proposed a hierarchical global reduction algorithm. This algorithm first utilizes high-speed interconnects to perform intra-node data reduction, then selects an appropriate algorithm based on bandwidth differences to perform inter-node reduction. This hierarchical design allows the algorithm to flexibly adapt to heterogeneous network environments, but its performance is significantly affected by the network topology.

[0009] In summary, although various existing global reduction algorithms have demonstrated their respective advantages in different scenarios, they still have some common limitations:

[0010] 1. It is difficult to balance latency and bandwidth utilization. Although the ring-based global reduction algorithm achieves high bandwidth utilization, its large number of communication rounds leads to large latency. While the butterfly-based global reduction algorithm has the advantage of low latency, its bandwidth utilization is not high.

[0011] 2. There are scalability issues. The performance of most algorithms will drop significantly as the number of nodes increases, and it is difficult to maintain stable performance in a complex network environment.

[0012] 3. The existing algorithms may cause imbalance in computing and communication loads in terms of data segmentation and transmission, affecting the overall training efficiency.

[0013] In addition, these algorithms often adopt a single communication mode and fail to fully utilize the parallel multi-transmitter and multi-receiver communication capabilities of modern network hardware; however, global reduction operations have strict global dependency properties, which poses a major challenge to achieving true multi-transmitter and multi-receiver communication.

[0014] Specifically, global reduction requires that all data be included in the calculation, and data can only be split according to matrix dimensions (e.g., rows). During the merging process, only vectors from the same number of rows are reduced; the rest are combined. More importantly, the final result of the reduction of all data must be obtained, and this result must be consistent across all nodes. This global dependency makes it difficult to truly achieve multi-transmit and multi-receive. Summary of the Invention

[0015] In order to address the deficiencies of the prior art, the present invention provides a butterfly network-based multi-transmitter and multi-receiver combined communication method, system, electronic device, computer-readable storage medium, and computer program product. Through an innovative multi-transmitter and multi-receiver communication mode, it achieves the goal of reducing communication delay while improving bandwidth utilization.

[0016] In a first aspect, the present invention provides a multi-transmitter and multi-collector communication method based on a butterfly network;

[0017] A multi-transmitter and multi-collector combined communication method based on a butterfly network, comprising:

[0018] Obtain the single-round communication startup delay, single data element calculation time and bandwidth of the butterfly network. Based on the single-round communication startup delay, single data element calculation time and bandwidth, determine the data scale critical point of a single computing node.

[0019] Obtain the data scale. If the data scale is greater than the data scale critical point, a global reduction operation is performed between multiple computing nodes of the butterfly network through a first multi-transmitter and multi-receiver communication paradigm. Otherwise, a global reduction operation is performed between multiple computing nodes of the butterfly network through a second multi-transmitter and multi-receiver communication paradigm.

[0020] In some embodiments, performing a global reduction operation between multiple computing nodes of the butterfly network using a first multi-transmitter and multi-receiver communication paradigm includes:

[0021] Based on the matrix dimension characteristics, the data to be reduced on each computing node is divided into multiple sub-data blocks;

[0022] The distributed reduction operation is performed between multiple computing nodes through multiple rounds of progressive reduction communication to achieve local reduction between each computing node; the global collection operation is performed in the opposite communication order of the local reduction operation through dimension alignment to achieve global reduction for each computing node;

[0023] In each round of progressive protocol communication, the sub-data blocks are grouped using the number of communication rounds and data dimension characteristics, and data is transmitted through a multi-transmitter and multi-receiver communication mode.

[0024] In some implementations, the amount of data transmitted in each round of progressive protocol communication is expressed as:

[0025] ;

[0026] Where, Indicates the size of the data to be reduced on each computing node, Indicates the number of computing nodes, Indicates the number of communication rounds in progressive protocol communication.

[0027] In some embodiments, performing a global reduction operation between multiple computing nodes of the butterfly network using the second multi-transmitter and multi-receiver communication paradigm includes:

[0028] Based on the number of computing nodes, the data to be reduced on each computing node is divided into multiple sub-data blocks;

[0029] Distributed protocol operations are performed between multiple computing nodes through a multi-transmitter and multi-receiver communication mode.

[0030] In some implementations, the time overhead of the first multi-transmit and multi-receive communication paradigm is:

[0031] ;

[0032] Where, Indicates the number of computing nodes, Indicates bandwidth, Indicates the delay in starting a single round of communication. Indicates the size of the data to be reduced on each computing node, Indicates the time taken to calculate a single data element.

[0033] In some implementations, the data scale critical point of the single computing node is expressed as:

[0034] ;

[0035] Where, Indicates the critical point of data scale of a single computing node, N Indicates the number of computing nodes, Indicates bandwidth, Indicates the time taken to calculate a single data element. Indicates the delay in starting a single round of communication.

[0036] In a second aspect, the present invention provides a multi-transmitter and multi-collector combined communication system based on a butterfly network;

[0037] A multi-transmitter and multi-collector combined communication system based on a butterfly network, comprising:

[0038] The adaptive mode selection module is configured to: obtain a single-round communication startup delay, a single data element calculation time, and a bandwidth of the butterfly network, and determine a data scale critical point of a single computing node based on the single-round communication startup delay, the single data element calculation time, and the bandwidth;

[0039] The global reduction module is configured to: obtain the data scale; if the data scale is greater than the data scale critical point, perform the global reduction operation between the multiple computing nodes of the butterfly network through the first multi-transmitter and multi-receiver communication paradigm; otherwise, perform the global reduction operation between the multiple computing nodes of the butterfly network through the second multi-transmitter and multi-receiver communication paradigm.

[0040] In a third aspect, the present invention provides an electronic device;

[0041] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the multi-transmitter and multi-collector combined communication method based on a butterfly network.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium;

[0043] A computer-readable storage medium stores a computer program / instruction, which, when executed by a processor, implements the steps of the multi-transmitter and multi-collector combined communication method based on a butterfly network.

[0044] In a fifth aspect, the present invention provides a computer program product;

[0045] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the multi-transmitter and multi-collector combined communication method based on a butterfly network.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The technical solution provided by the present invention is based on a butterfly network to design a multi-transmit and multi-receive paradigm that adapts to different numbers of nodes, providing the optimal multi-transmit and multi-receive expression for each number of computing nodes.

[0048] 2. The technical solution provided by the present invention designs an adaptive communication strategy based on data scale. It provides two complementary communication paradigms for distributed data transmission scenarios of different scales, one is a multi-transmitter and multi-receiver communication paradigm suitable for large-scale data transmission, and the other is a simplified solution suitable for small-scale data transmission. In actual applications, it can automatically select a communication solution based on the data scale, and continuously evaluate and adjust during system operation to ensure that the optimal communication strategy is always adopted. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1A schematic diagram of a flow chart of a multi-transmitter and multi-collector communication method based on a butterfly network according to an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the connection of computing nodes in an 8-node butterfly network provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram illustrating the connection of computing nodes in a 16-node butterfly network according to an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the communication process of the first multi-transmitter and multi-receiver communication paradigm provided by an embodiment of the present invention applied to an 8-node system;

[0054] Figure 5 A schematic diagram of the communication process of applying the second multi-transmitter and multi-receiver communication paradigm provided by an embodiment of the present invention to an 8-node system. DETAILED DESCRIPTION

[0055] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0056] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0058] Explanation of terms:

[0059] Distributed training: Distributed training is a method that distributes the training of large-scale machine learning models across multiple computing nodes for parallel execution. During training, each node calculates gradients based on local data and exchanges updates via network communication, collaboratively optimizing the model. This method can effectively improve training efficiency and support the training of larger models.

[0060] Collective communication: In a distributed computing environment, collective communication refers to communication operations involving multiple computing nodes. Its basic forms include broadcast, protocol, full protocol, collection, etc. These operations are the basis for building an efficient distributed system.

[0061] Global reduction operations: A global reduction operation is a fundamental collective communication operation that reduces data distributed across multiple compute nodes (such as summing or maximizing values) and distributes the final result to all participating nodes. In distributed machine learning, full reduction operations are primarily used for gradient aggregation and are a key mechanism for ensuring model consistency.

[0062] Communication latency: In distributed systems, communication latency refers to the time required to send data from a source node to a destination node. It typically consists of two components: startup latency, which is the fixed overhead required to establish a communication connection, and transmission latency, which is related to the amount of data. Communication latency is a significant factor affecting the performance of distributed systems.

[0063] Bandwidth utilization: Bandwidth utilization indicates the ratio of the actual network bandwidth used during data transmission to the theoretical bandwidth. A higher bandwidth utilization means that network resources are more fully utilized, which helps improve the overall performance of the system.

[0064] Data reduction: Data reduction is the process of combining data from multiple nodes into a single result using specific operations (such as summation or averaging). In distributed machine learning, gradient reduction is a necessary step to ensure model convergence.

[0065] Distributed reduction: Distributed reduction is the first stage of the global reduction operation. Its function is to store the reduction results in a distributed manner on each node participating in the calculation. Each node eventually obtains part of the reduction results to prepare for subsequent global data reconstruction.

[0066] Global collection: Global collection is the second stage of the global reduction operation. Through this operation, each node shares its own partial reduction results with all other nodes, ultimately achieving global consistency of data.

[0067] Computing node: The basic computing unit in a distributed system, equipped with independent processor, memory and other computing resources, capable of executing assigned computing tasks and communicating with other nodes.

[0068] Example 1

[0069] Existing global reduction algorithms cannot take into account both communication delay and bandwidth utilization at the same time, which affects the training efficiency of distributed training. Therefore, the present invention provides a multi-transmit and multi-collect communication method based on a butterfly network, which reduces communication delay while improving bandwidth utilization.

[0070] Next, combine Figure 1-Figure 5 , a multi-transmitter and multi-collector communication method based on a butterfly network disclosed in this embodiment is described in detail. The butterfly network includes multiple computing nodes, and the multi-transmitter and multi-collector communication method based on the butterfly network includes:

[0071] S1. Obtain the single-round communication startup delay, single data element calculation time, and bandwidth of the butterfly network. Determine the data scale critical point of a single computing node based on the single-round communication startup delay, single data element calculation time, and bandwidth.

[0072] In this embodiment, the critical point of the data scale of a single computing node is expressed as:

[0073] ;

[0074] Where, Indicates the critical point of data scale of a single computing node, N Indicates the number of computing nodes, Indicates bandwidth, Indicates the time taken to calculate a single data element. Indicates the delay in starting a single round of communication.

[0075] Based on this, the critical point of data scale is calculated by combining communication delay, bandwidth overhead and computing overhead, providing a theoretical basis for the selection of subsequent communication solutions.

[0076] In this embodiment, a butterfly network is used as the basic communication topology, combined with Figure 3-Figure 4 , further introduce the butterfly network.

[0077] The 4-node butterfly network adopts a square topology, including 4 computing nodes (numbered 1-4). Each computing node establishes a direct connection with its two adjacent computing nodes, forming a square structure as a whole, ensuring that any two computing nodes can complete communication through at most one intermediate node; the communication path is symmetrical to ensure load balancing. In each round of communication, each computing node performs 2 sends and 2 receives in parallel.

[0078] Combine Figure 3 , the 8-node butterfly network configuration forms a three-dimensional cube projection structure, with nodes numbered 1-8; additional connections are established between peripheral computing nodes according to the butterfly network rules, with one vertex connecting to three vertices; the network structure has good symmetry, ensuring the uniformity of communication distance; under this configuration, the global reduction requires In each round of communication, each computing node performs 3 sends and 3 receives in parallel.

[0079] Combine Figure 4 , a 16-node butterfly network configuration forms a complex ring interconnection structure, with nodes numbered 1-16; computing nodes establish cross-connections on the ring basis, with one vertex connecting to four vertices; each computing node establishes connections with adjacent computing nodes and corresponding cross-ring nodes, forming multiple parallel communication paths to support efficient data exchange; completing the global protocol under this configuration requires In each round of communication, each computing node performs 4 sends and 4 receives in parallel.

[0080] The number of connections between nodes in a butterfly network grows logarithmically with the scale of the system, avoiding the hardware overhead brought by full connectivity while maintaining a low network diameter. In terms of communication efficiency, the structure supports multiple parallel communication paths and achieves good load balancing through evenly distributed communication distances. In addition, the self-similarity of the topology makes it easy to expand to larger-scale systems, and the performance only changes logarithmically with the increase in the number of nodes, reflecting excellent scalability. Although the network topology presents different geometric structures under different scale configurations, they all follow the same butterfly network construction principles. This unified design pattern ensures that the algorithm can maintain stable performance in systems of different scales. Through the support of this network topology, combined with the subsequent multi-transmitter and multi-receiver communication mechanism, this embodiment can achieve efficient global data reduction and provide strong communication infrastructure support for distributed machine learning systems.

[0081] S2. Obtain the data scale. If the data scale is greater than the data scale critical point, execute S3; otherwise, execute S4.

[0082] Based on the data scale critical point S*, this embodiment implements a dynamic adaptive strategy: when the data scale S is greater than the data scale critical point S*, the first multi-transmitter and multi-receiver communication paradigm for large-scale data transmission is selected, which can achieve better bandwidth utilization; when S is less than or equal to the data scale critical point S*, the second multi-transmitter and multi-receiver communication paradigm for small-scale data transmission is selected to obtain lower communication overhead.

[0083] Here, data scale refers to the data size corresponding to the parameter matrix M being divided into n parts.

[0084] Furthermore, this adaptive mechanism can continuously evaluate and adjust during operation to ensure that the optimal communication strategy is always adopted.

[0085] S3. Multiple computing nodes in the butterfly network perform global reduction operations using the first multi-transmit and multi-receive communication paradigm. The specific process is as follows:

[0086] S301: Obtain the data set to be reduced and perform initial partitioning.

[0087] In this embodiment, the data set to be reduced is a parameter matrix in a deep learning model.

[0088] Specifically, based on the two-dimensional characteristics of the parameter matrix, a tensor parallel method is used for initial partitioning. For example, for a parameter matrix M with a dimension of p rows and q columns, it can be evenly divided into n parts along the row direction or column direction and assigned to n computing nodes respectively; when the parameter matrix M is divided along the row direction, each computing node obtains a sub-matrix of (p / n) rows and q columns; when the parameter matrix M is divided along the column direction, each computing node obtains a sub-matrix of p rows and (q / n) columns.

[0089] Furthermore, the choice of partitioning direction needs to consider the specific computing mode, that is, whether to choose row partitioning or column partitioning should be determined based on the specific operation type of matrix calculation (such as matrix-vector multiplication, matrix-matrix multiplication): when the calculation process requires frequent access to complete row data, column partitioning may be better; conversely, when the calculation depends on complete column data, row partitioning is more appropriate.

[0090] After each compute node completes its local gradient calculation, it needs to perform a global reduction operation (such as summing or averaging) to unify the gradient values of all nodes. To address packet loss during large-scale data transmission while ensuring pipeline efficiency for computation and transmission, this embodiment determines the data segmentation strategy before training begins.

[0091] Specifically, for a butterfly network with n computing nodes, the parameter matrix M is evenly divided into n parts, each of which is M / n, or S. The data block of size S on each computing node will be split step by step during the communication process and finally divided into equal sub-data blocks, each sub-data block is This pre-planned partitioning scheme enables each computing node to transmit and reduce these sub-data blocks one by one according to the determined steps, while ensuring data integrity and achieving efficient pipeline operation of computing and communication.

[0092] S302, perform distributed reduction operations among multiple computing nodes through multiple rounds of progressive reduction communication to achieve local reduction among each computing node. Round progressive protocol communication, to achieve local data protocol, in the i-th round protocol communication (i from 1 to ), the specific process of progressive protocol communication is as follows:

[0093] (1) Each computing node groups the currently held data into to ensure that the reducible data is in the same dimensional position.

[0094] Here, when i= When , the data size after segmentation is .

[0095] (2) Adoption hair The communication mode of receiving and receiving, each computing node sends Computing nodes (target nodes) send Size data, at the same time, from The data is received from each computing node (source node) and the received data is ensured to be of the same dimension as the local data for reduction.

[0096] (3) After each round of communication is completed, the computing node merges the received data of the same dimension through reduction operations (such as summation, maximum value, etc.).

[0097] Finally passed Each computing node in the round holds A locally specified The size of the data fragment.

[0098] Next, we will take the distributed training of the attention layer in the Llama model and the tensor parallelization of 8 computing nodes as an example to illustrate the specific process of decentralized reduction.

[0099] The size of the gradient data of the parameter matrix in the attention layer is 3072×3072. Under the tensor parallel strategy, the parameter matrix is divided into 8 computing nodes by row, so each computing node initially holds a gradient slice of size 384×3072.

[0100] The decentralized protocol passed 3 rounds ( ) Progressive communication is completed. In the first round of communication, each computing node divides its own 384×3072 gradient slice into 6 parts ( ), meaning each gradient slice is 384×512 in size. Compute nodes communicate using a butterfly network pairing relationship: Compute node 1 simultaneously sends gradient slices to compute nodes 2, 3, and 5, and receives gradient slices of the same size from these compute nodes. Similarly, compute node 4 sends and receives gradient slices from compute nodes 3, 2, and 8. After receiving a gradient slice, each compute node immediately performs a sum-reduce on the gradient slices at the same location.

[0101] In the second round of communication, the computing node divides the gradient shards after the first round of reduction into two parts, which is one-twelfth of the initial gradient shards ( ), each piece is 384 × 256 in size. Similarly, after many-to-many communication, the gradient slices are received and the gradient summation of the corresponding positions is performed.

[0102] In the third round of communication, the computing node divides the gradient shards after the second round of reduction into 24 parts ( ), each size is 384×128, and the final local reduction operation is completed.

[0103] After the above three rounds of communication, each computing node finally holds 3 ( ) The fragments of the locally reduced gradient shards are 384×128 in size. These fragments contain the local reduction results of the original gradient in different dimensions, preparing for the subsequent global collection operation. This multi-transmitter and multi-receiver communication scheme not only achieves a balanced distribution of the communication load, but also effectively supports the pipeline operation of communication and computing by gradually reducing the size of the data shards.

[0104] S303 , performing a global gather operation in a dimension-aligned manner in a communication order opposite to that of the local reduce operation, thereby achieving global reduction for each computing node.

[0105] Specifically, from The communication operation is executed in reverse order from the beginning of the round to the first round. In each round of communication, each computing node sends the local reduction result it holds to the corresponding target nodes, and from A source node receives data; after receiving the data, the computing node locates the dimension position according to the position offset of the data it holds in the original tensor and assembles it, gradually reconstructing the complete reduction result.

[0106] For example, the rank value of each computing node is pre-assigned. When performing a global reduction operation, each computing node has a receiving buffer equal to the size of the original tensor. When receiving data from other computing nodes, the position offset is directly calculated based on the rank value of the computing node and the data is placed in the corresponding position. The position offset is expressed as:

[0107] offset = rank * chunk_size;

[0108] In the formula, chunk_size represents the data size.

[0109] For example, the data sent by the computing node with a rank value of 0 is stored in [0: chunk_size], the data sent by the computing node with a rank value of 1 is stored in [chunk_size: 2*chunk_size], the data sent by the computing node with a rank value of 2 is stored in [2chunk_size: 3chunk_size], and so on.

[0110] This dimensional alignment ensures the correctness of the data during the reconstruction process, ultimately allowing all nodes to obtain completely consistent global reduction results. This two-way communication mode ensures both the integrity of data transmission and the consistency of reduction results.

[0111] In some embodiments, the time overhead of the first multi-transmit and multi-receive communication paradigm is evaluated and expressed as:

[0112] ;

[0113] Where, Indicates the number of computing nodes, Indicates bandwidth, Indicates the delay in starting a single round of communication. Represents the data to be reduced on each computing node, Indicates the time taken to calculate a single data element.

[0114] Based on this, when n=8, the sum of communication delays is 3 / 4 of that of the 2D-ring global reduction algorithm, and the sum of bandwidth consumption and computational consumption are both 1 / 3 of that of the 2D-ring global reduction algorithm.

[0115] S4. Multiple computing nodes in the butterfly network perform global reduction operations through the second multi-transmitter and multi-receiver communication paradigm. Specifically, it includes:

[0116] S401, the parameter matrix M is evenly divided into n parts, each part is M / n, that is, S, and is distributed to n computing nodes respectively. Each computing node divides the obtained data into sub-data blocks, the size of each sub-data block is .

[0117] This coarse-grained data partitioning method in this embodiment reduces data management overhead and is suitable for small-scale data transmission scenarios.

[0118] S402, use hair The communication mode of receiving and receiving, each computing node sends The computing nodes (target nodes) send S / (log2n) copies of data, and at the same time, The data is received from each computing node (source node) and the received data is ensured to be of the same dimension as the local data for reduction.

[0119] S403, after each round of communication is completed, the computing node merges the data of the same dimension through the reduction operation (such as summation, maximum value, etc.), and finally holds A locally specified The complete global protocol is reconstructed by data fragments of different sizes, that is, all the data, and all nodes obtain completely consistent results.

[0120] For example, taking an 8-node butterfly network as an example, the communication process of the second multi-transmitter and multi-receiver communication paradigm is further described. The specific process is as follows:

[0121] First, the gradient matrix to be reduced is evenly divided into , that is, 3 initial data shards (where n=8, Initially, each compute node is allocated three initial data shards. This coarse-grained data partitioning simplifies communication operations.

[0122] In the first round of communication, each computing node executes a 3-transmit-3-receive communication mode. Specifically, each computing node divides the initial data fragment it currently holds into 1 / 3 sized data blocks for transmission. Each computing node simultaneously sends data to three target nodes and receives data from three source nodes.

[0123] After completing the first round of data exchange, each computing node will hold 3 data fragments of 1 / 3 size that have undergone the first local reduction.

[0124] In the second round of communication, the computing nodes perform the next data exchange and execute the 3-transmit-3-receive communication mode again on the 1 / 3 data obtained in the previous round after local reduction; each computing node performs a second local reduction on the received data.

[0125] After the communication is completed, each computing node obtains three 1 / 3 data fragments that have undergone secondary reduction.

[0126] In the third round of communication, the final local reduction is completed through the last round of 3 sends and 3 receives. At this time, each node holds 3 fully reduced 1 / 3 data fragments, which together constitute the complete reduction result of the gradient matrix, and finally the complete gradient matrix is reconstructed on all computing nodes.

[0127] In some embodiments, the time overhead of the second multi-transmit and multi-receive communication paradigm is evaluated and expressed as:

[0128] ;

[0129] Where, Indicates the number of computing nodes, Indicates the delay in starting a single round of communication. Represents the data to be reduced on each computing node, Indicates bandwidth, Indicates the time taken to calculate a single data element.

[0130] Based on this, when n=8, compared with the 2D-Torus algorithm, the sum of communication delays is 3 / 8 of that of the 2D-Torus algorithm, and the sum of bandwidth consumption and computational time are both 8 / 14 of that of the 2D-Torus algorithm, which is 7 / 12 of the first multi-transmitter and multi-receiver communication paradigm.

[0131] Example 2

[0132] This embodiment discloses a multi-transmitter and multi-collector combined communication system based on a butterfly network, including:

[0133] The adaptive mode selection module is configured to: obtain a single-round communication startup delay, a single data element calculation time, and a bandwidth of the butterfly network, and determine a data scale critical point of a single computing node based on the single-round communication startup delay, the single data element calculation time, and the bandwidth;

[0134] The global reduction module is configured to: obtain the data scale; if the data scale is greater than the data scale critical point, perform the global reduction operation between the multiple computing nodes of the butterfly network through the first multi-transmitter and multi-receiver communication paradigm; otherwise, perform the global reduction operation between the multiple computing nodes of the butterfly network through the second multi-transmitter and multi-receiver communication paradigm.

[0135] It should be noted that the adaptive mode selection module and global reduction module described above correspond to the steps in Example 1. The examples and application scenarios implemented by these modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0136] Example 3

[0137] A third embodiment of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the multi-transmitter and multi-collector combined communication method based on the butterfly network are completed.

[0138] Example 4

[0139] A fourth embodiment of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the multi-transmitter and multi-collector combined communication method based on the butterfly network are completed.

[0140] Example 5

[0141] A fifth embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned butterfly network-based multi-transmitter and multi-collector combined communication method.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0145] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0146] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multi-transmitter and multi-collector communication method based on a butterfly network, characterized in that: include: Obtain the single-round communication startup delay, single data element calculation time and bandwidth of the butterfly network. Based on the single-round communication startup delay, single data element calculation time and bandwidth, determine the data scale critical point of a single computing node. Obtaining a data scale; if the data scale is greater than a data scale critical point, performing a global reduction operation among the multiple computing nodes of the butterfly network using a first multi-transmitter and multi-receiver communication paradigm; otherwise, performing a global reduction operation among the multiple computing nodes of the butterfly network using a second multi-transmitter and multi-receiver communication paradigm; The execution of a global reduction operation between the plurality of computing nodes of the butterfly network through the first multi-transmit multi-receive communication paradigm includes: Based on the matrix dimension characteristics, the data to be reduced on each computing node is divided into multiple sub-data blocks; The distributed reduction operation is performed between multiple computing nodes through multiple rounds of progressive reduction communication to achieve local reduction between each computing node; the global collection operation is performed in the opposite communication order of the local reduction operation through dimension alignment to achieve global reduction for each computing node; In each round of progressive protocol communication, the number of communication rounds and matrix dimension characteristics are used to group sub-data blocks, and data is transmitted through a multi-transmitter and multi-receiver communication mode; The execution of a global reduction operation between the plurality of computing nodes of the butterfly network through the second multi-transmit multi-receive communication paradigm includes: Based on the number of computing nodes, the data to be reduced on each computing node is divided into multiple sub-data blocks; Distributed protocol operations are performed between multiple computing nodes through a multi-transmitter and multi-receiver communication mode.

2. The multi-transmit and multi-collect communication method based on butterfly network according to claim 1, characterized in that: The amount of data transmitted in each round of progressive protocol communication is expressed as: ； Where, Indicates the size of the data to be reduced on each computing node, Indicates the number of computing nodes, Indicates the number of communication rounds in progressive protocol communication.

3. The multi-transmit and multi-collect communication method based on butterfly network according to claim 1, characterized in that: The time overhead of the first multi-transmit and multi-receive communication paradigm is: ； Where, Indicates the number of computing nodes, Indicates bandwidth, Indicates the delay in starting a single round of communication. Indicates the size of the data to be reduced on each computing node, Indicates the time taken to calculate a single data element.

4. The multi-transmit and multi-collect communication method based on butterfly network according to claim 1, characterized in that: The data scale critical point of a single computing node is expressed as: ； Where, Indicates the critical point of data scale of a single computing node, N Indicates the number of computing nodes, Indicates bandwidth, Indicates the time taken to calculate a single data element. Indicates the delay in starting a single round of communication.

5. The multi-transmitter and multi-collector combined communication system based on butterfly network is characterized by: include: The adaptive mode selection module is configured to: obtain a single-round communication startup delay, a single data element calculation time, and a bandwidth of the butterfly network, and determine a data scale critical point of a single computing node based on the single-round communication startup delay, the single data element calculation time, and the bandwidth; The global reduction module is configured to: obtain a data size, and if the data size is greater than a data size critical point, perform a global reduction operation between the multiple computing nodes of the butterfly network using a first multi-transmitter and multi-receiver communication paradigm; otherwise, perform a global reduction operation between the multiple computing nodes of the butterfly network using a second multi-transmitter and multi-receiver communication paradigm; The execution of a global reduction operation between the plurality of computing nodes of the butterfly network through the first multi-transmit multi-receive communication paradigm includes: Based on the matrix dimension characteristics, the data to be reduced on each computing node is divided into multiple sub-data blocks; The distributed reduction operation is performed between multiple computing nodes through multiple rounds of progressive reduction communication to achieve local reduction between each computing node; the global collection operation is performed in the opposite communication order of the local reduction operation through dimension alignment to achieve global reduction for each computing node; In each round of progressive protocol communication, the sub-data blocks are grouped using the number of communication rounds and matrix dimension characteristics, and data is transmitted through a multi-transmitter and multi-receiver communication mode; The multiple computing nodes of the butterfly network perform a global reduction operation through the second multi-transmit multi-receive communication paradigm, including: Based on the number of computing nodes, the data to be reduced on each computing node is divided into multiple sub-data blocks; Distributed protocol operations are performed between multiple computing nodes through a multi-transmitter and multi-receiver communication mode.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the butterfly network-based multi-transmitter and multi-collector communication method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the butterfly network-based multi-transmitter and multi-collector combined communication method according to any one of claims 1 to 4 are implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the butterfly network-based multi-transmitter and multi-collector combined communication method according to any one of claims 1 to 4 are implemented.

Images