Multiple-input-multiple-output collection communication method and system based on butterfly network

Abstract

The invention discloses a multiple-input-multiple-output collection communication method and system based on a butterfly network, and belongs to the technical field of distributed computing. The method comprises the following steps: acquiring single-round communication starting delay, single data element calculation time consumption and bandwidth of the butterfly network, and determining a data scale critical point of a single calculation node according to the single-round communication starting delay, the single data element calculation time consumption and the bandwidth; acquiring a data scale, if the data scale is greater than a data scale critical point, executing global protocol operation among the plurality of computing nodes of the butterfly network through a first multiple-input multiple-output communication normal form, otherwise, executing global protocol operation among the plurality of computing nodes of the butterfly network through a second multiple-input multiple-output communication normal form; adaptive communication can be carried out for distributed data transmission scenes of different scales, and the problem that an existing communication algorithm cannot give consideration to both communication delay and bandwidth utilization rate is solved.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a multi-transmit and multi-collect communication method and system based on a butterfly network. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the continuous expansion of the scale of deep learning models and the continuous growth of training data, distributed training has become an inevitable choice for large-scale machine learning. In distributed training, multiple computing nodes need to frequently exchange model parameters and gradient information, which makes the efficiency of collective communication directly affect the overall training performance. Especially in algorithms such as synchronous stochastic gradient descent, each round of iteration requires a global reduction operation to aggregate the gradient information on all computing nodes, so efficient collective communication becomes a key factor in improving distributed training performance.

[0004] In large-scale distributed training scenarios, communication overhead may take up most of the total training time. At the same time, synchronous stochastic gradient descent requires waiting for all computing nodes to complete gradient calculation and communication in each iteration. Load imbalance between computing nodes and network fluctuations may lead to reduced training efficiency. As the cluster size increases, the complexity of the communication topology increases significantly, which puts higher requirements on the scalability of the algorithm.

[0005] At present, a variety of global reduction algorithms have been proposed and widely used in the field of distributed collective communication. Among them, the ring global reduction algorithm organizes all nodes participating in the calculation into a logical ring and completes data aggregation through two stages; in the decentralized reduction stage, each node divides the data into N parts, and completes the reduction of some data through N-1 rounds of communication; in the global collection stage, the reduction results are broadcast to all nodes through N-1 rounds of communication. The ring global reduction algorithm has a high bandwidth utilization rate and is particularly suitable for large-scale data transmission scenarios, but its communication rounds will increase linearly with the number of nodes, resulting in large communication delays.

[0006] The butterfly global reduction algorithm is based on the divide-and-conquer concept and uses a tree-shaped communication mode for data aggregation. In the reduction phase, the algorithm pairs nodes in pairs for reduction in each round, and completes data reduction after log(N) rounds; in the broadcast phase, the algorithm broadcasts according to the same tree structure, which also requires log(N) rounds to complete. Compared with the ring global reduction algorithm, the butterfly global reduction algorithm has fewer communication rounds and has the advantage of lower latency, but its bandwidth utilization is relatively low, making it more suitable for transmission scenarios with small amounts of data.

[0007] The 2D-ring global reduction algorithm organizes nodes into a 2D grid structure. The whole process is divided into three stages: first, a ring-like global reduction is performed within the row, then a reduction operation is performed in the column direction, and finally, a ring-like global collection is performed in the row direction. The 2D-ring global reduction algorithm achieves a good balance in communication overhead and is particularly suitable for grid topology structures, but its implementation is relatively complex and has high requirements for the network topology structure.

[0008] In order to adapt to heterogeneous network environments, researchers also proposed a hierarchical global reduction algorithm; the algorithm first uses high-speed Internet networks to complete data reduction within nodes, and then selects a suitable algorithm based on bandwidth differences to complete inter-node reduction. This hierarchical design allows the algorithm to flexibly adapt to heterogeneous network environments, but its performance is largely affected by the network topology.

[0009] In summary, although various existing global reduction algorithms have shown their own advantages in different scenarios, they still have some common limitations: 1. It is difficult to balance latency and bandwidth utilization. Although the ring-type global reduction algorithm achieves high bandwidth utilization, its large number of communication rounds leads to large latency. Although the butterfly-type global reduction algorithm has the advantage of low latency, its bandwidth utilization is not high.

[0010] 2. There are scalability issues. The performance of most algorithms will drop significantly as the number of nodes increases, and it is difficult to maintain stable performance in a complex network environment.

[0011] 3. The existing algorithms may cause imbalance in computing and communication loads in terms of data segmentation and transmission, affecting the overall training efficiency.

[0012] In addition, these algorithms often adopt a single communication mode and fail to fully utilize the parallel multi-transmitter and multi-receiver communication capabilities of modern network hardware; however, global reduction operations have strict global dependency properties, which poses a major challenge to achieving true multi-transmitter and multi-receiver communication.

[0013] Specifically, global reduction requires that all data must participate in the operation, and the data can only be split according to the dimensions of the matrix (such as rows). During the merging process, only vectors from the same number of rows can be reduced, and the rest are combined. More importantly, the result of the reduction of all data must be obtained in the end, and this result must be consistent on all nodes. This global dependency makes it difficult to truly achieve multiple transmission and multiple reception. Summary of the invention

[0014] In order to address the deficiencies in the prior art, the present invention provides a multi-transmit and multi-receive combined communication method, system, electronic device, computer-readable storage medium and computer program product based on a butterfly network, which achieves improved bandwidth utilization while reducing communication delay through an innovative multi-transmit and multi-receive communication mode.

[0015] In a first aspect, the present invention provides a multi-transmit and multi-collect communication method based on a butterfly network; A multi-transmit and multi-collect communication method based on a butterfly network, comprising: Obtain the single-round communication startup delay, single data element calculation time and bandwidth of the butterfly network, and determine the data scale critical point of a single computing node based on the single-round communication startup delay, single data element calculation time and bandwidth; The data scale is obtained. If the data scale is greater than the data scale critical point, a global reduction operation is performed between multiple computing nodes of the butterfly network through a first multi-transmit and multi-receive communication paradigm; otherwise, a global reduction operation is performed between multiple computing nodes of the butterfly network through a second multi-transmit and multi-receive communication paradigm.

[0016] In some implementations, performing a global reduction operation between multiple computing nodes of the butterfly network through a first multiple-transmit multiple-receive communication paradigm includes: Based on the matrix dimension characteristics, the data to be reduced on each computing node is divided into multiple sub-data blocks; The decentralized reduction operation is performed through multiple rounds of progressive reduction communication between multiple computing nodes to achieve local reduction between each computing node; the global collection operation is performed through dimension alignment in the opposite communication order to the local reduction operation to achieve global reduction of each computing node; In each round of progressive protocol communication, the sub-data blocks are grouped using the number of communication rounds and data dimension characteristics, and data is transmitted through a multi-transmitter and multi-receiver communication mode.

[0017] In some implementations, the amount of data transmitted in each round of progressive protocol communication is expressed as: ; In the formula, Indicates the size of the data to be reduced on each computing node. represents the number of computing nodes, Indicates the number of communication rounds of progressive protocol communication.

[0018] In some implementations, performing a global reduction operation between multiple computing nodes of the butterfly network through the second multiple-transmit multiple-receive communication paradigm includes: Based on the number of computing nodes, the data to be reduced on each computing node is divided into a plurality of sub-data blocks; Distributed protocol operations are performed between multiple computing nodes through a multi-transmit and multi-receive communication mode.

[0019] In some implementations, the time overhead of the first multiple-transmit and multiple-receive communication paradigm is: ; In the formula, represents the number of computing nodes, represents bandwidth, Indicates the delay in starting a single round of communication. Indicates the size of the data to be reduced on each computing node. Indicates the time taken to calculate a single data element.

[0020] In some implementations, the data scale critical point of the single computing node is expressed as: ; In the formula, Indicates the critical point of data scale of a single computing node, N represents the number of computing nodes, represents bandwidth, Indicates the time taken to calculate a single data element. Indicates the delay in starting a single round of communication.

[0021] In a second aspect, the present invention provides a multi-transmitter and multi-collector combined communication system based on a butterfly network; A multi-transmitter and multi-collector combined communication system based on a butterfly network, comprising: The adaptive mode selection module is configured to: obtain the single-round communication startup delay, the calculation time of a single data element and the bandwidth of the butterfly network, and determine the data scale critical point of a single computing node according to the single-round communication startup delay, the calculation time of a single data element and the bandwidth; The global reduction module is configured to: obtain the data scale, if the data scale is greater than the data scale critical point, the multiple computing nodes of the butterfly network perform the global reduction operation through the first multi-transmit multi-receive communication paradigm, otherwise, the multiple computing nodes of the butterfly network perform the global reduction operation through the second multi-transmit multi-receive communication paradigm.

[0022] In a third aspect, the present invention provides an electronic device; An electronic device includes a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the multi-transmit and multi-collect combined communication method based on a butterfly network.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium; A computer-readable storage medium stores a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned multi-transmit and multi-collect combined communication method based on a butterfly network.

[0024] In a fifth aspect, the present invention provides a computer program product; A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned butterfly network-based multi-transmit and multi-collect combined communication method.

[0025] Compared with the prior art, the present invention has the following beneficial effects: 1. The technical solution provided by the present invention is based on a butterfly network to perform a multi-transmit and multi-receive paradigm design that adapts to different numbers of nodes, providing the optimal multi-transmit and multi-receive expression for each number of computing nodes.

[0026] 2. The technical solution provided by the present invention designs an adaptive communication strategy based on data scale, and provides two complementary communication paradigms for distributed data transmission scenarios of different scales, one is a multi-transmitter and multi-receiver communication paradigm suitable for large-scale data transmission, and the other is a simplified solution suitable for small-scale data transmission. In practical applications, the communication solution can be automatically selected according to the data scale, and continuous evaluation and adjustment can be performed during system operation to ensure that the optimal communication strategy is always adopted. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0028] Figure 1 A flowchart of a multi-transmit and multi-collect combined communication method based on a butterfly network provided in an embodiment of the present invention; Figure 2 A schematic diagram of computing node connections in an 8-node butterfly network provided by an embodiment of the present invention; Figure 3 A schematic diagram of the connection of computing nodes in a 16-node butterfly network provided by an embodiment of the present invention; Figure 4 A schematic diagram of a communication process in which a first multi-transmit and multi-receive communication paradigm provided in an embodiment of the present invention is applied to an 8-node system; Figure 5 A schematic diagram of a communication process in which the second multi-transmit and multi-receive communication paradigm provided in an embodiment of the present invention is applied to an 8-node system. DETAILED DESCRIPTION

[0029] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0032] Terminology explanation: Distributed training: Distributed training is a method that distributes the training tasks of large-scale machine learning models to multiple computing nodes for parallel execution. During the training process, each node calculates the gradient based on local data and exchanges update information through network communication, thereby collaboratively completing the model optimization process. This method can effectively improve training efficiency and support the training of larger-scale models.

[0033] Collective communication: In a distributed computing environment, collective communication refers to communication operations involving multiple computing nodes. Its basic forms include broadcast, protocol, full protocol, collection, etc. These operations are the basis for building an efficient distributed system.

[0034] Global reduction operation: Global reduction operation is a basic collective communication operation, which is used to reduce data distributed on multiple computing nodes (such as summing, taking the maximum value, etc.) and distribute the final result to all nodes participating in the calculation. In distributed machine learning, full reduction operation is mainly used for gradient aggregation and is a key mechanism to ensure model consistency.

[0035] Communication delay: In a distributed system, communication delay refers to the time required for data to be sent from the source node to the destination node. It usually consists of two parts: startup delay, which is the fixed overhead required to establish a communication connection; and transmission delay related to the amount of data. Communication delay is an important factor affecting the performance of distributed systems.

[0036] Bandwidth utilization: Bandwidth utilization refers to the ratio of the network bandwidth actually used during data transmission to the theoretical bandwidth. A higher bandwidth utilization means that network resources are more fully utilized, which helps to improve the overall performance of the system.

[0037] Data reduction: Data reduction refers to the process of combining data from multiple nodes into a single result according to specific operations (such as summation, averaging, etc.). In distributed machine learning, gradient reduction is a necessary step to ensure model convergence.

[0038] Distributed reduction: Distributed reduction is the first stage of the global reduction operation. Its function is to store the reduction results in a distributed manner on each node involved in the calculation. Each node eventually obtains part of the reduction results to prepare for subsequent global data reconstruction.

[0039] Global collection: Global collection is the second stage of the global reduction operation. Through this operation, each node shares its partial reduction results with all other nodes, ultimately achieving global consistency of data.

[0040] Computing node: The basic computing unit in a distributed system, which has independent processor, memory and other computing resources, can perform assigned computing tasks and communicate with other nodes.

[0041] Embodiment 1 The existing global reduction algorithm cannot take into account both communication delay and bandwidth utilization at the same time, which affects the training efficiency of distributed training; therefore, the present invention provides a multi-transmit and multi-collect communication method based on a butterfly network, which improves bandwidth utilization while reducing communication delay.

[0042] Next, combine Figure 1-Figure 5 , a multi-transmit and multi-collection communication method based on a butterfly network disclosed in this embodiment is described in detail. The butterfly network includes multiple computing nodes, and the multi-transmit and multi-collection communication method based on the butterfly network includes: S1. Obtain the single-round communication startup delay, single data element calculation time and bandwidth of the butterfly network, and determine the data scale critical point of a single computing node based on the single-round communication startup delay, single data element calculation time and bandwidth.

[0043] In this embodiment, the critical point of the data scale of a single computing node is expressed as: ; In the formula, Indicates the critical point of data scale of a single computing node, N represents the number of computing nodes, represents bandwidth, Indicates the time taken to calculate a single data element. Indicates the delay in starting a single round of communication.

[0044] Based on this, the critical point of data scale is calculated by combining communication delay, bandwidth overhead and computing overhead, which provides a theoretical basis for the selection of subsequent communication solutions.

[0045] In this embodiment, a butterfly network is used as the basic communication topology, combined with Figure 3-Figure 4 , further introduce the butterfly network.

[0046] The 4-node butterfly network adopts a square topology, including 4 computing nodes (numbered 1-4). Each computing node establishes a direct connection with its two adjacent computing nodes, forming a square structure as a whole, ensuring that any two computing nodes can communicate through at most one intermediate node; the communication path is symmetrical to ensure load balancing. In each round of communication, each computing node performs 2 sends and 2 receives in parallel.

[0047] Combination Figure 3 , the 8-node butterfly network configuration forms a three-dimensional cube projection structure, with nodes numbered 1-8; additional connections are established between peripheral computing nodes according to the butterfly network rules, with one vertex connecting three vertices; the network structure has good symmetry to ensure the uniformity of communication distance; the global reduction under this configuration requires In each round of communication, each computing node performs 3 sends and 3 receives in parallel.

[0048] Combination Figure 4 , the 16-node butterfly network configuration forms a complex ring interconnection structure, with nodes numbered 1-16; the computing nodes establish cross-connections on the basis of the ring, with one vertex connecting 4 vertices; each computing node establishes connections with adjacent computing nodes and corresponding cross-ring nodes to form multiple parallel communication paths, supporting efficient data exchange; the global protocol needs to be completed under this configuration In each round of communication, each computing node performs 4 sends and 4 receives in parallel.

[0049] The number of connections between nodes in a butterfly network grows logarithmically with the scale of the system, which avoids the hardware overhead caused by full connectivity while maintaining a low network diameter; in terms of communication efficiency, the structure supports multiple parallel communication paths and achieves good load balancing through evenly distributed communication distances; in addition, the self-similarity of the topology makes it easy to expand to larger-scale systems, and the performance only changes logarithmically with the number of nodes, reflecting excellent scalability. Although the network topology presents different geometric structures under different scale configurations, they all follow the same butterfly network construction principles; this unified design pattern ensures that the algorithm can maintain stable performance in systems of different scales. With the support of this network topology, combined with the subsequent multi-transmit and multi-receive communication mechanism, this embodiment can achieve efficient global data reduction and provide strong communication infrastructure support for distributed machine learning systems.

[0050] S2. Obtain the data scale. If the data scale is greater than the data scale critical point, execute S3; otherwise, execute S4.

[0051] Based on the data scale critical point S*, this embodiment implements a dynamic adaptive strategy: when the data scale S is greater than the data scale critical point S*, the first multi-transmit and multi-receive communication paradigm for large-scale data transmission is selected, and better bandwidth utilization can be achieved at this time; when S is less than or equal to the data scale critical point S*, the second multi-transmit and multi-receive communication paradigm for small-scale data transmission is selected to obtain lower communication overhead.

[0052] Here, the data scale refers to the data size corresponding to the parameter matrix M being divided into n parts.

[0053] Furthermore, this adaptive mechanism can continuously evaluate and adjust during operation to ensure that the optimal communication strategy is always adopted.

[0054] S3. Multiple computing nodes in the butterfly network perform global reduction operations through the first multi-transmit and multi-receive communication paradigm. The specific process is as follows: S301: Obtain a data set to be reduced and perform initial partitioning.

[0055] In this embodiment, the data set to be reduced is a parameter matrix in a deep learning model.

[0056] Specifically, based on the two-dimensional characteristics of the parameter matrix, a tensor parallel method is used for initial partitioning. For example, for a parameter matrix M with a dimension of p rows and q columns, it can be evenly divided into n parts along the row direction or column direction and assigned to n computing nodes respectively; when the parameter matrix M is divided along the row direction, each computing node obtains a sub-matrix of (p / n) rows and q columns; when the parameter matrix M is divided along the column direction, each computing node obtains a sub-matrix of p rows and (q / n) columns.

[0057] Furthermore, the choice of partitioning direction needs to consider the specific computing mode, that is, whether to choose row partitioning or column partitioning is determined according to the specific operation type of matrix calculation (such as matrix-vector multiplication, matrix-matrix multiplication): when the calculation process needs to frequently access complete row data, column partitioning may be better; conversely, when the calculation depends on complete column data, row partitioning is more appropriate.

[0058] After each computing node completes the local gradient calculation, it needs to perform a global reduction operation (such as summing or averaging) to unify the gradient values ​​of all nodes. In order to solve the packet loss problem during large-scale data transmission and ensure the pipeline efficiency of calculation and transmission, in this embodiment, the data segmentation strategy is determined before the training starts.

[0059] Specifically, for a butterfly network with n computing nodes, the parameter matrix M is evenly divided into n parts, each of which is M / n, or S. The data block of size S on each computing node will be split step by step during the communication process and finally divided into equal sub-blocks, each of which is of size This pre-planned partitioning scheme enables each computing node to transmit and reduce these sub-data blocks one by one according to the determined steps, while ensuring data integrity and realizing efficient pipeline operation of computing and communication.

[0060] S302, performing distributed reduction operations among multiple computing nodes through multiple rounds of progressive reduction communications to achieve local reduction among each computing node. Round progressive protocol communication, to achieve local data protocol, in the i-th round of protocol communication (i from 1 to ), the specific process of progressive protocol communication is as follows: (1) Each computing node groups the currently held data into to ensure that the reducible data is located in the same dimensional position.

[0061] Here, when i= When , the size of the data after segmentation is .

[0062] (2) Adoption hair The communication mode of receiving and receiving, each computing node sends The computing nodes (target nodes) send Size data, at the same time, from The data is received from a computing node (source node) and the received data is ensured to be of the same dimension as the local data for reduction.

[0063] (3) After each round of communication is completed, the computing node merges the received data of the same dimension through reduction operations (such as summation, maximum value, etc.).

[0064] Finally passed Each computing node in the round holds A locally regulated The size of the data fragment.

[0065] Next, we take the distributed training of the attention layer in the Llama model and the tensor parallelism of 8 computing nodes as an example to illustrate the specific process of decentralized reduction.

[0066] The size of the gradient data of the parameter matrix in the attention layer is 3072×3072. Under the tensor parallel strategy, the parameter matrix is ​​divided into 8 computing nodes by rows, so each computing node initially holds a gradient slice of size 384×3072.

[0067] The decentralized protocol was implemented through 3 rounds ( ) Progressive communication is completed. In the first round of communication, each computing node divides its own 384×3072 gradient slice into 6 parts ( ), that is, the size of each gradient slice is 384×512. The computing nodes communicate with each other according to the butterfly network pairing relationship: computing node 1 sends gradient slices to computing nodes 2, 3, and 5 at the same time, and receives gradient slices of the same size from these computing nodes; similarly, computing node 4 sends and receives gradient slices to computing nodes 3, 2, and 8; after receiving the gradient slices, each computing node immediately sums and reduces the gradient slices at the same position.

[0068] In the second round of communication, the computing node divides the gradient shards after the first round of reduction into two parts, which is one-twelfth of the initial gradient shards ( ), each size is 384 × 256. Similarly, after many-to-many communication, the gradient slices are received and the gradients of the corresponding positions are summed.

[0069] In the third round of communication, the computing node divides the gradient shards after the second round of reduction into 24 parts ( ), each with a size of 384×128, completing the final local reduction operation.

[0070] After the above three rounds of communication, each computing node finally holds 3 ( ) The fragments of the locally reduced gradient slices, each of which is 384×128 in size; these fragments contain the local reduction results of the original gradient in different dimensions, preparing for the subsequent global collection operation; this multi-send and multi-receive communication scheme not only achieves a balanced distribution of the communication load, but also effectively supports the pipeline operation of communication and computing by gradually reducing the size of the data slices.

[0071] S303 , performing a global collection operation in a dimension-aligned manner in a communication order opposite to that of the local reduction operation, thereby realizing a global reduction of each computing node.

[0072] Specifically, from The communication operation is performed in reverse order from the beginning of the round to the first round. In each round of communication, each computing node sends the local reduction result it holds to the corresponding target nodes, and from A source node receives data; after receiving the data, the computing node locates the dimension position according to the position offset of the data it holds in the original tensor and assembles it, gradually reconstructing the complete reduction result.

[0073] Exemplarily, the rank value of each computing node is pre-assigned. When performing a global reduction operation, each computing node has a receiving buffer equal to the size of the original tensor. When receiving data from other computing nodes, the position offset is directly calculated according to the rank value of the computing node, and the data is placed in the corresponding position. The position offset is expressed as: offset = rank * chunk_size; In the formula, chunk_size represents the data size.

[0074] For example, the data sent by the computing node with a rank value of 0 is stored in [0: chunk_size], the data sent by the computing node with a rank value of 1 is stored in [chunk_size: 2*chunk_size], the data sent by the computing node with a rank value of 2 is stored in [2chunk_size: 3chunk_size], and so on.

[0075] This dimension alignment ensures the correctness of the data during the reconstruction process, and ultimately all nodes obtain completely consistent global reduction results. Through this two-way communication mode, both the complete transmission of data and the consistency of the reduction results are guaranteed.

[0076] In some embodiments, the time overhead of the first multiple-transmit and multiple-receive communication paradigm is evaluated and expressed as: ; In the formula, represents the number of computing nodes, represents bandwidth, Indicates the delay in starting a single round of communication. Represents the data to be reduced on each computing node, Indicates the time taken to calculate a single data element.

[0077] Based on this, when n=8, the sum of communication delays is 3 / 4 of that of the 2D-ring global reduction algorithm, and the sum of bandwidth consumption and computational consumption is 1 / 3 of that of the 2D-ring global reduction algorithm.

[0078] S4. Multiple computing nodes in the butterfly network perform global reduction operations through the second multi-transmit and multi-receive communication paradigm. Specifically, it includes: S401, the parameter matrix M is evenly divided into n parts, each part is M / n, that is, S, and is distributed to n computing nodes respectively. Each computing node divides the obtained data into sub-blocks, each sub-block is of size .

[0079] This coarse-grained data partitioning method in this embodiment reduces data management overhead and is suitable for small-scale data transmission scenarios.

[0080] S402, use hair The communication mode of receiving and receiving, each computing node sends The computing nodes (target nodes) send S / (log2n) copies of data, and at the same time, The data is received from a computing node (source node) and the received data is ensured to be of the same dimension as the local data for reduction.

[0081] S403. After each round of communication is completed, the computing nodes merge the data of the same dimension through reduction operations (such as summation, maximum value, etc.) and finally hold A locally regulated The complete global protocol is reconstructed with data fragments of different sizes, that is, all the data, and all nodes obtain completely consistent results.

[0082] Exemplarily, taking an 8-node butterfly network as an example, the communication process of the second multi-transmit and multi-receive communication paradigm is further described, and the specific process is as follows: First, the gradient matrix to be reduced is evenly divided into , that is, 3 initial data shards (where n=8, ); In the initial state, each computing node is allocated 3 initial data shards. This coarse-grained data partitioning simplifies the communication operation steps.

[0083] In the first round of communication, each computing node executes a 3-transmit 3-receive communication mode; specifically, each computing node divides the initial data fragment currently held into 1 / 3 size data blocks for transmission, and each computing node simultaneously sends data to three target nodes and receives data from three source nodes.

[0084] After completing the first round of data exchange, each computing node will hold 3 data fragments of 1 / 3 size that have undergone the first local reduction.

[0085] In the second round of communication, the computing nodes carry out the next data exchange and execute the 3-transmit-3-receive communication mode again on the 1 / 3 data obtained through local reduction in the previous round; each computing node performs a second local reduction on the received data.

[0086] After the communication is completed, each computing node obtains three 1 / 3 data fragments that have been reduced twice.

[0087] In the third round of communication, the final local reduction is completed through the last round of 3 sends and 3 receives. At this time, each node holds 3 fully reduced 1 / 3 data fragments, which together constitute the complete reduction result of the gradient matrix, and finally reconstruct the complete gradient matrix on all computing nodes.

[0088] In some embodiments, the time overhead of the second multiple-transmit and multiple-receive communication paradigm is evaluated and expressed as: ; In the formula, represents the number of computing nodes, Indicates the delay in starting a single round of communication. Represents the data to be reduced on each computing node, represents bandwidth, Indicates the time taken to calculate a single data element.

[0089] Based on this, when n=8, compared with the 2D-Torus algorithm, the sum of the communication delays is 3 / 8 of that of the 2D-Torus, and the sum of the bandwidth consumption and the computational consumption are both 8 / 14 of that of the 2D-Torus, which is 7 / 12 of the first multi-transmitter and multi-receiver communication paradigm.

[0090] Embodiment 2 This embodiment discloses a multi-transmitter and multi-collector combined communication system based on a butterfly network, including: The adaptive mode selection module is configured to: obtain the single-round communication startup delay, the calculation time of a single data element and the bandwidth of the butterfly network, and determine the data scale critical point of a single computing node according to the single-round communication startup delay, the calculation time of a single data element and the bandwidth; The global reduction module is configured to: obtain the data scale, if the data scale is greater than the data scale critical point, the multiple computing nodes of the butterfly network perform the global reduction operation through the first multi-transmit multi-receive communication paradigm, otherwise, the multiple computing nodes of the butterfly network perform the global reduction operation through the second multi-transmit multi-receive communication paradigm.

[0091] It should be noted that the above-mentioned adaptive mode selection module and global reduction module correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Embodiment 1. It should be noted that the above-mentioned modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0092] Embodiment 3 Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps of the multi-transmit and multi-collect combined communication method based on the butterfly network are completed.

[0093] Embodiment 4 Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned multi-transmitter and multi-collector combined communication method based on a butterfly network are completed.

[0094] Embodiment 5 Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned butterfly network-based multi-transmit and multi-collect combined communication method when executed by a processor.

[0095] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0098] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-transmit and multi-collect communication method based on a butterfly network, characterized in that: include: Obtain the single-round communication startup delay, single data element calculation time and bandwidth of the butterfly network, and determine the data scale critical point of a single computing node based on the single-round communication startup delay, single data element calculation time and bandwidth; The data scale is obtained. If the data scale is greater than the data scale critical point, a global reduction operation is performed between multiple computing nodes of the butterfly network through a first multi-transmit and multi-receive communication paradigm; otherwise, a global reduction operation is performed between multiple computing nodes of the butterfly network through a second multi-transmit and multi-receive communication paradigm.

2. The multi-transmit and multi-collect communication method based on butterfly network as claimed in claim 1, characterized in that: The multiple computing nodes of the butterfly network perform a global reduction operation through a first multi-transmit multi-receive communication paradigm, including: Based on the matrix dimension characteristics, the data to be reduced on each computing node is divided into multiple sub-data blocks; The decentralized reduction operation is performed through multiple rounds of progressive reduction communication between multiple computing nodes to achieve local reduction between each computing node; the global collection operation is performed through dimension alignment in the opposite communication order to the local reduction operation to achieve global reduction of each computing node; In each round of progressive protocol communication, the sub-data blocks are grouped using the number of communication rounds and matrix dimension characteristics, and data is transmitted through a multi-transmitter and multi-receiver communication mode.

3. The multi-transmit and multi-collect communication method based on butterfly network as claimed in claim 2, characterized in that: The amount of data transmitted in each round of progressive protocol communication is expressed as: ； In the formula, Indicates the size of the data to be reduced on each computing node. represents the number of computing nodes, Indicates the number of communication rounds of progressive protocol communication.

4. The multi-transmit and multi-collect communication method based on butterfly network as claimed in claim 1, characterized in that: The multiple computing nodes of the butterfly network perform a global reduction operation through the second multiple-transmit multiple-receive communication paradigm, including: Based on the number of computing nodes, the data to be reduced on each computing node is divided into a plurality of sub-data blocks; Distributed protocol operations are performed between multiple computing nodes through a multi-transmit and multi-receive communication mode.

5. The multi-transmit and multi-collect communication method based on butterfly network as claimed in claim 1, characterized in that: The time overhead of the first multi-transmit and multi-receive communication paradigm is: ； In the formula, represents the number of computing nodes, represents bandwidth, Indicates the delay in starting a single round of communication. Indicates the size of the data to be reduced on each computing node. Indicates the time taken to calculate a single data element.

6. The multi-transmit and multi-collect communication method based on butterfly network as claimed in claim 1, characterized in that: The data scale critical point of a single computing node is expressed as: ； In the formula, Indicates the critical point of data scale of a single computing node, N represents the number of computing nodes, represents bandwidth, Indicates the time taken to calculate a single data element. Indicates the delay in starting a single round of communication.

7. A multi-transmitter and multi-collector combined communication system based on a butterfly network, characterized in that: include: The adaptive mode selection module is configured to: obtain the single-round communication startup delay, the calculation time of a single data element and the bandwidth of the butterfly network, and determine the data scale critical point of a single computing node according to the single-round communication startup delay, the calculation time of a single data element and the bandwidth; The global reduction module is configured to: obtain the data scale, if the data scale is greater than the data scale critical point, the multiple computing nodes of the butterfly network perform the global reduction operation through the first multi-transmit multi-receive communication paradigm, otherwise, the multiple computing nodes of the butterfly network perform the global reduction operation through the second multi-transmit multi-receive communication paradigm.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the butterfly network-based multi-transmit and multi-collect communication method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the butterfly network-based multi-transmit and multi-collect communication method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the butterfly network-based multi-transmit and multi-collect communication method according to any one of claims 1 to 6 are implemented.

Images