Distributed data processing method and device, computer storage medium and electronic equipment

Abstract

The invention provides a distributed data processing method and device, electronic equipment and a computer storage medium, and relates to the technical field of distributed data processing. The method comprises the following steps: determining 2N computational nodes from 2N + 1 computational nodes, and taking the remaining computational nodes as exclusion nodes; a merging node is determined from the 2N computing nodes, and the merging node is connected with the exclusion node so as to merge data of the exclusion node; 2 < N > computing nodes form a bidirectional ring, bidirectional ring full reduction data processing is carried out, and during processing of each time step, a merging node receives and aggregates data from an exclusion node and a father node. According to the method, the even number of nodes are determined from the odd number of nodes to form the bidirectional ring, and the data of the nodes are eliminated by combining the nodes, so that the full reduction calculation processing of the bidirectional ring is realized, and the link utilization rate is improved.

Description

Background Art

[0002] When training large models, data parallel training is one of the main methods for processing large-scale data sets. The current mainstream technical solutions usually use the back propagation algorithm to optimize the weights and biases of deep neural networks (DNNs).

[0003] During this process, the model goes through the following steps:

[0004] Forward propagation: Input data is calculated through multiple layers of neurons to generate output results.

[0005] Loss function calculation: Calculate the loss value based on the difference between the model's output and the true label.

[0006] Gradient calculation: Through the back-propagation algorithm, the chain rule is used to calculate the gradient of the loss function with respect to each parameter.

[0007] Parameter update: Use an optimization algorithm such as stochastic gradient descent (SGD) to update the model parameters based on the calculated gradients.

[0008] In order to improve training efficiency, especially when processing large-scale data, data parallel distributed training is widely used. This process usually involves dividing the data into small batches and distributing them to multiple computing nodes. After each node independently calculates the gradient, these gradients are synchronized through the AllReduce algorithm to ensure that the models of all nodes remain consistent.

[0009] Currently, the ring-based AllReduce technology solution has many obstacles and faces performance bottlenecks.

[0010] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0011] The purpose of the present disclosure is to provide a distributed data processing method, device, electronic device and computer storage medium, which at least to some extent overcome the problem of low link utilization in related technologies.

[0012] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0013] According to one aspect of the present disclosure, there is provided a distributed data processing method, which is applied to a distributed network, and the method includes: determining 2N computing nodes from 2N+1 computing nodes, and using the remaining computing nodes as exclusion nodes; determining a merging node from the 2N computing nodes, and the merging node is connected to the exclusion node to merge the data of the exclusion node; forming a bidirectional ring with the 2N computing nodes, and performing bidirectional ring (Ring) AllReduce (AllReduce) data processing, wherein, during the processing of each time step, the merging node receives and aggregates data from the exclusion node and from the parent node; wherein N is a natural number.

[0014] In one embodiment, the merging node includes a first merging node and a second merging node, which correspond to merging nodes performing full reduction data processing in a first direction and a second direction, respectively.

[0015] In one embodiment, the merge node is a computing node, which is used to process the full-reduce data processing in the first direction and the second direction.

[0016] In one embodiment, data of 2N+1 computing nodes are divided into a first part and a second part; and the first part and the second part are respectively divided into 2N parts; the 2N computing nodes are used to form a bidirectional ring and perform bidirectional ring (Ring) AllReduce data processing, including: performing a first direction of reduction and scatter processing, so that each of the 2N computing nodes obtains one 2N part of the first part of the data through 2N time steps, wherein at each time step, the merging node receives and aggregates data from the excluded node and the parent node; performing a first direction of collective collection processing, so that each of the 2N computing nodes obtains all aggregated results of the first part through 2N time steps; performing a second direction of reduction and scatter processing, so that each of the 2N computing nodes obtains one 2N part of the second part of the data through 2N time steps, wherein at each time step, the merging node receives and aggregates data from the excluded node and the parent node; performing a second direction of collective collection processing, so that each of the 2N computing nodes obtains all aggregated results of the second part through 2N time steps.

[0017] In one embodiment, determining 2N computing nodes from 2N+1 computing nodes includes: determining 2N computing nodes from the 2N+1 computing nodes based on a linear time algorithm, the 2N computing nodes forming a Hamiltonian cycle.

[0018] In one embodiment, the linear time algorithm comprises a genetic algorithm, a probabilistic algorithm, or a greedy algorithm.

[0019] In one embodiment, the distributed network includes the 2N+1 computing nodes.

[0020] According to another aspect of the present disclosure, a distributed data processing device is provided, including: an exclusion node determination module, used to determine 2N computing nodes from 2N+1 computing nodes, and use the remaining computing nodes as exclusion nodes; a merging node determination module, used to determine a merging node from the 2N computing nodes, the merging node is connected to the exclusion node to merge the data of the exclusion node; a reduction processing module, used to form a bidirectional ring with the 2N computing nodes, and perform bidirectional ring (Ring) all-reduce (AllReduce) data processing, wherein, during the processing of each time step, the merging node receives and aggregates data from the exclusion node and from the parent node; wherein N is a natural number.

[0021] According to another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned distributed data processing method by executing the executable instructions.

[0022] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned distributed data processing method is implemented.

[0023] The distributed data processing method and device provided by the embodiments of the present disclosure form a bidirectional ring by determining an even number of nodes from an odd number of nodes, and implement full reduction calculation processing of the bidirectional ring by merging the node processing data of the excluded nodes, thereby improving link utilization.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0026] Figure 1 An exemplary system architecture for a distributed computing system according to an embodiment of the present disclosure is shown;

[0027] Figure 2 A flow chart showing a distributed data processing method according to an embodiment of the present disclosure;

[0028] Figure 3 An example of an all-reduce algorithm is shown;

[0029] Figure 4 A schematic diagram of a 3*5 grid in one embodiment of the present disclosure is shown;

[0030] Figure 5A A flow chart showing a distributed data processing method according to another embodiment of the present disclosure;

[0031] Figure 5B Show Figure 5A A schematic diagram of a grid of an example in the embodiment;

[0032] Figure 5C Show Figure 5B In the example, each node divides its data into 2 parts, and each part is divided into 8 parts;

[0033] Figure 5D Show Figure 5B Schematic diagram of data flow at each time step in the example;

[0034] Figure 6 An exemplary computer system capable of implementing an all-reduce operation based on a network interface according to an embodiment of the present disclosure is shown.

[0035] Figure 7 A schematic diagram of a distributed data processing device in one embodiment of the present disclosure is shown;

[0036] Figure 8 A structural block diagram of a computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0038] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0039] To facilitate understanding, several terms involved in this application are first explained below.

[0040] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0043] AllReduce: is a common operation in distributed computing, used to exchange and aggregate data between different computing nodes. Its function is to aggregate and synchronize the data on each computing node to achieve global data consistency across the entire cluster. This operation is usually used in scenarios that require large-scale parallel computing, such as deep learning training.

[0044] The inventors of the present application have found that although bidirectional ring AllReduce can significantly improve bandwidth utilization, its construction depends on the underlying physical topology. In a mesh topology that lacks surrounding links, the process of finding a bidirectional ring becomes complicated and not always feasible. In addition, there are no available Hamiltonian rings in an odd-sized mesh topology, which limits the possibility of all nodes forming a bidirectional ring. Therefore, it is necessary to design solutions for these specific topologies to improve the overall data transmission efficiency.

[0045] Related technical solutions such as hierarchical Ring2D and DBTree perform poorly in grids, resulting in delays and limited applicability; while one-way ring AllReduce is applicable to all grid configurations, its link utilization is low and MultiTree's link utilization is also constrained.

[0046] The technical solution provided by the inventor is specifically for the algorithm of mesh topology, ensuring the effective application of bidirectional ring AllReduce in various topological structures, solving the problem of building bidirectional rings in odd-sized mesh topologies, optimizing the implementation of bidirectional ring AllReduce, and thus improving link utilization.

[0047] By proposing a new algorithm, these limitations can be effectively overcome, link utilization in the grid topology can be improved, and data transmission efficiency can be improved. This technical effect will promote collective communication applications under various grid structures, making them more efficient and reliable.

[0048] Figure 1 FIG. 2 shows an exemplary system architecture for a distributed computing system in one embodiment of the present disclosure. Figure 1 In the example shown, the distributed computing system 100 may include multiple computer nodes 102, 104, 106, etc., each of which includes a central processing unit (CPU) 111, which may be a processor group, which may include one or more processors. For simplicity, Figure 1 Only one processor is shown in FIG. 1. CPU 111 may be coupled to multiple interconnected GPUs 114.

[0049] In one embodiment, the CPU 111 in each computing node can be coupled to a NIC. Figure 1As shown, NICs 120, 130, and 140 are coupled to (multiple) CPUs in corresponding computing nodes 102, 104, and 106, respectively. In one embodiment, each processor within a computing node can be coupled to multiple memories 112 (e.g., dynamic random access memory (DRAM)) using, for example, double data rate (DDR) dual in-line memory module (DIMM) slots and other memory technologies (e.g., high bandwidth memory (HBM)). Interconnect 150 (e.g., Infiniband or Ethernet) can be used for inter-node traffic between nodes 102, 104, and 106 via NICs (e.g., 120, 130, and 140).

[0050] In one embodiment, the AllReduce operation is performed by CPU 111 and GPU 114. In another embodiment, the AllReduce operation may be performed by an enhanced NIC, such as 120, 130, and 140. The NIC may be used to communicate between different computing nodes (i.e., 102, 104, and 106) in the distributed computing system 100 via interconnect 150. A system associated with the NIC may provide a process for computing the AllReduce.

[0051] Some embodiments described herein may provide a distributed computing system 100 that may support different variants of high performance computing (HPC) Ethernet NIC application specific integrated circuits (ASICs), such as 200Gb / s, 400Gb / s, and / or 800Gb / s HPC Ethernet NICs. The HPC Ethernet NIC (e.g., 108) may provide low latency, may enable user mode access to the network, and may offload critical path message processing functions of HPC programming models, such as the Message Passing Interface (MPI) or Symmetric Hierarchical Memory (SHMEM) standards.

[0052] In one embodiment, the performance of the all-reduce algorithm is improved by enhancing the NIC hardware. In other words, the NIC 120 hardware may include a processing unit 121, an arithmetic logic unit (ALU) 122, and a cache 123. The processing unit 121 may perform arithmetic operations on batches of data and organize the data through the ALU 122. Therefore, integrating the ALU 122 in the NIC 120 may reduce the number of requests sent to the host CPU 111, which may result in a significant reduction in the latency involved. In addition, the ALU 122 may combine input data received from different computing nodes at the maximum possible rate (e.g., at the line rate).

[0053] In one embodiment, the NIC 120 hardware is enhanced to accumulate incoming data into a temporary local buffer (e.g., cache 123) at full network bandwidth. This can be achieved through ALU 122. For example, ALU 122 can operate on 128 bytes per cycle to match the network rate of 800 Gb / s.

[0054] In one embodiment, an address translation unit (ATU) or infrastructure is included in the NIC 120, which can control which areas of memory can be cached in the NIC 120 in a non-coherent manner. The address translation unit 124 can be extended to enable areas to be marked as cacheable to use the local cache 123, thereby separating cacheable memory from non-cacheable memory. In other words, the address translation unit 124 can implement a marking operation that can identify which operations can target certain areas in the cache 123. For example, the address translation unit 124 can identify an address associated with an operation and determine whether the address belongs to a cacheable area or an non-cacheable area. This marking / marking operation is desirable to simplify software management of the cache to achieve a consistent state, thereby providing control over information in a cacheable memory area so that the information associated with the operation is in a consistent state. Other operations are provided to control the use of the cache 123, which enables software to flush one or more cache lines associated with a specific user, process, or job.

[0055] In one embodiment, results are transferred from the buffer rather than pushed to host memory, such as memory 112. The system can achieve this by defining on-chip interactions. In addition, the system can utilize triggered operations to pace (or add flow control to) operations across all participants (e.g., computing nodes in a network) so that the all-reduce algorithm can maximize reuse of cached data and minimize the required cache size. Utilizing triggered operations to pace (or add flow control to) network operations is described in the following subsection: All-Reduce with Triggered Operations to Add Flow Control.

[0056] Below, each step of the distributed data processing method in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0057] Figure 2 A flow chart showing a distributed data processing method according to an embodiment of the present disclosure is shown. This embodiment is applied to a distributed network. The method provided by the embodiment of the present disclosure can be executed by any electronic device with computing and processing capabilities.

[0058] like Figure 2As shown, S202, 2N computing nodes are determined from the 2N+1 computing nodes, and the remaining computing nodes are used as excluded nodes, where N is a natural number. In one embodiment, the distributed network includes 2N+1 computing nodes. The 2N computing nodes can be conveniently determined based on the connection relationship between the computing nodes, the network transmission rate of the connection between the computing nodes, the computing power of the computing nodes, etc., so that the selected 2N computing nodes can form a bidirectional ring connection.

[0059] S204, determining a merge node from the 2N computing nodes, and connecting the merge node to the exclusion node to merge the data of the exclusion node. One or two merge nodes may be selected from the 2N computing nodes based on aspects such as the network connection between the computing node and the exclusion node, computing capability, etc. The selected merge node is connected to the exclusion node to merge the data of the exclusion node.

[0060] S206, forming a bidirectional ring with 2N computing nodes, and performing bidirectional ring (Ring) AllReduce data processing, wherein, during the processing of each time step, the merging node receives and aggregates data from the excluded node and from its parent node.

[0061] In the above embodiment, 2N nodes are selected from a distributed network of 2N+1 computing nodes to form a bidirectional ring, and the data of the remaining 1 node is merged into the bidirectional ring through a merging node, thereby realizing full reduction data processing of the bidirectional ring, overcoming the deficiencies and shortcomings of the prior art, and applying the full reduction data processing of the bidirectional ring to a distributed network with an odd number of nodes.

[0062] Figure 3 The illustrated example shows a high-level hierarchical implementation of the RA (Rabenseifner algorithm) all-reduce algorithm involving steps (i)-(v) for a four-node distributed computing system. Figure 3 The all-reduce algorithm shown represents a tree implementation of RA. Each node (e.g., nodes A 202, B 204, C 206, and D 208) may include a vector having multiple elements, i.e., a vertical column. RA may divide the vector in each node into four segments, e.g., 210, 212, 214, and 216. Figure 3(i) may represent the first step in RA, in which node A 202 may send half of the data in the vector (i.e., vector elements A0 and A1) to node B 204. This is indicated by the dashed arrows between nodes A 202 and B 204 (i.e., from A0 to B0 and from A1 to B1). Node B 204 then combines the incoming data with its local data to create two new partial results, i.e., node B 204 will include partial results {A0, B0} and {A1, B1} in its first and second segments of the vector; at the same time, node C 206 and node D 208 perform similar data exchanges. During this period, the opposite operations are performed between node B 204 and node A 202 for the other half of the vector (i.e., from B2 to A2 and from B3 to A3), and between node D 208 and node C 206 for the other half of the vector (i.e., from D2 to C2 and from D3 to C3);

[0063] In step (ii), the remaining "fan-in" of each partial result can continue up the tree to form the complete result of the first fragment at node D, the second fragment at node B, the third fragment at node A, and the fourth fragment at node C. Steps (i) and (ii) are essentially "fan-in" of a binary tree with a radix of two, and use four concurrent trees, where each node is the root of a tree. In step (iii), the RA begins the tree-based fan-out process and copies the partial results "down the tree". In step (iv), the RA continues the tree-based fan-out process, and in step (v), the RA ensures that the complete result is stored on each node in the distributed computing system.

[0064] Tree-based all-reduce methods were designed to scale to hundreds or even thousands of nodes in a period when the point-to-point latency was low relative to the time to send a unit of data. Recently, the balance of system performance has shifted, making the point-to-point latency large relative to the time to transmit a moderately sized payload. This change has created an opportunity for new methods. Further, it is also desirable to develop an all-reduce mechanism that can utilize the actual buffer size of the NIC with a simple NIC-based flow control scheme.

[0065] The above embodiment is described by taking four distributed nodes as an example, and those skilled in the art should understand that the number of nodes in the distributed network can be other numbers. Similarly, other algorithms besides the RA algorithm can also be used.

[0066] Currently, there is a growing demand for larger models, which are used in artificial intelligence training, such as deep neural network training. Gradient descent is used as an optimization technique to improve the accuracy of large models. Gradient descent requires collective communication between computing nodes. The Allreduce algorithm is an iterative algorithm used in the training process.

[0067] Figure 4 FIG. 3 is a schematic diagram of a 3*5 grid in one embodiment of the present disclosure. Figure 4 As shown, for a distributed network 400 with 15 (2*7+1) computing nodes. Figure 4 In the 2D network topology shown, including the odd-sized grid, it is impossible to form a bidirectional ring containing all nodes. The inventor breaks the traditional thinking and excludes one node in the m×n odd-sized grid, leaving mn-1 nodes to create a bidirectional ring. This ensures that each node has a bidirectional connection. Figure 4 As shown, among the 15 computing nodes, 2*7 (N=7) computing nodes are selected to form a bidirectional ring, where 41 represents an excluded node, 42 represents a merged node; 411 represents a first direction of the bidirectional ring, 412 represents a second direction of the bidirectional ring; 420 represents a bidirectional connection between an excluded node and a merged node (such as Figure 4 430 represents the unused connections between computing nodes in the bidirectional ring full reduction calculation.

[0068] First, 2N computing nodes are determined from the 2N+1 computing nodes based on a linear time algorithm, and the 2N computing nodes form a Hamiltonian cycle. In an m×n odd-sized grid excluding a corner node, a Hamiltonian cycle is found based on a linear time algorithm. That is, a path is found that passes through each node exactly once to form a cycle. Such a path can be found in a reasonable time to ensure communication between all nodes. The linear time algorithm, for example, uses an approximate algorithm, such as a greedy algorithm: starting from a vertex, each time a vertex adjacent to the current vertex and not yet visited is selected until all vertices are visited or it is impossible to continue. Genetic algorithm: simulates the evolutionary process of natural selection and finds an approximate Hamiltonian cycle through iterative improvement. Simulated annealing algorithm: a probabilistic algorithm that allows some inferior solutions to be accepted during the search process to avoid falling into a local optimal solution.

[0069] It is also possible to use brute-force search to find the Hamiltonian cycle, that is, to try all possible arrangements of vertices and check whether each arrangement forms a Hamiltonian cycle.

[0070] Then, in order to schedule the AllReduce processes in parallel, two unidirectional loops are used. Figure 4 In the example of FIG. 1 , two unidirectional rings are marked with thick solid arrows and thin solid arrows. Such a scheduling method can improve communication efficiency, so that data can be transmitted between multiple rings simultaneously.

[0071] Then, the excluded node’s data is transmitted to the nodes in the formed bidirectional ring using any two available bidirectional connections between the excluded node and its neighbors. In this way, the excluded node’s data can be transmitted to other nodes in the ring through these connections, ensuring that all nodes can participate in the AllReduce communication process. This method can make full use of existing connections and improve communication efficiency.

[0072] It should be pointed out that although Figure 4 Two merging nodes 42 are shown, which are used for full reduction calculations in the first direction and the second direction respectively. Those skilled in the art should understand that only one merging node may be used for full reduction calculations in the two directions respectively.

[0073] Figure 5A A flow chart showing a distributed data processing method according to another embodiment of the present disclosure is shown. This embodiment provides an implementation for optimization in all-reduce calculation.

[0074] As shown in FIG. 5 , S502 , 2N nodes are determined to establish a dual ring and divide data.

[0075] 2N computing nodes are determined from the 2N+1 computing nodes, the 2N computing nodes form a bidirectional ring, the remaining computing nodes are used as excluded nodes, and the merged nodes are determined. The data of each computing node is divided into a first part and a second part, and each part is divided into 2N parts.

[0076] For example, Figure 5B Show Figure 5A Schematic diagram of a grid in the embodiment. Figure 5B As shown in the figure, the network includes 9 computing nodes 1-9. Nodes 1, 2, 3, 4, 5, 6, 8 and 9 are selected to form a bidirectional ring, and node 7 is excluded. After establishing a bidirectional ring consisting of nodes (1, 2, 3, 6, 9, 8, 5, 4), each node divides its gradient data into the first part and the second part, and each part is further divided into 8 parts, as shown in the figure. Figure 5C Different from the traditional ring-based all-reduce method, node 4 is designated as a merge node, responsible for aggregating data from its parent node (node ​​5) and the excluded node (node ​​7).

[0077] S504, ReduceScatter process in the first direction.

[0078] In the ReduceScatter process, the exclusion node (for example, node 7) sends the second copy of the first part (i.e., the forward part) to the merge node (for example, node 4) at the first time step t1, and the merge node aggregates the received second copy of the data. In the subsequent second time step (t2) to the 2Nth time step (t8), the merge node (for example, node 4) starts the ring-based AllReduce, and transmits the aggregated second copy of the data from the parent node (for example, node 5) and the exclusion node (for example, node 7) to node 1 in each time step, while receiving and aggregating the first copy of the data from these two nodes. This process lasts for 2N (for example, 8) time steps, and eventually each node in the ring has 1 / 8 of the complete aggregated data. Among them, Figure 5D Show Figure 5B Schematic diagram of data flow at each time step in the embodiment.

[0079] S506, AllGather (collective collection) stage of the first direction.

[0080] In the AllGather phase, the link from node 7 to node 4 is used, and node 4 retrieves all aggregated partial data. This process also takes 8 time steps to ensure that all nodes can obtain the final complete aggregation results.

[0081] S508: AllReduce processing in the second direction.

[0082] Specifically, it includes: performing a reduction scatter process in the second direction, so that each of the 2N computing nodes obtains one 2N part of the second part of the data through 2N time steps, wherein at each time step, the merging node receives and aggregates data from the excluded node and the parent node;

[0083] Perform collective collection processing in the second direction, so that each of the 2N computing nodes obtains all aggregated results of the second part through 2N time steps.

[0084] For the Allreduce for the second direction (opposite to the first direction), the same procedure is followed, but this time using node 8 as the merge node.

[0085] In the above embodiment, in terms of network topology scalability, the problem of lack of Hamiltonian circuits in odd-sized grids is solved by forming two unidirectional rings (in opposite directions) on N-1 chips, thereby improving sensitivity to network topology. In terms of AllReduce bandwidth utilization, by optimizing AllReduce scheduling communication, both the ReduceScatter and AllGather stages can be completed in (N-1) hops. This makes the number of hops of the AllReduce algorithm match that of the bidirectional ring AllReduce algorithm used in even-sized grids, thereby improving efficiency.

[0086] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0087] Figure 6 An exemplary computer system equipped with a NIC for implementing a full-reduce operation based on a network interface in one embodiment of the present disclosure is shown. In this example, the computer system 600 may include a processor 602, a memory 604, and a storage device 606. The computer system 600 may be coupled to a peripheral input / output (I / O) user device 640, such as a display device 641, a keyboard 642, and a mouse device 643. The storage device 606 may store instructions for an operating system. One or more applications may operate on the operating system.

[0088] The computer system 600 may be equipped with a host interface for coupling to a NIC 630. The NIC 630 may include an ALU 632, a cache 634, and an address translation unit 636. The NIC 630 may be coupled to a network 610 via a network interface. The NIC 630 may include a full-reduce operation logic module 620 to perform the methods and / or processes described in the present disclosure. The full-reduce operation logic module 620 may include a communication unit 621 and a full-reduce calculation unit 622.

[0089] In one embodiment, the full-reduce operation logic module 620 may also be located in the computer system 600, and the NIC 630 is mainly used for communication work.

[0090] The communication unit 621 may send and / or receive data packets from other computing nodes in the network 610. The all-reduce computing unit 622 may implement the all-reduce algorithm described in the present disclosure.

[0091] The communication unit 621 can receive incoming packets from other nodes in the network 610. The incoming packets can be stored in one or more buffers in the cache 634. This can be achieved by the ALU 632. The ALU 632 can perform arithmetic operations on batches of data and organize the data, thereby accelerating the full-reduce operation in the NIC 630. The cache 634 can optimize the latency involved in combining and forwarding the results to other nodes in the network 610. The address translation unit 636 can enable regions to be marked as cacheable to use the local cache 634, thereby separating cacheable memory from non-cacheable memory. In other words, the address translation unit 636 can implement a tagging operation that can identify which operations can target certain regions in the cache 634. Such tagging / tagging operations are desirable to simplify software management of the cache to achieve a consistent state.

[0092] An embodiment described in the present disclosure may provide an apparatus. The apparatus may include: a network interface for transmitting and receiving data packets through a network; a memory including one or more buffers; an arithmetic logic unit for performing arithmetic operations to organize and combine the data packets; and a circuit. The circuit is used to perform the following operations: receiving data packets from a network via a network interface; aggregating the received data packets into one or more buffers via an arithmetic logic unit; and transmitting the aggregated data packets to one or more computing nodes in the network via the network interface to accelerate batch data all-reduce operations.

[0093] Figure 7 FIG. 1 is a schematic diagram of a distributed data processing device in one embodiment of the present disclosure. Figure 7 As shown, the device includes: an exclusion node determination module 71, used to determine 2N computing nodes from 2N+1 computing nodes, and use the remaining computing nodes as exclusion nodes; a merge node determination module 72, used to determine a merge node from the 2N computing nodes, and the merge node is connected to the exclusion node to merge the data of the exclusion node; a reduction processing module 73, used to form a bidirectional ring with the 2N computing nodes, and perform bidirectional ring (Ring) all-reduce (AllReduce) data processing, wherein, during the processing of each time step, the merge node receives and aggregates data from the exclusion node and from the parent node; wherein N is a natural number.

[0094] The technical solution disclosed in the present invention can be applied to technical fields such as deep learning and machine learning, high-performance computing, large-scale data processing and cloud computing.

[0095] In the embodiments of the present disclosure, a unique design for odd-numbered grids is proposed: in order to overcome the challenge of not being able to form a Hamiltonian cycle in an odd-numbered grid, a solution is proposed to construct two unidirectional rings in opposite directions through N-1 nodes. This design effectively overcomes the limitations of traditional bidirectional ring AllReduce under odd-numbered nodes.

[0096] Flexibility and compatibility: The design of the present disclosure allows it to be applicable to various grid topologies, not just odd-numbered grids. This flexibility provides more architectural options for distributed training, helping to better utilize computing resources. In terms of compatibility, it can be seamlessly integrated with existing DNN collective communication libraries (such as NCCL and Horovod), leveraging the bidirectional ring AllReduce implementation that these frameworks have established. This compatibility lowers the threshold for applying new algorithms, making it easier for researchers and engineers to adopt new technologies.

[0097] In summary, the technical solution disclosed in this disclosure has shown remarkable innovation in design, optimization, flexibility, compatibility and practical utility, etc. These features make it an important solution for processing AllReduce problems in distributed systems, especially in odd-numbered grid topologies, and promote the development of related fields.

[0098] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0099] Refer to the following Figure 8 An electronic device 800 according to this embodiment of the present invention will be described. Figure 8 The electronic device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0100] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0101] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 As shown in S202, 2N computing nodes are determined from 2N+1 computing nodes, and the remaining computing nodes are used as exclusion nodes; S204, a merge node is determined from the 2N computing nodes, and the merge node is connected to the exclusion node to merge the data of the exclusion node; S206, a bidirectional ring is formed with 2N computing nodes, and bidirectional ring full reduction data processing is performed, wherein, during the processing of each time step, the merge node receives and aggregates data from the exclusion node and from the parent node.

[0102] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0103] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0104] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0105] The electronic device 800 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0106] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0107] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0108] A program product for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0109] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0110] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0111] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0112] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0113] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0114] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0115] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0116] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A distributed data processing method, applied to a distributed network, characterized in that: The method comprises: Determine 2N computing nodes from the 2N+1 computing nodes, and use the remaining computing nodes as excluded nodes; Determine a merging node from the 2N computing nodes, wherein the merging node is connected to the exclusion node to merge data of the exclusion node; The 2N computing nodes form a bidirectional ring and perform bidirectional ring full reduction data processing, wherein, during the processing of each time step, the merging node receives and aggregates data from the excluded node and from the parent node; Wherein, N is a natural number.

2. The method according to claim 1, characterized in that The merging nodes include a first merging node and a second merging node, which correspond to merging nodes performing full reduction data processing in a first direction and a second direction, respectively.

3. The method according to claim 1, characterized in that The merging node is a computing node used to process the full reduction data processing in the first direction and the second direction.

4. The method according to any one of claims 1 to 3, characterized in that The data of the 2N+1 computing nodes are divided into a first part and a second part; and the first part and the second part are respectively divided into 2N parts; The forming of a bidirectional ring with the 2N computing nodes and performing bidirectional ring full reduction data processing includes: Performing a reduce-scatter process in a first direction, so that each of the 2N computing nodes obtains one 2Nth of the first portion of data through 2N time steps, wherein at each time step, the merging node receives and aggregates data from the excluded node and the parent node; Performing collective collection processing in the first direction, so that each of the 2N computing nodes obtains all aggregated results of the first part through 2N time steps; Performing a reduction-scatter process in the second direction, so that each of the 2N computing nodes obtains one 2Nth of the second portion of data through 2N time steps, wherein at each time step, the merging node receives and aggregates data from the excluded node and the parent node; Perform collective collection processing in the second direction, so that each of the 2N computing nodes obtains all aggregated results of the second part through 2N time steps.

5. The method according to claim 4, characterized in that Determining 2N computing nodes from the 2N+1 computing nodes includes: 2N computing nodes are determined from the 2N+1 computing nodes based on a linear time algorithm, and the 2N computing nodes form a Hamiltonian cycle.

6. The method according to claim 5, characterized in that The linear time algorithm includes a genetic algorithm, a probabilistic algorithm or a greedy algorithm.

7. The method according to claim 1, characterized in that The distributed network includes the 2N+1 computing nodes.

8. A distributed data processing device, characterized in that: include: An exclusion node determination module, used to determine 2N computing nodes from the 2N+1 computing nodes, and use the remaining computing nodes as exclusion nodes; A merging node determining module, used to determine a merging node from the 2N computing nodes, wherein the merging node is connected to the exclusion node to merge data of the exclusion node; A reduction processing module, used to form a bidirectional ring with the 2N computing nodes, and perform bidirectional ring full reduction data processing, wherein, when processing each time step, the merging node receives and aggregates data from the excluded node and from the parent node; Wherein, N is a natural number.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the distributed data processing method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed data processing method according to any one of claims 1 to 7 is implemented.

Images