Communication method, device, equipment, storage medium and program product of computing cluster

Abstract

The application provides a communication method, device and equipment of a computing cluster, a storage medium and a program product. The method comprises the following steps: selecting a plurality of data segments from data stored in a plurality of graphic processors; performing reduction calculation on the selected data segments in sequence along a positive direction of a horizontal communication ring to obtain a plurality of reduction data segments; dividing the reduction data segments into a plurality of first reduction channel blocks; for each first reduction channel block, performing reduction calculation between the graphic processors in sequence along a positive direction of a vertical communication ring to obtain a second reduction channel block; in the case that the second reduction channel block is obtained by reduction calculation at any graphic processor, distributing the second reduction channel block in sequence along a reverse direction of the vertical communication ring, and distributing the second reduction channel block in sequence along a reverse direction of the horizontal communication ring. Through the application, the communication efficiency of the cluster can be improved, and the computing speed of the computing cluster can be improved.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a communication method, apparatus, device, storage medium, and program product for a computing cluster. Background Technology

[0002] With the development of deep learning technology, the model parameter size of deep learning models, such as large language models, is growing exponentially, and the computing power and memory capacity of a single device can hardly meet the training requirements of large language models.

[0003] In related technologies, the training dataset is distributed across multiple Graphics Processing Units (GPUs) for parallel processing. During training, each GPU independently calculates the gradient update values ​​of the model parameters based on its local data. Then, the local gradients calculated by all GPUs are aggregated through a serial communication link, and the aggregated complete parameters are synchronized back to each GPU. This communication method transmits data between GPUs in the form of data packets, and each communication step requires waiting for the previous data packet to complete its calculation, resulting in insufficient computing speed of the computing cluster. Summary of the Invention

[0004] This application provides a communication method, apparatus, device, storage medium, and program product for a computing cluster, which can achieve partial overlap of execution time in different communication stages, thereby improving the computing speed of the computing cluster.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a communication method for a computing cluster. The computing cluster includes multiple computing nodes, each computing node includes multiple graphics processors (GPUs), and each GPU includes multiple storage areas. Multiple storage areas within the same computing node form a horizontal communication ring, and multiple storage areas in different computing nodes form a vertical communication ring. The storage areas forming the horizontal communication ring have the same first sequence number within the GPUs, and the GPUs containing the storage areas forming the vertical communication ring have the same second sequence number within the computing nodes, and the first sequence numbers are also the same. The method includes: Multiple data segments are selected from the data stored in the multiple graphics processors, and the selected data segments are sequentially reduced along the positive direction of the horizontal communication ring to obtain multiple reduced data segments. The first sequence number of the storage area occupied by the selected data segments is different for different graphics processors. The reduced data segment is divided into multiple first reduced channel blocks. For each first reduced channel block, reduction calculations are performed sequentially between the graphics processors along the positive direction of the vertical communication ring to obtain a second reduced channel block. If the second reduced channel block is obtained by any of the graphics processors through reduction calculation, the second reduced channel block is distributed sequentially along the reverse direction of the vertical communication ring; If any of the graphics processors receives the second reduction channel block, the second reduction channel block is distributed sequentially in the reverse direction of the transverse communication ring.

[0006] This application provides a communication device for a computing cluster. The computing cluster includes multiple computing nodes, each computing node includes multiple graphics processors (GPUs), and each GPU includes multiple storage areas. Multiple storage areas within the same computing node form a horizontal communication ring, and multiple storage areas in different computing nodes form a vertical communication ring. The storage areas forming the horizontal communication ring have the same first sequence number within the GPUs, and the GPUs containing the storage areas forming the vertical communication ring have the same second sequence number within the computing nodes, and the first sequence numbers are also the same. The device includes: The horizontal reduction module is used to select multiple data segments from the data stored in the multiple graphics processors, and to perform reduction calculations on the selected data segments sequentially along the positive direction of the horizontal communication ring to obtain multiple reduced data segments. The first sequence number of the storage area occupied by the selected data segments is different for different graphics processors. The vertical reduction module is used to divide the reduced data segment into multiple first reduction channel blocks. For each first reduction channel block, reduction calculations are performed sequentially between the graphics processors along the positive direction of the vertical communication loop to obtain a second reduction channel block. The vertical distribution module is used to distribute the second reduced channel block sequentially along the reverse direction of the vertical communication ring when the graphics processor calculates the second reduced channel block. A horizontal distribution module is used to distribute the second reduction channel block sequentially along the reverse direction of the horizontal communication ring when any of the graphics processors receives the second reduction channel block.

[0007] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the communication method of the computing cluster provided in the embodiments of this application.

[0008] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the communication method of the computing cluster provided in this application.

[0009] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the communication method of the computing cluster provided in this application.

[0010] The embodiments of this application have the following beneficial effects: By constructing a horizontal communication ring between the storage areas of multiple graphics processors within the same computing node, and a vertical communication ring between multiple storage areas of different computing nodes, data segments are reduced within the node along the horizontal communication ring (i.e., horizontal reduction). The data segments obtained from the horizontal reduction (the first reduced data segment) are then divided, and the divided reduced data segments are reduced along the vertical communication ring (i.e., vertical reduction). The data segments obtained from the vertical reduction (the second reduced channel block) are then distributed sequentially to the storage areas of multiple graphics processors along the reverse direction of both the vertical and horizontal communication rings. Since the reduction calculation is an element-wise computation of a vector matrix, and the data elements in the reduction calculation are mutually independent, this method allows the current computation in the computing cluster to proceed without waiting for the previous data packet to complete its computation. This results in partial overlap of execution times between different communication stages, improving the computational efficiency of the computing cluster. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the computing cluster system architecture provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the communication method of the computing cluster provided in the embodiments of this application. Figure 1 ; Figure 4 This is a flowchart illustrating the communication method of the computing cluster provided in the embodiments of this application. Figure 2 ; Figure 5 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 1 ; Figure 6 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 2 ; Figure 7 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 3 ; Figure 8 This is a flowchart illustrating the communication method of the computing cluster provided in the embodiments of this application. Figure 3 ; Figure 9 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 4 ; Figure 10 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 5 ; Figure 11 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 6 ; Figure 12 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 7 ; Figure 13 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 8 ; Figure 14 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 9 ; Figure 15 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 10 ; Figure 16 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 10 one; Figure 17 This is a flowchart illustrating the communication method of the computing cluster provided in the embodiments of this application. Figure 4 ; Figure 18 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 10 two; Figure 19 This is a schematic diagram illustrating the data flow principle of the communication method for the computing cluster provided in this application embodiment. Figure 10 three; Figure 20 This is a schematic diagram illustrating the data flow principle of the communication method for a computing cluster in the application scenario provided in this application embodiment. Figure 1 ; Figure 21 This is a schematic diagram illustrating the data flow principle of the communication method for a computing cluster in the application scenario provided in this application embodiment. Figure 2 ; Figure 22This is a schematic diagram illustrating the data flow principle of the communication method for a computing cluster in the application scenario provided in this application embodiment. Figure 3 ; Figure 23 This is a schematic diagram illustrating the data flow principle of the communication method for a computing cluster in the application scenario provided in this application embodiment. Figure 4 .

[0012] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0015] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0017] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0018] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0019] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0020] 1) A computing cluster refers to a parallel computing system formed by tightly connecting multiple independent computing devices (such as servers, workstations, etc.) through a high-performance communication network. In the embodiments of this application, a computing cluster specifically refers to a hardware environment used for distributed deep learning training. Externally, it appears as a unified pool of computing resources, and internally, it achieves high-speed data exchange between nodes through a specific network topology.

[0021] 2) A computing node is a physical unit (such as a server) in a computing cluster that has an independent operating system and computing capabilities. Each computing node typically includes a central processing unit (CPU), memory, and multiple accelerators (such as GPUs) for parallel computing. In the embodiments of this application, multiple GPUs are deployed within a computing node. The GPUs within a node communicate with each other via a high-speed interconnect bus, and the nodes communicate with each other via a network interface card (NIC).

[0022] 3) A Graphics Processing Unit (GPU) is a specially designed electronic circuit that excels at handling large-scale parallel computing tasks. In the embodiments of this application, the GPU serves as the primary computing unit, responsible for executing the forward propagation, backward propagation, and gradient calculation and updating of the deep learning model. It should be noted that the GPU mentioned in this application can also be replaced by other types of artificial intelligence (AI) acceleration chips, such as embedded neural network processors (NPUs) and tensor processors.

[0023] 4) The Ring-AllReduce Algorithm is an algorithm used in distributed systems to reduce data (such as gradients) distributed across different nodes, performing operations like summation or mean calculation, and synchronizing the results. In this algorithm, all participating devices logically form a unidirectional closed loop. Each device only sends data to its logical next neighbor and receives data from its previous neighbor, thus efficiently utilizing network bandwidth.

[0024] 5) Reduce-Scatter refers to a type of collective communication operation. It typically includes two parts: reduction computation and distribution. Its function is to reduce (e.g., sum) the complete data vectors held by all participants at corresponding positions. However, the result is not held completely by all participants; instead, it is split, and each participant ultimately holds only a portion of the reduced result (usually the first part). The participant holds the result vector of the first (Divide into blocks), then aggregate the reduction results held by each participant, and distribute the aggregated results to each participant. At this point, each participant has obtained the complete reduction result.

[0025] 6) A data partition (or part) refers to a unit formed by logically dividing the complete data to be communicated (e.g., the gradient tensor of the entire model) according to the number of devices participating in the communication. In the standard ring algorithm, if there is... With multiple GPUs involved, the complete data is typically divided evenly. There are 10 data segments, each corresponding to a specific location on the ring.

[0026] 7) A channel block is a data unit that is further divided from a "data segment (part)". The division is usually based on the physical bandwidth capacity of the communication channel or the specific number of communication steps required. Channel blocks are introduced to adapt to heterogeneous network environments. For example, when transmitting data in a vertical channel (between nodes) with low bandwidth, more granular data control may be required, or to accommodate the number of concurrent communication channels.

[0027] 8) A data chunk / slice is the smallest unit of data transmission after further fine-grained division of a channel block. In the embodiments of this application, the slice is the basic granularity for realizing pipeline transmission. By dividing a channel block into multiple consecutive slices, the communication engine can transmit data in a more efficient manner. While processing the first slice, the computing engine is also processing the second slice. This allows for the creation of multiple slices, maximizing the overlap between computation and communication time and reducing transmission latency.

[0028] 9) Pipelined Reduction is a streaming processing technique that decomposes the reduction operation of a large dataset into multiple fine-grained data slices. In this technique, data no longer passes through each processing stage sequentially in a completely blocked manner. Instead, as soon as a data slice completes the operation of its current processing stage (e.g., receiving or vertical reduction calculation), it is immediately pushed to the next processing stage (e.g., sending to the next node or entering a horizontal communication link). At the same time, the next data slice immediately enters the current processing stage. This allows data from different slices to be in different processing states such as transmission and computation simultaneously. It achieves overlap between communication transmission and computation tasks on the time axis, as well as deep overlap of communication links in different dimensions (such as vertical and horizontal), in order to maximize the system's concurrent throughput.

[0029] With the development of deep learning technology, the model parameter size of deep learning models, such as large language models, is growing exponentially. The computing power and memory capacity of a single device are insufficient to meet the training requirements of large language models. To meet the data processing needs of deep learning models, a data parallelism strategy is typically adopted, which involves splitting the training dataset and distributing it to multiple computing nodes (such as GPUs) for parallel processing. During training, each node independently calculates the gradient update values ​​of the model parameters based on its local data. However, to ensure the correct convergence of the model globally, the local gradients calculated by all nodes must be aggregated and the aggregated complete parameters must be synchronized back to each computing node to ensure that all nodes perform calculations based on a completely consistent copy of the model in the next iteration.

[0030] In related technologies, the global synchronization of gradients mentioned above is achieved through the AllReduce algorithm. AllReduce is an efficient aggregate communication primitive. In a standard Ring-AllReduce process, the communication process is decomposed into two stages: a reduction stage and a distribution stage. Each stage contains multiple data transmission steps. Each GPU sequentially sends its local partial gradient data to the next GPU and receives data from the upstream GPU for accumulation. Ultimately, the local gradients scattered across various nodes are aggregated into a global gradient, which is then distributed back to all nodes, thereby completing parameter synchronization.

[0031] However, the AllReduce algorithm described above suffers from slow computation speed. First, the AllReduce process typically has strict serial dependencies, meaning the next communication or computation operation can only begin after a complete data packet has been received and the previous operation completed. This "stop-and-wait" mechanism causes computation threads to remain idle while waiting for data transmission, failing to fully utilize hardware computing power and creating significant pipeline bubbles. Second, in large-scale cluster environments, the interconnect bandwidth between GPUs within a compute node is usually higher than the network bandwidth between GPUs between compute nodes, such as Ethernet or InfiniBand. This heterogeneous bandwidth characteristic makes slow cross-node communication a bottleneck in the entire synchronization process. High-speed links are forced to wait for the completion of low-speed links, further exacerbating communication congestion and latency, limiting the overall cluster training efficiency (cluster computation speed).

[0032] This application provides a communication method, apparatus, device, readable storage medium, and program product for a computing cluster, which can achieve partial overlap of execution time in different communication stages, thereby improving the computing speed of the computing cluster.

[0033] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of a computing cluster provided in this application embodiment. The computing cluster 100 includes multiple computing nodes 110 (or communication nodes). Each computing node 110 includes multiple graphics processors 120, and each graphics processor 120 includes multiple storage areas. The storage areas can be regions partitioned from the video memory of the graphics processors. The storage areas of the graphics processors 120 within the same computing node 110 are connected via high-bandwidth intra-node interconnect channels, such as connecting to a Peripheral Component Interconnect Express (PCIe) bus, forming a horizontal communication ring. Figure 1 Only the horizontal communication ring containing the first and last memory areas is shown; each memory area is set within the horizontal communication ring. Figure 1 (Not shown). Different computing nodes can be connected to a switch via network interface cards. In addition, the computing cluster 100 also includes a control device 140 (…). Figure 1(Not shown), control device 140 is used to control the process of data transmission between storage areas. It should be noted that although the graphics processors 120 of different computing nodes 110 are physically connected to the switch, logically they are equivalent to any graphics processor 120 in any computing node 110 being able to establish a data transmission channel with the graphics processors 120 in other computing nodes 110. A vertical communication ring is formed between the storage areas of different computing nodes 110 (…). Figure 1 Only the vertical communication ring between the first memory area of ​​the first graphics processor 120 and the last graphics processor 120 in each compute node 110 is shown, with each memory area set in the vertical communication ring. Figure 1 (Not shown). Data communication between graphics processors 120 can be scheduled and controlled by a control element, which can be implemented as a server. A high-speed serial bus refers to a transmission rate of 32GT / s or higher (32 billion transmissions per second), such as a point-to-point serial communication link in the PCIe 5.0 standard. In a specific embodiment, this high-speed serial bus supports differential signal transmission. It should be noted that "horizontal" and "vertical" are logical divisions based on the interconnection within computing nodes and the interconnection between nodes, and do not refer to specific physical address divisions.

[0034] In some embodiments, distributed training of large language models can be performed in the computing cluster provided in this application. The graphics processors (GPUs) within each computing node perform reduction calculations on the trained gradient data along the horizontal communication ring, so that each GPU holds only a portion of the aggregated data segment within the node (reduced data segment). Then, the GPUs divide the reduced data segment into a first reduced channel block matching the network bandwidth between computing nodes, and perform reduction calculations on the first reduced channel block along the vertical communication ring to obtain a second reduced channel block. The data flow is then distributed in reverse along both the vertical and horizontal communication rings, and the next distribution can begin immediately after the second reduced channel block completes one distribution, without waiting for other channel blocks within the same data segment to finish distributing or reducing, thus improving the computing speed of the computing cluster.

[0035] In some embodiments, the electronic device may be Figure 1 The control device 140 for controlling data transmission is installed within the computing node 110. Figure 1 (not shown), see Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2The illustrated electronic device includes at least one processor 143, a memory 141, and at least one network interface 146. The various components of the electronic device are coupled together via a bus system 144. It is understood that the bus system 144 is used to implement communication between these components. In addition to a data bus, the bus system 144 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 144.

[0036] Processor 143 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0037] The memory 141 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 141 may optionally include one or more storage devices physically located away from the processor 143.

[0038] Memory 141 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 141 described in this application embodiment is intended to include any suitable type of memory.

[0039] In some embodiments, memory 141 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0040] Operating system 142 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks. In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A communication device 145 for a computing cluster stored in memory 141 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a horizontal reduction module 1451, a vertical reduction module 1452, a vertical distribution module 1453, and a horizontal distribution module 1454. These modules are logically linked and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0041] See Figure 3 , Figure 3 This is a flowchart illustrating the communication method of a computing cluster provided in this application embodiment. The computing cluster includes multiple computing nodes, each computing node includes multiple graphics processors (GPUs), and each GPU includes multiple storage areas. Multiple storage areas within the same computing node form a horizontal communication ring, and multiple storage areas across different computing nodes form a vertical communication ring. The storage areas forming the horizontal communication ring share the same first sequence number within the GPUs, and the GPUs containing the storage areas forming the vertical communication ring share the same second sequence number within the computing nodes, and also share the same first sequence number. The following will combine... Figure 3 The steps shown are explained. The cluster communication method includes the following steps 101 to 104.

[0042] In this embodiment, the aforementioned horizontal and vertical communication rings constitute a two-dimensional dual-ring structure, logically arranging all participating graphics processors into an R×C two-dimensional matrix network. In this structure, the storage areas of the C graphics processors (within the same computing node) in each row constitute multiple horizontal communication rings, and the storage areas of the R graphics processors in each column constitute multiple vertical communication rings. That is, the horizontal communication rings are logical loops built upon the high-speed interconnect bus between graphics processors within the same computing node, while the vertical communication rings are logical loops built upon the interconnect bus between graphics processors in different computing nodes. This design decomposes the global communication task on a single large ring in related technologies into two independent levels interwoven with rows and columns, effectively isolating the high-speed interconnect within nodes from the low-speed network between nodes, thereby improving communication efficiency in heterogeneous bandwidth environments and ultimately increasing the distributed training speed of the computing cluster. In this embodiment, the first sequence number corresponds to the logical rank of the storage area within the graphics processor, and the second sequence number corresponds to the logical rank of the graphics processor within the computing node.

[0043] In step 101, multiple data segments are selected from the data stored in multiple graphics processors, and the selected data segments are reduced sequentially along the positive direction of the horizontal communication ring to obtain multiple reduced data segments.

[0044] The first sequence number of the storage area occupied by the selected data segment differs for different graphics processors.

[0045] Here, reduction computation refers to an operation that aggregates information after communication, with the aim of reducing data vectors distributed across different nodes by performing specific operations (such as summation, averaging, and finding the maximum value), so that each node obtains a portion of the complete result vector of its respective level. In this embodiment, the reduction computation in step 101 can be referred to as the horizontal reduction stage.

[0046] In some embodiments, the data stored in multiple graphics processing units (GPUs) can be constructed as follows: During the backpropagation training of a deep learning model, each GPU independently computes model parameter gradients based on its loaded training data batch. These gradient data are typically stored as tensors in the high-bandwidth memory (HBM) of the multiple GPUs, i.e., the storage area in this embodiment, and the gradient data of each GPU are logically aligned. Data segments are selections of tensors; in this embodiment, the number of data segments is typically the same as the number of GPUs in a computing node to avoid congestion.

[0047] As an example, reducing the selected data segment along the positive direction of the horizontal communication ring can be achieved as follows: using a ring communication algorithm, assuming there are [data points] in the horizontal ring. Each graphics processor logically divides the complete gradient data into... The data segment. In the first... In round-robin communication, the first The No. 1 graphics processor will hold the current No. 1 The data segment is transmitted to the first The graphics processor receives data from the first graphics processor. The data segments of each graphics processor are accumulated. The phrase "different first sequence numbers for different graphics processors" means that in the first round of communication, GPU 0 sends the 0th data segment, GPU 1 sends the 1st data segment, and so on, thereby achieving load balancing and avoiding communication congestion.

[0048] In some embodiments, see Figure 4 The step 101, which involves selecting multiple data segments from the data stored in multiple graphics processors and performing reduction calculations on the selected data segments sequentially along the positive direction of the horizontal communication ring, can be achieved through the following steps 1011 to 1012.

[0049] In step 1011, if the nth graphics processor does not receive the transmitted data segment in its storage area, the nth data segment is transmitted to the (n+1)th graphics processor's storage area along the positive direction of the horizontal communication loop.

[0050] Here, "the situation where no transmitted data segment is received" typically refers to the time step (Time Step 0) at the start of the reduction computation (lateral reduction) process on the lateral communication ring. At this stage, since data flow on the lateral communication ring has not yet begun, the receive buffer of each graphics processing unit (GPU) is empty, thus triggering the initial transmission logic. The "nth data segment" can be the data segment corresponding to the first index of the current nth graphics processing unit after the complete data to be reduced (e.g., the model gradient tensor) is logically divided into N parts (N being the total number of GPUs in a computing node). For example, part size represents the number of bytes to be processed for a single rank, i.e., the tensor size. The tensor size part size satisfies partsize = full tensor size / nranks, where nranks refers to the number of nodes in the cluster, rank refers to the computing node number, and full tensor size refers to the complete tensor size. Unlike traditional algorithms that divide data by a fixed segment, where a segment refers to a fixed data segment / fragment size, this embodiment directly associates part size with the number of ranks to ensure balanced data distribution. Channel size refers to the amount of data within each rank that is independently processed by a specific storage area. The following relationship is satisfied: channelsize = part size / nchannels, where nchannels refers to the number of GPU threads, channel refers to the channel number, and channel size refers to the channel block size.

[0051] As an example, see Figure 5 , Figure 5 This is a schematic diagram illustrating the data flow when the nth graphics processor's memory does not receive the transmitted data segment. The horizontal communication ring consists of four graphics processors (hereinafter, for simplicity, n=4 graphics processors are referred to as Rank N1 to Rank N4), and the total gradient data to be reduced is 400MB. The system divides the data into four 100MB segments: segment 1, segment 2, segment 3, and segment 4. In step 1011, as an initial action: Rank N1 reads segment 1 from its local memory and sends it to Rank N2; Rank N2 reads segment 2 from its local memory and sends it to Rank N3; Rank N3 reads segment 3 from its local memory and sends it to Rank N4; Rank N4 reads segment 4 from its local memory and sends it to Rank N1. At this time, each graphics processor "transmits" its corresponding data segment to the next memory area of ​​its horizontal communication ring, preparing for subsequent reduction calculations.

[0052] In step 1012, when the transmitted data segment is received in the storage area of ​​the nth graphics processor, the received data segment is reduced by the local data segment, and the reduced data segment is transmitted to the storage area of ​​the (n+1)th graphics processor along the positive direction of the horizontal communication ring.

[0053] Here, "reduction computation" performs mathematical aggregation operations on data from different sources at the same logical location. In deep learning training scenarios, reduction computation is typically an element-wise floating-point addition operation, designed to combine gradient values ​​computed on different devices.

[0054] Here, because the gradient data of the model is stored in the high-speed video memory of multiple graphics processors in the form of tensors during the distributed training of the large language model, each graphics processor stores a channel block locally before the data segment is transmitted.

[0055] In some embodiments, this step constitutes a time step (Time Step 1 to N-2) in the intermediate iteration process of the ring algorithm. Specifically, when the nth graphics processor detects a data write completion signal from the upstream node (the (n-1)th node), its arithmetic logic unit (ALU) is triggered. The ALU reads the input data segment from the receive buffer and simultaneously reads the local data segment from the corresponding offset address in the local video memory, performing an addition operation. The result is not only updated to the local video memory, but is also (or immediately thereafter) fetched by the Direct Memory Access (DMA) engine and forwarded to the downstream (n+1)th graphics processor.

[0056] For example, continuing with the above embodiment where each computing node contains 4 GPUs, proceed to the loop logic in step 1012: see Figure 6 In the second iteration, Rank N2 receives "Segment 1" from Rank N1 (n=1). Rank N2 performs a reduction operation on the received "Segment 1" and the "Segment 1" in its local memory to obtain the "reduced result of Segment 1". Then, Rank N2 (n=2) sends this calculated "cumulative sum of Segment 1" to Rank N3. Rank N3 (n=3) receives "Segment 2" from Rank N2, adds it to its local "Segment 2", and sends it to Rank N4 (n=4), and so on. This process continues for N-1 rounds. In each round, the data segment moves along the positive direction of the horizontal communication ring, accumulating the local data corresponding to each node it passes through. Finally, let's take Segment 1 as an example for further explanation. Figure 7Segment 1 will pass through all other nodes before reaching Rank N4, and complete the final reception and reduction calculation at Rank N4, thus forming a fragment of the globally complete aggregated gradient at Rank N4.

[0057] This application embodiment improves communication efficiency by selecting data segments from different locations in the tensors stored by different graphics processors (GPUs) on the same computing node, using a staggered startup strategy. This ensures that every physical link on the horizontal communication ring is activated simultaneously and is in a state of transmitting valid data. Furthermore, by selecting data segments from different memory locations according to the GPU's sequence number and performing reduction calculations on these selected data segments in parallel, the responsibility for storing the final aggregation result and the computational overhead of the reduction calculation are distributed to each GPU. This ensures that no single node bears the burden of storing or computing the entire data, avoiding situations where a single node's memory overflows or computational overload causes the entire cluster's training to stall, thus improving the stability and resource utilization of the computing cluster. Moreover, this staggered startup method enables conflict-free pipeline scheduling at the hardware level; that is, conflict-free data transmission is achieved during the subsequent transmission and broadcasting of data to each GPU in the vertical communication ring.

[0058] In step 102, the reduced data segment is divided into multiple first reduced channel blocks. For each first reduced channel block, reduction calculations are performed sequentially between the graphics processors along the positive direction of the vertical communication ring to obtain a second reduced channel block.

[0059] Here, a "channel block" refers to a transmission unit that involves secondary segmentation of logical data segments to adapt to the physical bandwidth characteristics of the communication link. The size of a channel block is usually less than or equal to the size of a data segment, and its granularity is typically associated with the maximum transmission unit (MTU) or the buffer size of direct memory access (DMA) for network transmission between computing nodes.

[0060] In some embodiments, the reduced data segment can be obtained as follows: upon completion of step 101, each graphics processor has generated local reduced data (reduced data segment) aggregated within its local video memory in a specific address space. The sum of the local reduced data represents the sum of contributions of all graphics processors within the current compute node to the gradient of that part of the model parameters.

[0061] As an example, reducing computations sequentially between graphics processors along the positive direction of the vertical communication loop can be achieved by using pipeline technology to divide a reduced data segment into... The first consecutive channel block. When the first... When the first channel block is being sent to the downstream node through the inter-node network interface card (NIC), the second... Each channel block can be prepared or prefetched. For each channel block, each node writes data directly to the storage area of ​​the downstream node through Remote Direct Memory Access (RDMA) technology, and after receiving upstream data, calls the Arithmetic and Logic Unit (ALU) to perform reduction operations, finally obtaining the globally aggregated partial gradient channel block at the end node of the vertical loop.

[0062] In some embodiments, see Figure 8 In the case that each reduced data segment occupies M storage areas, where M is a set positive integer, the reduction calculation in step 102, which is performed sequentially between graphics processors along the positive direction of the vertical communication ring, can be achieved through the following steps 1021 to 1022.

[0063] In step 1021, if the first reduction channel block of the m-th graphics processor is not received in the storage area of ​​the m-th graphics processor, the first reduction channel block of the m-th storage area occupied by the reduction data segment is transmitted to the storage area of ​​the (m+1)-th graphics processor along the positive direction of the vertical communication ring.

[0064] Here, "the m-th storage area" refers to the storage area segment with index m after the cache containing the reduced data segment is divided in the logical address space according to the number of computing nodes M in the computing cluster.

[0065] Here, "the case where the first reduction channel block of the transmission is not received" typically refers to the start of the reduction process (vertical reduction) on the vertical communication ring. At this stage, since data flow on the vertical communication ring has not yet started, the receive buffer of each graphics processing unit (GPU) is empty, thus triggering the initial transmission logic.

[0066] In some embodiments, the transmission of the first reduced channel block of the m-th memory area can be achieved as follows: First, the control device calculates the starting address of the memory area to be transmitted based on the logical number m of the m-th graphics processor in the vertical communication ring. Specifically, if the total length of the reduced data segment is... The starting address is Then the offset of the m-th storage area is The control device reads the reduced channel block stored at the address corresponding to the offset. This is then transmitted to the next node in the vertical communication ring (the (m+1)th graphics processor) via the network interface card between the computing nodes.

[0067] As an example, see Figure 9 The following example illustrates a vertical communication ring containing M=4 graphics processors (hereinafter, for simplification, the graphics processors are referred to as Rank M1 to Rank M4, i.e., m takes values ​​from 1 to 4 respectively). Assuming the reduced data segment size in each graphics processor is 1GB, then each storage area after partitioning contains 256MB of data. In step 1021, Rank M1 reads the first first reduced channel block of its local reduced data segment and sends it to Rank M2. Rank M2 reads the second first reduced channel block of its local reduced data segment and sends it to Rank M3, while simultaneously receiving the first reduced channel block transmitted by Rank M1 in the first storage area. Rank M3 reads the third first reduced channel block of its local reduced data segment and sends it to Rank M4, while simultaneously receiving the first reduced channel block transmitted by Rank M2 in the second storage area. Rank M4 reads the fourth first reduced channel block of its local reduced data segment and sends it to Rank M1, while simultaneously receiving the first reduced channel block transmitted by Rank M3 in the third storage area. In this way, all communication links on the vertical communication ring are It is always fully loaded and there is no resource contention.

[0068] In other embodiments, if the total size of the reduced data segment is not divisible by M, zero-value data can be added to the end of the last storage area using padding techniques, or before configuration. The size of each storage area is The remaining storage area is [size missing]. This is to ensure the integrity of data partitioning. The embodiments of this application define specific block alignment strategies.

[0069] In step 1022, when the first reduced channel block is received in the storage area of ​​the m-th graphics processor, the received first reduced channel block is reduced with the local channel block, and the reduced first reduced channel block is transmitted to the storage area of ​​the (m+1)-th graphics processor along the positive direction of the vertical communication ring.

[0070] In some embodiments, reducing the received first reduced channel block to the local channel block can be achieved in the following way: when the network interface card of the m-th graphics processor detects a signal from the first... When the receive buffer of each node completes writing, the arithmetic logic unit is triggered to perform a reduction operation. This arithmetic logic unit loads the data vector of the first reduced channel block from the receive buffer in parallel. and the data vector of the corresponding channel block in the local video memory Perform element-wise addition: Calculation results The result will be written directly to the send buffer or overwritten with the corresponding data in the local storage area, and then a send command will be triggered immediately to forward the result to the next... One storage area.

[0071] As an example, continuing from the above embodiments, see... Figure 10 Rank M1 receives the fourth first reduction channel block transmitted from Rank M4, performs reduction calculation with the fourth first reduction channel block of its local reduced data segment, and simultaneously reads its second first reduction channel block and sends it to Rank M2. Rank M2 receives the second first reduction channel block transmitted from Rank M1, performs reduction calculation with the second first reduction channel block of its local reduced data segment, and simultaneously reads its third first reduction channel block and sends it to Rank M3. Rank M3 receives the third first reduction channel block transmitted from Rank M2, performs reduction calculation with the third first reduction channel block of its local reduced data segment, and simultaneously reads its fourth first reduction channel block and sends it to Rank M4. Rank M4 receives the third first reduction channel block transmitted from Rank M3, performs reduction calculation with the third first reduction channel block of its local reduced data segment, and simultaneously reads its first first reduction channel block and sends it to Rank M1. After repeating 1022 twice, at the starting time, the first reduced channel block in Rank M1 has traversed every memory area in the vertical communication loop, and full gradient aggregation is completed in Rank M4. After all channel blocks have completed full gradient aggregation, each graphics processor obtains a fragment of the global gradient (the second reduced channel block), that is, the memory area with the first index 2 in Rank M1, the memory area with the first index 3 in Rank M2, the memory area with the first index 4 in Rank M3, and the memory area with the first index 1 in Rank M4 become the second reduced channel block.

[0072] It should be noted that the memory areas used by the graphics processors on different vertical communication rings to store the reduced data segments are different. Taking a computing cluster including 4×4 graphics processors as an example, this can be illustrated by... Figure 11 As shown, data transfer for each graphics processor is performed in parallel. Figure 11 In this configuration, each row of four graphics processors represents the same graphics processor within the same computing node, while each column of four graphics processors represents the same second sequence number across different computing nodes. After obtaining the first reduced channel block, it can be transmitted along the positive direction of the vertical communication ring, so that the next computing node obtains the vertically reduced first reduced channel block.

[0073] This embodiment of the application divides the reduced data segment at the vertical communication ring level and performs reduction calculations on the vertical communication ring after dividing the reduced data segment. This ensures that every physical link on the vertical communication ring is in an effective transmission state at every time step, improving the bandwidth utilization between computing nodes. Furthermore, this embodiment divides a reduced data segment into first reduced channel blocks stored in M ​​storage areas. Each graphics processor is only responsible for the final aggregation of one storage area at each step, achieving even load distribution and distributing the computing power of the reduction calculation evenly among all graphics processors, avoiding computational delays caused by overload of a single graphics processor. It should be noted that since the reduction calculation is an element-wise calculation of a vector matrix, the transmission of the first reduced channel block along the vertical communication ring can be initiated when the reduced data segment is partially completed, i.e., when a certain first reduced channel block is obtained (but the complete reduced data segment is not yet obtained). The first reduced channel block is then used for reduction calculation with the channel block local in the transmitted storage area. In this case, the process of reducing transmission along the horizontal communication ring and the process of reducing along the vertical communication ring may overlap in time, improving the calculation speed.

[0074] In step 103, if a second reduced channel block is obtained by any graphics processor reduction calculation, the second reduced channel block is distributed sequentially along the reverse direction of the vertical communication ring.

[0075] Here, "reverse direction" refers to the data transmission flow in a logical ring topology that is opposite to the aforementioned "positive direction." For example, if the positive direction is defined as a node... node Then the reverse direction is defined as a node. node .

[0076] It should be noted that the reduction distribution of gradients in deep learning models aims to calculate the cumulative sum of corresponding data segments on different nodes. Furthermore, since the data elements in the reduction calculation are independent of each other, logically splitting the reduced data segment into several consecutive channel blocks essentially decomposes a large-scale vector accumulation task into several non-overlapping small-scale subtasks. The reduction calculation for each channel block is performed within its specific memory address range, and the calculation processes of different channel blocks do not interfere with each other spatially. Regardless of whether these channel blocks are transmitted continuously at once or in batches via pipeline, as long as each channel block completes the prescribed number of reduction steps according to the predetermined ring topology, the aggregated data (the second reduced channel block) formed by finally splicing all reduced channel blocks is strictly consistent with the result of directly reducing and distributing the complete data segment. In some embodiments, for a specific channel block, the reduction calculation operation on the vertical communication ring has been completed, indicating that the channel block contains the global gradient contributions of all computing nodes in the cluster to this part of the parameters, becoming a fragment of the global gradient, i.e., the second reduced channel block. Therefore, by distributing the second reduction channel block of multiple computing nodes to each computing node, each computing node can have a fragment of the global gradient of the tensor of the deep learning model.

[0077] It should be noted that since the second reduced channel block already has a global gradient contribution, it only needs to be distributed among the memory areas of each graphics processor on each computing node, and no further reduction calculations are performed with the channel blocks local to each memory. After receiving the second reduced channel block, each memory area can store one copy locally and then distribute the second reduced channel block sequentially along the reverse direction of the horizontal communication ring and sequentially along the reverse direction of the vertical communication ring.

[0078] In some embodiments, distribution along the reverse direction of the vertical communication ring can be achieved as follows: Each graphics processor sends its computed second reduced channel block, containing a global gradient fragment, as source data to its predecessor node (the previous memory location in the vertical communication ring) via the inter-node network. After the previous memory location in the vertical communication ring receives and stores the second reduced channel block, it continues to forward it to even more predecessor memory locations until the channel block is broadcast to all memory locations in the vertical ring. For example, a store-and-forward-based ring broadcast algorithm can be used to ensure that each node receives the globally updated value of the channel block.

[0079] As an example, see Figure 12 Continuing with the above description of an embodiment where a vertical communication ring includes M=4 graphics processors, the following explanation will be provided. Figure 12In this context, assume that the first sequence number of memory area 2 in Rank M1, the first sequence number of memory area 3 in Rank M2, the first sequence number of memory area 4 in Rank M3, and the first sequence number of memory area 1 in Rank M4 are used to obtain the second reduced channel block. The first step of distributing the second reduced channel block between graphics processors along the reverse direction of the vertical communication loop is as follows: Rank M4 distributes the second reduced channel block to memory area 1 in Rank M3, Rank M3 distributes the second reduced channel block to memory area 4 in Rank M2, Rank M2 distributes the second reduced channel block to memory area 3 in Rank M1, and Rank M1 distributes the second reduced channel block to memory area 2 in Rank M4. After completing the first distribution step, the memory areas with first sequence numbers 2 and 3 in Rank M1, the memory areas with first sequence numbers 3 and 4 in Rank M2, the memory areas with first sequence numbers 4 and 1 in Rank M3, and the memory areas with first sequence numbers 1 and 2 in Rank M4 already have the second reduction channel blocks. The second distribution step is as follows: Figure 13 As shown, this includes: Rank M4 sending a second reduction channel block to the storage area with the first sequence number 2 in Rank M3; Rank M3 sending a second reduction channel block to the storage area with the first sequence number 1 in Rank M2; Rank M2 sending a second reduction channel block to the storage area with the first sequence number 4 in Rank M1; and Rank M1 sending a second reduction channel block to the storage area with the first sequence number 3 in Rank M4. See also... Figure 14 After four rounds of distributing the second reduction channel block along the reverse direction of the vertical communication loop among the graphics processors, each graphics processor obtains a global aggregation of the reduced data segment. That is, each graphics processor obtains a global gradient aggregation value of a specific fragment of the target tensor, which is the same index range as the reduced data segment in the model parameters.

[0080] As an example, different graphics processing units (GPUs) may have different hardware properties (silicon lotteries), heat dissipation and power consumption, and system interference (background processes, momentary congestion, etc.). Therefore, the computation speed on different GPUs may vary (a computational delay in one GPU step will affect the computation time of subsequent GPU steps). This can lead to a situation where some GPUs receive the second reduced channel block first, while others receive it later. In this case, the process of transmitting the first reduced channel block along the forward direction of the vertical communication ring and the process of distributing the second reduced channel block sequentially along the reverse direction of the vertical communication ring will be executed simultaneously throughout the entire computing cluster. See also... Figure 15 , Figure 15Taking a vertical communication ring containing M=4 graphics processors as an example, Rank M1, Rank M2, and Rank M3 have all completed the reduction calculation to obtain the second reduced channel block, while Rank M4 has not completed the reduction calculation to obtain the second reduced channel block (i.e., Rank M4 includes the first reduced channel block and the first reduced channel block after reduction calculation, but does not include the second reduced channel block). In this case, Rank M1, Rank M2, and Rank M3 are executed in parallel, distributing the second reduced channel block sequentially in the reverse direction of the vertical communication ring. Rank M4 receives the first reduced channel block transmitted by Rank M3 and calls its local first reduced channel block to perform the reduction calculation.

[0081] In step 104, if any graphics processor receives a second reduced channel block, the second reduced channel block is distributed sequentially in the reverse direction of the transverse communication loop.

[0082] In some embodiments, distribution along the reverse direction of the horizontal communication ring can be achieved by utilizing the high-speed point-to-point copy capability between GPUs within the same compute node, where the graphics processor holding the latest global gradient channel block copies it to the memory area of ​​the previous GPU within the same node. This process is typically pipelined in parallel with the vertical receiving process in step 103; that is, once a channel block is received from the vertical ring, broadcasting of that channel block is immediately initiated within the horizontal ring (if there is a previous memory area in the vertical communication ring, it can be distributed in parallel to the previous memory area in the vertical communication ring), thereby achieving communication masking and ultimately ensuring that all graphics processors within the cluster are updated to consistent global model parameters.

[0083] In some embodiments, since the second reduced channel block is a global gradient aggregation value of a specific fragment of a tensor, the graphics processor does not need to perform reduction calculations on the received second reduced channel block. After receiving the second reduced channel block, the graphics processor distributes the second reduced channel block not only in the reverse direction along the vertical communication loop but also in the reverse direction along the horizontal communication loop. It should be noted that the graphics processor that has completed the reduction calculation to obtain the second reduced channel block also needs to distribute the second reduced channel block among multiple graphics processors in the reverse direction along the horizontal communication loop.

[0084] As an example, see Figure 16 , Figure 16The distribution process initiated by Rank M2 and Rank M3 is illustrated. In step 1, after Rank M1 (Rank N4) completes the reduction calculation to obtain the second reduced channel block, it transmits the second reduced channel block to Rank M4 (Rank N4) in the reverse direction of the vertical communication ring, and then transmits it to Rank M1 (Rank N3) in the reverse direction of the horizontal communication ring. At this time, Rank M4 (Rank N4) transmits the second reduced channel block to Rank M3 (Rank N3) in the reverse direction of the vertical communication ring. In step 2, Rank M4 (Rank N4) transmits the second reduced channel block to Rank M3 (Rank N4) in the reverse direction of the vertical communication ring, and then transmits it to Rank M4 (Rank N3) in the reverse direction of the horizontal communication ring.

[0085] This application embodiment constructs a horizontal communication ring between the storage areas of multiple graphics processors within a unified computing node, and a vertical communication ring between the storage areas of multiple computing nodes. Data segments are transmitted within the node via the horizontal communication ring, and after horizontal reduction and distribution are completed, the reduced data segments (reduced data segments) are divided. Since reduction computation is an element-wise calculation of a vector matrix, and the data elements in the reduction computation are independent of each other, the current computation in the computing cluster does not need to wait for the data packets from the previous step to complete their computation. This results in partial overlap of execution time in different communication stages, improving the computational efficiency of the computing cluster. Furthermore, the cluster communication method provided in this application embodiment can be naturally extended to two-dimensional structures of different sizes, such as 8×8 and 16×16 computing clusters, simply by selecting different partitioning methods according to the cluster size, demonstrating good scalability.

[0086] In some embodiments, see Figure 17 In the case that the first reduction channel block of the transmission is not received in the storage area of ​​the m-th graphics processor in step 1021, step 103 can be replaced by steps 105 to 106.

[0087] In step 105, the first reduction channel block of the m-th storage area occupied by the reduction data segment is divided into multiple first slices.

[0088] Here, "segmentation" does not refer to physical memory data movement or reorganization, but rather to the logical control level, further subdividing the "first reduction channel block," which was originally a whole transmission unit, into smaller transmission units, namely "first slices."

[0089] In some embodiments, the first reduced channel block can be segmented by setting a predefined slice size, which is typically determined based on the underlying communication hardware, such as the Peripheral Component Interconnect Bus (PCIe) or High Speed ​​Interconnect Bus Interconnect Protocol's Maximum Transmission Unit (MTU) or Optimal Transfer Block Size (OPS). The control device sets the total length of the "first reduced channel block". Divide by the slice size to get the number of slices. For the first slices ( Its starting address in video memory is calculated as follows: This fine-grained segmentation allows the graphics processor's direct memory access (DMA) engine to generate multiple independent transfer requests in units of "slices," resulting in a denser pipeline in the transfer queue.

[0090] As an example, let's consider a first reduction channel block size of 128MB. Directly transmitting this 128MB first reduction channel block between graphics processors could lead to prolonged communication link occupation and long periods of idle time for subsequent graphics processors in the vertical communication ring. Therefore, the first reduction channel block can be divided into 16 first slices of 8MB each. For example, first slice 1 is 0-8MB, first slice 2 is 8-16MB, and so on, with first slice 15 being 120-128MB. This segmentation strategy allows the receiving end to immediately begin computation on "first slice 1" once it is transmitted, while the sending end is transmitting "slice 1" in parallel. This creates a fine-grained pipeline that achieves sub-millisecond overlap between computation and communication.

[0091] In step 106, the first slice is transmitted along the positive direction of the vertical communication loop to the storage area of ​​the (m+1)th graphics processor.

[0092] The first slice is used to enable the storage area of ​​the (m+1)th graphics processor to perform a reduction calculation on the first slice and the second slice of the local channel block after receiving the first slice, and then transmit the second slice obtained by the reduction calculation to the storage area of ​​the (m+2)th graphics processor along the positive direction of the vertical communication loop.

[0093] Here, "second slice" refers to the data segment in the (m+1)th graphics processor's local memory that logically corresponds to the received "first slice". Although the physical addresses of the first and second slices may differ, they represent the same index range in the model parameters.

[0094] As an example, see Figure 18Let's take a computing cluster with 4×2 graphics processors as an example. The graphics processors in each cluster are represented as A, B, C, and D. Each graphics processor has 16 memory areas, and every four memory areas are labeled as a segment; that is, D0.0 to D0.3 are the four memory areas of the first segment. Figure 18 In this process, after the first compute node D0.0 receives the first slice, it can initiate the step of reducing and distributing the first slice to the second compute node D0.0 along the vertical communication loop. At this time, the first compute node D0.0 is also synchronously receiving the second slice, which has not yet been reduced and distributed, sent from C0.0. While node D is performing the above operations, nodes A, B, and C are also performing similar pipelined transmission and computation operations at their corresponding slice locations.

[0095] This embodiment refines the coarse-grained first reduction channel block into multiple first slices and performs pipelined reduction in the vertical loop. This achieves partial overlap between the "head" of the vertical communication stage and the "tail" of the horizontal communication stage on the time axis, masking the startup delay of the vertical communication stage and improving the overall reduction efficiency of the computing cluster. Furthermore, sliced ​​transmission eliminates idle bubbles in the communication link, and only requires a smaller on-chip cache than the transmission of the first reduction channel block to complete the atomic operation of "receive-compute-distribute," reducing the instantaneous peak occupancy of computing resources and making data transmission smoother, thereby enhancing the stability of distributed training in the computing cluster.

[0096] In some embodiments, after transmitting the first slice along the positive direction of the longitudinal communication loop to the (m+1)th graphics processor in step 106 above, the following steps may also be performed.

[0097] If the second slice obtained from the reduction calculation is a third slice used to form the second reduction channel block, the third slice is distributed in the reverse direction of the longitudinal communication ring.

[0098] Here, the "third slice" refers to a component that can be used to form the second reduction channel block, namely, a slice that has completed all vertical reduction calculations (transmissions have been performed a set number of times along the positive direction of the vertical communication loop) and has a global gradient aggregation value. The so-called second reduction channel block refers to a portion of the complete data block containing the aggregation information of the entire vertical loop that each graphics processor should ultimately hold after undergoing the complete vertical reduction process. Due to the slicing mechanism, the second reduction channel block is not generated all at once, but rather the third slice is obtained by successively reducing the second slices one by one at the corresponding address segment (Offset) in the receive buffer, and then filling and splicing the third slices together.

[0099] As an example, see Figure 19After reducing a specific slice (i.e., obtaining the third slice), a reverse distribution or full collection operation can be performed immediately on the third slice. Figure 19 Continuing from the previous explanation, this section uses a 4×2 graphics processor cluster as an example. Since only two compute nodes are involved in the reduction distribution, the second compute node's D0.0 receives the first slice from the first compute node's D0.0 and performs a reduction calculation with the first slice locally on the second compute node's D0.0. The resulting first slice is the third slice. After obtaining the first third slice, the second compute node's D0.0 distributes this first third slice to the second compute node's C0.0 and simultaneously to the first compute node's D0.0. During the distribution of the third slice by the second node's D0.0, it may also receive the second slice from the first compute node's D0.0 and perform a reduction calculation with the local second slice. This achieves a pipelined reduction calculation where the receiving and sending phases partially overlap on the timeline.

[0100] This application introduces a slicer pipeline mechanism, which allows the forward reduction calculation and reverse distribution process to be executed concurrently within the same time window. This enables both the sending and receiving ends of the graphics processor to maintain near-full throughput, reduces the time bubble of waiting for data preparation or link idle, improves the computing speed of the computing cluster, and thus improves the iteration speed of distributed training.

[0101] In some embodiments, if the second slice obtained from the reduction calculation is a third slice used to form the second reduction channel block, after distributing the third slice in the reverse direction of the longitudinal communication loop, the following steps may also be performed.

[0102] When a third slice is received in the memory area of ​​the graphics processor, the received third slice is distributed in the reverse direction along the horizontal communication ring, and the received third slice is also distributed in the reverse direction along the vertical communication ring.

[0103] In some embodiments, when any graphics processor receives a complete third slice, it indicates that the global aggregation of that slice has been completed. At this point, the graphics processor acts as a "distribution source" or "relay station" for that data slice. By distributing along the reverse direction of the horizontal communication loop, third slices at different locations can be transmitted to different graphics processors on the same computing node. The third slice received by each graphics processor is a global aggregation result at a different location. By concatenating the third slice results from graphics processors on the same node, the complete global aggregation result of the tensor can be obtained.

[0104] This application embodiment fully utilizes the multi-port concurrent communication capability of the graphics processor and reduces serial dependencies by immediately triggering the reverse distribution of the third slice along the horizontal communication ring and the reverse distribution along the vertical communication ring after receiving the slice of the global aggregation result (the third slice), thereby improving the computation speed of distributed training.

[0105] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0106] In related technologies, during data-parallel training, the training dataset is split and distributed across multiple graphics processing units (GPUs) for parallel processing. Each GPU independently computes gradients based on its local data. To ensure global convergence of the model, the local gradients scattered across each GPU must be aggregated, and the complete global gradient must be synchronized back to all nodes to ensure that the next iteration is based on a consistent copy of the model. Related technologies often employ the AllReduce algorithm to achieve this process. Through two phases—reduction and full collection—each node sequentially sends partial gradients and receives the accumulated gradients, ultimately completing parameter synchronization.

[0107] However, the traditional AllReduce algorithm suffers from computational efficiency bottlenecks. First, its process is strictly serially dependent, meaning the next operation must wait for the complete data block to be transmitted or the previous computation to complete. This "stop-and-wait" mechanism leads to significant idle time for computation threads during communication, causing substantial pipeline bubbles. Second, in large-scale clusters, the high-speed interconnect bandwidth of GPUs within a node is far higher than the network bandwidth between nodes, such as Ethernet or wireless networks. This heterogeneous bandwidth characteristic makes low-speed communication across nodes a bottleneck, causing high-speed links to wait for low-speed transmissions to complete, leading to communication congestion and latency accumulation, severely limiting the overall cluster training efficiency.

[0108] To address the problem of insufficient training efficiency caused by low communication speed in distributed training, the embodiments of this application can partially overlap the time axis between multiple steps in the reduction phase, between multiple steps in the full collection phase, and between the reduction phase and the full collection phase, thereby reducing the idle time of computing threads during communication and the generation of pipeline bubbles.

[0109] The following describes the specific implementation process of the communication method for the computing cluster provided in the embodiments of this application.

[0110] Step 1: Perform horizontal reduction.

[0111] During the training phase of a deep learning model, the graphics processing unit (GPU) first loads a copy of the model and reads a subset of independent samples from the total training dataset for forward inference, calculating the prediction error. Then, using the chain rule, it infers backward from the output layer to the input layer, calculating the partial derivative of each model parameter with respect to the error, i.e., the local gradient. This local gradient is calculated based on a subset of local data processed by the current GPU, representing only the optimization direction of that subset of data for the model parameters; therefore, it has high variance and locality. Global aggregation accumulates or averages the local gradients across all GPUs, generating a global gradient that reflects the characteristics of the entire training batch. This process eliminates the randomness introduced by a single data subset, ensuring the unbiasedness and accuracy of gradient estimation. This allows all GPUs to synchronously update model parameters based on a consistent and accurate global gradient, guaranteeing the convergence and stability of distributed training.

[0112] A horizontal communication ring is pre-constructed between multiple graphics processors (GPUs) within the same compute node, and a vertical communication ring is pre-constructed between GPUs on different compute nodes. The horizontal communication ring connects the corresponding (same first sequence number) local high-bandwidth storage areas of each GPU within the same compute node, forming a closed path for high-speed data flow within the node. The vertical communication ring connects the storage areas of corresponding GPUs (same second sequence number) on different compute nodes, utilizing network interface cards (NICs) and RDMA technology to connect remote video memory address spaces. The horizontal and vertical communication rings construct a two-dimensional, two-layer ring communication topology covering the entire cluster's video memory resources, ensuring data communication both between storage areas within a node and between storage areas between nodes.

[0113] See Figure 20 Multiple graphics processing units (GPUs) within the same computing node each select data segments physically located from their respective stored tensors. Simultaneously, these GPUs transmit their selected data segments along the positive direction of the horizontal communication ring and perform reduction calculations between the transmitted segments and the local data segments in the GPU's memory. Figure 20In this computational cluster, there are four compute nodes, each containing four graphics processing units (GPUs). The tensors stored by each GPU can be divided into four data segments. The first GPU of the first compute node selects the first data segment, the second GPU of the first compute node selects the second data segment, and so on. In the first step of transmission, the first GPU of the first compute node transmits the first data segment to the second GPU of the first compute node. After receiving the first data segment, the second GPU of the first compute node performs a reduction calculation by combining the received data segment with its local data segment, and then sends the second data segment to the third GPU of the first compute node. After completing the third step of reduction, the fourth GPU of the first compute node obtains the local gradient aggregation (reduced data segment) of the first data segment. At this point, the first graphics processor of the first computing node obtains the local gradient aggregation (reduced data segment) of the second data segment, the second graphics processor of the first computing node obtains the local gradient aggregation (reduced data segment) of the third data segment, and the third graphics processor of the first computing node obtains the local gradient aggregation (reduced data segment) of the fourth data segment. After completing three rounds of reduction calculations, the reduced data segments obtained by each graphics processor of the first computing node can be concatenated to obtain the gradient aggregation of the data segment of the first computing node. Furthermore, since the first sequence number (the sequence number of the location in the graphics processor) and the second sequence number (the sequence number of the computing node) of the reduced data segments in the first computing node are different, i.e., their physical addresses are different, parallel reduction calculations and transmission are possible in the subsequent vertical reduction stage. Since the reduced data segments of multiple computing nodes are all located in the same position, vertical reduction based on the completed horizontally reduced data segments can obtain the global gradient aggregation of the corresponding data segments.

[0114] Step two: Perform vertical reduction.

[0115] Since the communication bandwidth between nodes is typically less than the communication bandwidth within a node, directly performing vertical reduction of the original reduced data segment on the graphics processors between computing nodes may lead to communication congestion. The probability of congestion is even higher when multiple graphics processors are performing vertical reduction in parallel. To address this, the reduced data segment can be divided into multiple reduced data segments according to the communication bandwidth between nodes. See also... Figure 20Each time a vertical reduction step is performed, a first reduction channel block is selected from the reduced data segment, and the selected first reduction channel block is reduced along the vertical communication loop. For example, the first reduction channel block of the first memory area is selected from the reduced data segment of the fourth graphics processor of the first compute node, and vertically reduced to the graphics processor of the second compute node, while simultaneously receiving the first reduction channel block of the fourth memory area from the reduced data segment of the fourth graphics processor of the fourth compute node. After three vertical reductions, the first first reduction channel block in the fourth graphics processor of the first compute node has been reduced with the data segment in the first memory area of ​​the graphics processor of each compute node (i.e., data aggregation is achieved), resulting in the global gradient aggregation (second reduction channel block) corresponding to the physical address of the first first reduction channel block. At this point, the graphics processor of the first computing node obtains the second reduction channel block of the second storage area where the reduced data segment is located, the graphics processor of the second computing node obtains the second reduction channel block of the third storage area where the reduced data segment is located, the graphics processor of the third computing node obtains the second reduction channel block of the fourth storage area where the reduced data segment is located, and the graphics processor of the fourth computing node obtains the second reduction channel block of the first storage area where the reduced data segment is located.

[0116] In some embodiments, see Figure 21 Since communication bandwidth is usually inconsistent with the size of the data segment or channel block, the data segment can be divided into multiple slices. Because the reduction calculation involves adding the elements within the data segment one by one, after completing the reduction calculation for one slice, the calculated first slice can be vertically reduced (transmitted to the next graphics processor for reduction calculation). That is, after the first compute node's D0.0 obtains the first slice, it can initiate the step of transmitting the first slice along the vertical communication loop to the second compute node's D0.0. At this time, the first compute node's D0.0 is also synchronously receiving the incomplete second slice sent from C0.0.

[0117] Step 3: Perform vertical and horizontal inverse distribution.

[0118] See Figure 22 After obtaining the second reduction channel block, each graphics processor distributes the second reduction channel block in the reverse direction of the vertical communication ring. For example, the fourth graphics processor of the first compute node transmits the second reduction channel block of the second storage area to the graphics processor of the fourth compute node in the reverse direction of the vertical communication ring, and at the same time distributes the second reduction channel block to the third graphics processor of the first compute node (in the reverse direction of the horizontal communication ring).

[0119] In addition, see Figure 23Alternatively, channel blocks can be inversely distributed in the graphics processor in the form of slices. That is, after the second compute node's D0.0 receives the first slice from the first compute node's D0.0 and performs calculations with the first slice locally on the second compute node's D0.0, the resulting first slice is the third slice. After obtaining the first third slice, the second compute node's D0.0 distributes this first third slice to the second compute node's C0.0, and simultaneously to the first compute node's D0.0. During the distribution of the third slice by the second node's D0.0, it may also receive the second slice from the first compute node's D0.0 and perform reduction calculations with its local second slice.

[0120] After completing the vertical and horizontal inverse distribution, the data on all graphics processors are uniformly aggregated with global gradients.

[0121] The following description continues to illustrate the exemplary structure of the communication device 145 of the computing cluster provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the communication device 145 of the computing cluster in memory 141 may include: The horizontal reduction module 1451 is used to select multiple data segments from the data stored in multiple graphics processors, and perform reduction calculations on the selected data segments sequentially along the positive direction of the horizontal communication ring to obtain multiple reduced data segments. The first sequence number of the storage area occupied by the selected data segments is different for different graphics processors. The vertical reduction module 1452 is used to divide the reduced data segment into multiple first reduction channel blocks. For each first reduction channel block, reduction calculations are performed sequentially between the graphics processors along the positive direction of the vertical communication loop to obtain a second reduction channel block. The vertical distribution module 1453 is used to distribute the second reduced channel block sequentially along the reverse direction of the vertical communication loop when the second reduced channel block is obtained by any graphics processor reduction calculation. The horizontal distribution module 1454 is used to distribute the second reduction channel block sequentially along the reverse direction of the horizontal communication ring when any graphics processor receives the second reduction channel block.

[0122] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the communication method of the computing cluster described in this application embodiment.

[0123] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the communication method of the computing cluster provided in this application embodiment, for example, such as... Figure 3 The communication method of the computing cluster is shown.

[0124] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0125] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0126] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0127] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0128] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A communication method for a computing cluster, characterized in that, The computing cluster includes multiple computing nodes, each computing node includes multiple graphics processors (GPUs), and each GPU includes multiple storage areas. Multiple storage areas within the same computing node form a horizontal communication ring, and multiple storage areas across different computing nodes form a vertical communication ring. The storage areas forming the horizontal communication ring share the same first sequence number within the GPUs, and the GPUs containing the storage areas forming the vertical communication ring share the same second sequence number within the computing nodes, and the first sequence numbers are also the same. The method includes: Multiple data segments are selected from the data stored in the multiple graphics processors, and the selected data segments are sequentially reduced along the positive direction of the horizontal communication ring to obtain multiple reduced data segments. The first sequence number of the storage area occupied by the selected data segments is different for different graphics processors. The reduced data segment is divided into multiple first reduced channel blocks. For each first reduced channel block, reduction calculations are performed sequentially between the graphics processors along the positive direction of the vertical communication ring to obtain a second reduced channel block. If the second reduced channel block is obtained by any of the graphics processors through reduction calculation, the second reduced channel block is distributed sequentially along the reverse direction of the vertical communication ring; If any of the graphics processors receives the second reduction channel block, the second reduction channel block is distributed sequentially in the reverse direction of the transverse communication ring.

2. The method according to claim 1, characterized in that, The step of selecting multiple data segments from the data stored in the multiple graphics processors, and sequentially performing reduction calculations on the selected data segments along the positive direction of the horizontal communication loop, includes: If the nth graphics processor does not receive the transmitted data segment in its storage area, the nth data segment is transmitted along the positive direction of the horizontal communication loop to the (n+1)th graphics processor's storage area. When the transmitted data segment is received in the storage area of ​​the nth graphics processor, the received data segment is reduced by the local data segment, and the reduced data segment is transmitted to the storage area of ​​the (n+1)th graphics processor along the positive direction of the horizontal communication ring.

3. The method according to claim 1, characterized in that, Each of the reduced data segments occupies M of the storage areas, where M is a set positive integer; The step of performing reduction calculations sequentially between the graphics processors along the positive direction of the longitudinal communication loop for each of the first reduction channel blocks includes: If the first reduction channel block is not received in the storage area of ​​the m-th graphics processor, the first reduction channel block of the m-th storage area occupied by the reduction data segment is transmitted to the storage area of ​​the (m+1)-th graphics processor along the positive direction of the vertical communication loop. When the first reduced channel block is received in the storage area of ​​the m-th graphics processor, the received first reduced channel block is reduced with the local channel block, and the reduced first reduced channel block is transmitted to the storage area of ​​the (m+1)-th graphics processor along the positive direction of the vertical communication ring.

4. The method according to claim 3, characterized in that, In the case that the first reduced channel block is not received in the storage area of ​​the m-th graphics processor, the method further includes: The first reduction channel block of the m-th storage area occupied by the reduction data segment is divided into multiple first slices; The first slice is transmitted along the positive direction of the vertical communication ring to the storage area of ​​the (m+1)th graphics processor, wherein the first slice is used to enable the storage area of ​​the (m+1)th graphics processor to perform a reduction calculation on the first slice and the second slice of the local channel block after receiving the first slice, and to transmit the second slice obtained by the reduction calculation along the positive direction of the vertical communication ring to the storage area of ​​the (m+2)th graphics processor.

5. The method according to claim 4, characterized in that, After transmitting the first slice along the positive direction of the longitudinal communication loop to the storage area of ​​the (m+1)th graphics processor, the method further includes: If the second slice obtained from the reduction calculation is a third slice used to form the second reduction channel block, the third slice is distributed sequentially in the reverse direction of the longitudinal communication ring.

6. The method according to claim 5, characterized in that, After distributing the third slice sequentially along the reverse direction of the longitudinal communication ring, the method further includes: When the third slice is received in the memory area of ​​the graphics processor, the received third slice is distributed in the reverse direction of the horizontal communication ring, and the received third slice is also distributed in the reverse direction of the vertical communication ring.

7. A communication device for a computing cluster, characterized in that, The computing cluster includes multiple computing nodes, each computing node includes multiple graphics processors (GPUs), and each GPU includes multiple storage areas. Multiple storage areas within the same computing node form a horizontal communication ring, and multiple storage areas across different computing nodes form a vertical communication ring. The storage areas forming the horizontal communication ring share the same first sequence number within the GPUs, and the GPUs containing the storage areas forming the vertical communication ring share the same second sequence number within the computing nodes, and the first sequence numbers are also the same. The device includes: The horizontal reduction module is used to select multiple data segments from the data stored in the multiple graphics processors, and to perform reduction calculations on the selected data segments sequentially along the positive direction of the horizontal communication ring to obtain multiple reduced data segments. The first sequence number of the storage area occupied by the selected data segments is different for different graphics processors. The vertical reduction module is used to divide the reduced data segment into multiple first reduction channel blocks. For each first reduction channel block, reduction calculations are performed sequentially between the graphics processors along the positive direction of the vertical communication loop to obtain a second reduction channel block. The vertical distribution module is used to distribute the second reduced channel block sequentially along the reverse direction of the vertical communication ring when the graphics processor calculates the second reduced channel block. A horizontal distribution module is used to distribute the second reduction channel block sequentially along the reverse direction of the horizontal communication ring when any of the graphics processors receives the second reduction channel block.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 6.

10. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 6.

Images