A logic sparse multi-node hierarchical communication method and apparatus

Abstract

This application belongs to the field of distributed computing technology, specifically relating to a logically sparse multi-node hierarchical communication method and apparatus. The method includes: a computing unit performing node-level aggregation on data replicas destined for the same target remote node to generate an aggregated data packet; a scheduling communication module performing aggregated communication in a single global communication group, sending the aggregated data packet to a designated computing unit on the target remote node, and filling non-target communication paths with invalid data to achieve logically sparse exchange, with all computing units accessing the global communication group as independent endpoints; after receiving the aggregated data packet, the designated computing unit determines the intra-node distribution path based on the routing information within the packet, and distributes the data to the target computing unit through the intra-node communication group. This application significantly reduces the amount of effective cross-node data transmission while maintaining global aggregated communication semantics and implicit synchronization, significantly improving communication efficiency and reducing overall system latency.

Description

Technical Field

[0001] This application belongs to the field of distributed computing technology, specifically relating to a logically sparse multi-node hierarchical communication method and apparatus. Background Technology

[0002] In multi-machine, multi-GPU distributed computing systems, devices within a node are interconnected via high-speed links, while nodes rely on lower-bandwidth networks for interconnection. In this heterogeneous environment, when performing tasks requiring frequent data exchange, such as Sparse Mixture-of-Experts (MoE) inference, traditional flat all-to-all communication suffers from severe synchronization waiting and long-tail latency problems because the overall progress is limited by the slowest cross-node link.

[0003] To alleviate this problem, hierarchical full-to-full communication has been widely adopted as an improved technique. Its typical implementation organizes the communication process into three phases: First, in the intra-node aggregation phase, each Graphics Processing Unit (GPU) aggregates the data to be sent to its designated aggregation unit within its node; second, in the cross-node exchange phase, these aggregation units from various nodes form a communication group, and a full global data exchange is performed within this group; finally, in the intra-node distribution phase, the aggregation units of each node distribute the received data to the target GPU within their node.

[0004] However, in hierarchical all-to-all communication, the efficiency of effective data transmission between nodes has not been fundamentally improved: Because cross-node communication is confined to local communication groups composed of aggregation units, each aggregation unit must exchange full or near-full data with all other aggregation units to complete data relay. This results in the "total effective data volume" transmitted on the cross-node link always approaching the sum of the global data volume, regardless of the actual data demand of the remote node, failing to achieve a structural reduction. The continuous presence of large amounts of data consuming low-speed link bandwidth makes synchronization waiting and long-tail latency problems still prominent.

[0005] The "convergence-relay" communication architecture itself introduces additional overhead: This architecture forces all cross-node data flow through the aggregation unit, increasing the number of hops in the data path from "send GPU -> local aggregation unit -> remote aggregation unit -> target GPU". The resulting overhead of queuing, processing, and forwarding data in the aggregation unit becomes a new burden on top of communication latency, making the aggregation unit itself a performance bottleneck.

[0006] Therefore, existing hierarchical full-to-full communication schemes are limited by their architectural characteristics of "full exchange within local communication groups" and "centralized aggregation relay". When applied to high-performance computing scenarios, it is difficult to coordinate the optimization of "communication efficiency" and "processing overhead". The bottleneck of effective data transmission between nodes and the increase in processing overhead inherent in the hierarchical architecture constitute technical problems that urgently need to be solved. Summary of the Invention

[0007] This application proposes a logically sparse multi-node hierarchical communication scheme, which aims to solve two major technical problems faced by existing hierarchical all-to-all communication methods when applied to distributed computing scenarios such as sparse expert hybrid model (MoE) inference: the bottleneck of effective data transmission between nodes and the inherent processing overhead of the hierarchical architecture.

[0008] A first aspect of this application provides a logically sparse multi-node hierarchical communication method, applied to a distributed computing system composed of multiple computing nodes. Each computing node contains multiple computing units, and a control module and a communication module are deployed on each computing unit. The method is executed by the control module and includes: Node-level aggregation is performed on data copies belonging to the computing unit itself that need to be sent to the same target remote node to generate aggregated data packets to be sent to different target remote nodes; The communication module is scheduled to perform aggregated communication operations in a single global communication group. The aggregated data packet is sent to a designated computing unit in the target remote node that has a predetermined mapping relationship with the computing unit. Invalid data is filled on the non-target communication path of the aggregated communication to realize logically sparse data exchange in the aggregated communication. Each computing unit in the distributed computing system is directly connected to the single global communication group as an independent communication endpoint. When this computing unit receives the aggregated data packet as the designated computing unit, it determines the data distribution path within the node based on the routing information in the aggregated data packet, and distributes the data in the aggregated data packet to one or more target computing units within the node through the intra-node communication group within the node.

[0009] In some embodiments of this application, determining the data distribution path within the node includes determining, at one time, the data distribution path to be distributed to one or more target computing units within the node based on the routing information in the aggregated data packet; When this computing unit receives a data packet from another computing unit within the same node as the target computing unit, it triggers the processing of the data in that data packet.

[0010] In some embodiments of this application, the single global communication group and the intra-node communication groups are pre-built and configured during the system initialization phase.

[0011] In some embodiments of this application, the step of scheduling the communication module to send aggregate data packets to a designated computing unit of the target remote node and the step of determining the data distribution path within the local node are executed at least partially in time.

[0012] In some embodiments of this application, the distributed computing system is a multi-machine, multi-card system; The computing unit is a graphics processor, and multiple graphics processors within the same computing node are interconnected through intra-node high-bandwidth interconnect links to perform data transmission of the intra-node communication group.

[0013] In some embodiments of this application, each computing node participating in the communication contains the same number of graphics processors, and each graphics processor has a unique local identifier within its own node; The designated computing unit in the target remote node that has a predetermined mapping relationship with this computing unit is the graphics processor in the target remote node that has the same local identifier as this graphics processor.

[0014] In some embodiments of this application, the method is applied to distributed reasoning of sparse expert hybrid models, including the distribution of model input tokens and the aggregation of reasoning results.

[0015] In some embodiments of this application, the data to be distributed is the input token of the MoE layer; For input tokens that are the same as those of the target remote nodes, node-level aggregation is performed to generate aggregated data packets. The graphics processors belonging to the target remote nodes and the target experts they carry are determined by the gated network of MoE based on the input tokens. The communication module is scheduled to perform aggregated communication operations in a single global communication group, route the aggregated data packets to the graphics processor that is equivalent to the graphics processor in the target remote node, and fill invalid data on non-target communication paths; The peer graphics processor is the graphics processor within the target remote node that has the same local identifier as this graphics processor; After receiving the aggregated data packet, the peer graphics processor determines the data distribution path within the target remote node, and distributes the input token to the graphics processor carrying the corresponding target expert within the local node through the intra-node communication group of the target remote node, so as to perform inference calculation.

[0016] In some embodiments of this application, the step of performing node-level aggregation on input tokens that are identical to those of the target remote nodes to generate aggregated data packets includes: Multiple copies of the same input token that need to be sent to the same target remote node are merged into a single aggregate data packet.

[0017] A second aspect of this application provides a logically sparse multi-node hierarchical communication device applied to a distributed computing system composed of multiple computing nodes. Each computing node includes multiple computing units, and a control module and a communication module are deployed on each computing unit. The device is implemented by the control module and includes: The node granularity aggregation module is used to perform node granularity aggregation on data copies belonging to the computing unit itself that need to be sent to the same target remote node, and generate aggregated data packets to be sent to different target remote nodes. An inter-node communication module is used to schedule the communication module to perform aggregated communication operations in a single global communication group. The aggregated data packet is sent to a designated computing unit in the target remote node that has a predetermined mapping relationship with the computing unit. Invalid data is filled on the non-target communication path of the aggregated communication to achieve logically sparse data exchange. Each computing unit in the distributed computing system is directly connected to the single global communication group as an independent communication endpoint. The intra-node communication module is used to determine the data distribution path within the node based on the routing information in the aggregated data packet after the computing unit receives the aggregated data packet as the designated computing unit, and distribute the data in the aggregated data packet to one or more target computing units within the node through the intra-node communication group within the node.

[0018] In summary, the logically sparse multi-node hierarchical communication method and apparatus provided in the embodiments of this application firstly construct a "single global communication group" and realize "logically sparse data exchange" under this architecture. Each computing unit prepares data locally through "node-level aggregation," and then, within this global communication group, sends only valid aggregated data packets to peer computing units in the target remote node, while filling all non-target communication paths with invalid data. This significantly reduces the total amount of valid data that must be transmitted on cross-node links from the perspectives of communication architecture and data exchange mode, directly alleviating the bandwidth pressure of low-speed links and overcoming the efficiency bottleneck of effective data transmission between nodes. Secondly, by pipelined overlapping execution of "inter-node data transmission" and "intra-node routing calculation," the intra-node data distribution path is determined in parallel while scheduling and executing long-term cross-node data transmission. This allows the additional computation and preparation overhead introduced by the layered design to be "hidden" within the unavoidable transmission latency, avoiding the significant latency burden that the existing "three-stage" design itself may bring, and effectively alleviating the inherent processing overhead of the layered architecture. Ultimately, the above technical features work together to achieve synergistic optimization of communication efficiency and processing overhead, providing a high-performance, low-latency communication solution for multi-machine, multi-card distributed computing systems, which can fully meet the high throughput and low latency requirements of MoE distributed inference scenarios. Attached Figure Description

[0019] The features and advantages of this application will become clearer with reference to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 This is a schematic diagram of the system architecture of the multi-machine, multi-card system to which this application applies; Figure 2 This is a flowchart illustrating a logically sparse multi-node hierarchical communication method according to some embodiments of this application; Figure 3 This is a schematic diagram of a logically sparse multi-node hierarchical communication device according to some embodiments of this application. Detailed Implementation

[0020] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.

[0021] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.

[0022] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.

[0023] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.

[0024] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.

[0025] With the exponential growth in computing power demands for tasks such as Large Language Modeling (LLM) and scientific computing, single computing devices can no longer meet their computational and memory requirements. Distributed computing systems consisting of multiple machines and multiple GPUs have become the core architecture supporting such high-performance computing workloads. Figure 1 As shown, such systems contain multiple computing nodes that are interconnected via an inter-node network (such as Ethernet). Each node contains multiple computing units (such as graphics processing units, GPUs) that are connected via high-speed interconnect links within the node (such as NVLink), thus forming a typical heterogeneous hardware environment characterized by "high bandwidth within nodes and relatively low bandwidth between nodes".

[0026] Based on this hardware, coordinating multiple computing units to collaboratively complete parallel computing tasks (such as distributed inference tasks of Sparse Mixture-of-Experts (MoE) models) requires Collective Communication, which is an operational paradigm for collaborative data exchange among a group of processes or devices.

[0027] A communication group is the core configurable logical unit for implementing aggregated communication. It defines the set of all devices participating in a communication operation and their interaction rules. A key system property of a communication group is that when a round of data exchange is completed, an inherent implicit synchronization barrier is formed within it, which automatically and forcibly aligns the progress of all devices within the group. This greatly simplifies the distributed programming model, but the timing of the synchronization directly determines the overall system latency.

[0028] Traditional flat, global all-to-all communication is a basic form of aggregated communication in such systems. It operates within a communication group containing all participating computing units, requiring each member to send and receive all data to and from all other members in the group. However, in the aforementioned heterogeneous networks, cross-node links become a performance bottleneck. This causes an implicit synchronization barrier for the entire communication group to wait for the slowest cross-node transmission to complete, resulting in long synchronization wait times and significant long-tail latency for all devices, leading to system inefficiency.

[0029] To overcome this bottleneck, layered full-to-full communication has been proposed as an improvement. Its core design intent is to reduce the number of devices participating in cross-node communication, thereby avoiding the bottleneck of large-scale global synchronization of devices in traditional flat full-to-full communication. Its typical implementation includes three stages: First, in the intra-node aggregation stage, each computing unit aggregates the data that needs to be sent to external nodes into a designated aggregation unit within its own node; then, in the cross-node communication stage, these aggregation units from various nodes form a communication group, and data aggregation communication (full-to-full exchange) is performed within this group; finally, in the intra-node distribution stage, each aggregation unit distributes the received data to the final target computing unit within its own node. However, while changing the communication architecture, this solution also introduced new drawbacks: First, the efficiency of inter-node communication has not been fundamentally improved. Because cross-node communication is restricted to between aggregation units, the data exchange mode within this communication group remains full or near-full exchange. Each aggregation unit must send full or near-full data to all other aggregation units, resulting in no structural reduction in the total effective data transmission volume on cross-node links, and the bandwidth bottleneck remains prominent.

[0030] Secondly, its "aggregation-relay" communication architecture introduces significant additional overhead. This architecture forces all cross-node data flows through the aggregation unit, increasing the number of data path hops from "sending computing unit → local aggregation unit → remote aggregation unit → target computing unit". The resulting queuing, processing, and forwarding overhead of data in the aggregation unit becomes a new burden superimposed on communication latency, and the aggregation unit itself is prone to becoming a system performance bottleneck.

[0031] In summary, existing all-to-all communication and layered improvement schemes cannot collaboratively solve the bottleneck of effective cross-node data transmission and the inherent processing overhead of layered architecture in heterogeneous hardware environments.

[0032] To systematically solve the above problems, this application proposes a logically sparse multi-node hierarchical communication scheme, the core of which lies in the logically sparse data exchange under a single global communication group architecture, and the pipelined overlapping execution of communication steps between and within nodes.

[0033] This application adopts the basic flattened all-to-all communication global communication group architecture, allowing all computing units in the distributed computing system to directly access a single global communication group as independent communication endpoints. This architecture naturally avoids the centralized forwarding overhead and performance bottlenecks caused by the traditional layered communication "aggregation-relay" model, while inheriting the implicit synchronization advantages of global aggregated communication. Based on this global communication group, this application specifically modifies the data exchange mode of global aggregated communication. First, each computing unit performs node-level aggregation on data replicas destined for the same target remote node, generating corresponding aggregated data packets. Then, it schedules and executes global aggregated communication operations, sending only valid aggregated data packets to designated computing units with predetermined mapping relationships within the target remote node, and filling non-target communication paths with invalid data. This achieves logically sparse data exchange. While maintaining the semantics and implicit synchronization mechanism of global aggregated communication, it structurally reduces the total amount of effective data transmission across node links, alleviating bandwidth pressure on low-speed links. Meanwhile, this application at least partially overlaps the steps of sending aggregated data packets between nodes with the steps of determining the data distribution path within nodes in terms of time. This hides the computational preparation overhead introduced by the layered architecture within the unavoidable cross-node transmission latency, effectively mitigating the inherent processing delay of the layered design and avoiding the accumulation of additional overhead. Finally, this application achieves synergistic optimization of cross-node communication efficiency and layered architecture processing overhead through logical sparsification reconstruction of global set communication, combined with pipeline overlap optimization of inter-node and intra-node communication steps. It adapts to the heterogeneous hardware characteristics of distributed computing systems, characterized by "high bandwidth within nodes and low bandwidth between nodes," and can efficiently support high-performance parallel computing tasks such as distributed inference of sparse expert hybrid models.

[0034] The logically sparse multi-node hierarchical communication scheme proposed in this application runs on a typical multi-machine, multi-card distributed computing system. For example... Figure 1 As shown, the system physically comprises at least two computing nodes ( Figure 1 (Using node 0 and node 1 as an example), these nodes communicate through an inter-node network (such as...) Figure 1 The “Ethernet” links marked in the diagram enable interconnection and form the basis for cross-machine communication.

[0035] Each computing node constitutes a complete multi-GPU subsystem. Specifically, each node is configured with multiple computing units. In this embodiment, the computing unit is preferably a graphics processing unit, hereinafter referred to as a GPU. Figure 1 As shown, both node 0 and node 1 contain two GPUs, and their hardware configurations are completely symmetrical. Multiple GPUs located within the same node are interconnected via high-speed interconnect links within the node (such as...). Figure 1 The link (marked with a bidirectional arrow indicating "NVLink") is used for connection and is designed for high-bandwidth, low-latency data exchange within a node.

[0036] Furthermore, such as Figure 1 As shown, each GPU integrates a control module and a communication module. The control module is the entity that executes the core communication control logic of this application, and is responsible for generating control decisions such as data aggregation and routing calculation. The communication module, under the scheduling of the control module (through the "control signal" connection between the two), specifically executes data transmission and reception operations via NVLink or Ethernet. Figure 1 The text uses arrows with different markings to visually illustrate the data paths for "intra-node communication" and "cross-node communication" driven by the communication module.

[0037] based on Figure 1 The symmetrical hardware architecture shown in this application embodiment exhibits inherent peer-to-peer equality among computing units: each computing node participating in the system contains the same number of GPUs, and each GPU has a unique local identifier within its respective node (e.g., ...). Figure 1 (0 or 1). In this embodiment, cross-node communication is preferentially performed between "peer GPUs", that is, in the target remote node, the GPU with the same local identifier as this GPU, such as GPU0 of node 0 and GPU0 of node 1 are peer GPUs, and this GPU only sends aggregate data packets to the peer GPU of the target node.

[0038] It should be noted that the technical solution of this application is not limited to the above-mentioned symmetric hardware configuration scenario, nor does it require that cross-node communication be performed only between peer GPUs. In practical applications, according to the system deployment requirements, a specified computing unit (i.e., a "predetermined mapping relationship") in the target remote node can be pre-configured for each GPU. As long as the mapping relationship is satisfied, the logical sparse data exchange of this application can be realized. Peer GPU communication under the symmetric architecture is only a preferred implementation method.

[0039] Figure 2 This is a flowchart illustrating a logically sparse multi-node hierarchical communication method according to some embodiments of this application. In some embodiments, the logically sparse multi-node hierarchical communication method is composed of... Figure 1 The control module in the system shown executes the following steps: S210, perform node-level aggregation on data copies belonging to the computing unit itself that need to be sent to the same target remote node, and generate aggregated data packets to be sent to different target remote nodes.

[0040] In a multi-node environment, this application breaks down the data distribution process into two stages: cross-node distribution and intra-node distribution. Before cross-node distribution, each GPU first performs node-level aggregation based on the target node information corresponding to the data to be sent. Only data pointing to remote nodes is sent to the peer GPUs within the corresponding node, thereby reducing cross-node communication. When multiple target GPUs are located on the same target node, only a single data copy is retained for cross-node transmission to reduce the amount of duplicate data transmission across nodes.

[0041] S220, the communication module is scheduled to perform a collective communication operation in a single global communication group, wherein the aggregated data packet is sent to a designated computing unit in the target remote node that has a predetermined mapping relationship with the computing unit, and invalid data is filled on the non-target communication path of the collective communication to realize logically sparse data exchange in the collective communication, and each computing unit in the distributed computing system directly accesses the single global communication group as an independent communication endpoint.

[0042] To mitigate the high synchronization overhead and long-tail latency caused by decoupling communication progress between nodes in a hierarchical structure, this application utilizes the implicit synchronization barrier inherent in global set communication to achieve "soft synchronization" between nodes. The following design is adopted in the cross-node communication phase: First, during the system initialization phase, a single global communication group is configured, and each computing unit in the system is directly connected to this group as an independent communication endpoint.

[0043] During the cross-node communication phase, each GPU performs aggregated communication operations within the global communication group based on the data aggregated at the node level. The core of this approach lies in achieving logical sparsity through zero-byte padding: each GPU only sends valid aggregated data packets to its peer GPUs within the target node, while filling all non-target communication paths in this aggregated communication with invalid data (such as empty data or zero bytes). This completes a global aggregated communication at the physical level, but constructs a point-to-point sparse transmission structure at the logical level.

[0044] This design structurally reduces the amount of effective data transfer across nodes while fully preserving the implicit synchronization characteristics inherent in global collection communication at round boundaries. This allows all GPUs to automatically align their progress at the end of communication, thereby avoiding progress decoupling caused by physical grouping or stage splitting, and alleviating device idle waiting and long-tail latency issues.

[0045] S230, when this computing unit receives the aggregated data packet as the designated computing unit, it determines the data distribution path within this node according to the routing information in the aggregated data packet, and distributes the data in the aggregated data packet to one or more target computing units within this node through the intra-node communication group within this node.

[0046] When this computing unit receives a cross-node aggregated data packet as a designated computing unit (i.e., a peer GPU), it triggers intra-node secondary distribution. At this time, the computing unit's own control module determines the complete data distribution path to be distributed to one or more target GPUs within the node in one go, based on the routing information in the aggregated data packet. "Determining in one go" means that the control module calculates all the routes required to deliver all the data to be distributed to their respective final target GPUs in parallel, rather than performing multiple, serial calculations. Subsequently, it schedules its own communication module to accurately forward the data to each target GPU through the intra-node communication group.

[0047] When the target GPU receives a data packet through the intra-node communication group, its control module can immediately schedule subsequent processing of the data in that packet (e.g., initiating inference by the hosted expert). Since the complete distribution path within the node is predetermined before the data arrives, the target GPU does not need to perform any additional routing decisions or forwarding operations after receiving the data, thus achieving direct triggering of processing actions.

[0048] In some embodiments of this application, the intra-node communication group is pre-built during system initialization and consists of all GPUs within the node connected via intra-node high-speed interconnect links.

[0049] Furthermore, in a preferred embodiment, the step of determining the data distribution path within a node (represented by calculating routing index information) can be executed at least partially in time overlapping with the step of sending cross-node aggregated data packets. That is, while the scheduling communication module performs time-consuming cross-node data transmission, the control module calculates the routing information required for secondary distribution within the node in parallel. Through this fine-grained pipeline mechanism, the additional computational and preparation overhead introduced by the hierarchical structure is hidden within the unavoidable cross-node transmission delay, thereby avoiding new significant processing delays caused by stage division.

[0050] In some embodiments of this application, the logical sparse multi-node hierarchical communication method can be efficiently applied to distributed inference tasks of sparse expert hybrid models, including the distribution of model input tokens and the aggregation of expert calculation results.

[0051] The following text combines Figure 1 The system architecture shown here, taking an input token that needs to be distributed in the MoE layer as an example, details the execution process of the method.

[0052] Assume the token is located on GPU0 of node 0.

[0053] Routing decision: First, the control module deployed on GPU0 calls the MoE gating network to calculate the routing decision based on the token. The output of the gating network determines the multiple target experts to which the token needs to be sent, and thus clarifies the physical location of each target expert: the target remote node (e.g., node 1) and the target GPUs within that node that carry the expert (e.g., GPU0 and GPU1 of node 1).

[0054] Node granularity aggregation: The control module then performs node-level aggregation. It identifies that the two target GPUs (GPU0, GPU1) are located on the same target remote node (Node 1). Therefore, it merges the two logical copies of the token destined for the same node 1 to generate an aggregated data packet destined for node 1. This aggregated packet contains the token data and routing information indicating where it should be distributed to GPU0 and GPU1 within node 1.

[0055] Logical sparse global communication: Next, the control module of GPU0 schedules its communication module to initiate aggregated communication within a single global communication group. Based on a predetermined mapping (in this embodiment, the "peer GPU" rule), it only sends the aforementioned valid aggregated data packets to GPUs with the same local identifier in the target node (Node 1) (i.e., GPU0 of Node 1, which is a peer GPU). During this global aggregated communication, GPU0's communication module fills all other non-target communication paths, such as GPU1 of Node 1, with invalid data (zero bytes).

[0056] Secondary distribution and processing within the node: Upon receiving the aggregated data packet, the control module of GPU0 (the peer GPU) on node 1 immediately parses the routing information within the packet, determining in one go the data distribution path that needs to distribute the token to itself (GPU0) and GPU1 on the same node. Subsequently, it schedules its own communication module to distribute the token to GPU0 and GPU1 on node 1 through the intra-node communication group composed of NVLink. After GPU0 and GPU1 receive the token through intra-node communication, their respective control modules immediately trigger the expert models they carry to begin inference computation.

[0057] The aggregation process of expert calculation results can be completed using a sparse communication method that is logically symmetrical with the above token distribution process.

[0058] To verify the technical effectiveness of the layered sparse communication mechanism described in this application in a multi-machine, multi-GPU distributed system, a comparative experiment was conducted in a real GPU cluster environment using one embodiment of this application.

[0059] The experimental platform is a multi-node distributed system containing two logical computing nodes, each configured with two NVIDIA A100-SXM4 (80GB) GPUs. The GPUs within a node are interconnected via NVLink, with each GPU having 12 links and a unidirectional bandwidth of 50GB / s. The nodes are interconnected via 25Gbps Ethernet to simulate the typical hardware structure difference of "high bandwidth within nodes and low bandwidth between nodes" in a real-world multi-node deployment.

[0060] To construct a representative high-communication-pressure operational load, the experiment selected inference tasks for three SMoE models as the system test scenario: OLMoE, DeepSeek-v2-lite-chat, and Qwen3-30B-A3B. All of these models contain multiple MoE layers, requiring frequent All-to-All aggregate communication during inference, thus placing continuous pressure on the multi-machine, multi-card communication mechanism. Experimental data used was WikiText-2-v1. The inference runtime configuration was: batch size=256, prefill length=128, decode length=16, to ensure a stable and reproducible communication load intensity during testing.

[0061] The comparison scheme is the existing flat global all-to-all communication mechanism, while the scheme in this application is a hierarchical communication mechanism based on a logically sparse structure (abbreviated as HSC). During the experiment, the following were statistically analyzed: cross-node communication volume, total all-to-all communication time, and end-to-end inference latency. All results are averages from multiple rounds of execution. The experimental results are shown in Table 1.

[0062] Table 1:

[0063] Table 1 shows that in a multi-model experimental environment with multiple machines and multiple cards, the method (HSC) described in this application reduces the all-to-all communication time by an average of about 35.19% and the cross-node communication traffic by an average of about 35.64% across all models. This verifies that the communication mode can significantly alleviate the pressure on the low-bandwidth link side and solve the problem that the cross-node link becomes the main performance bottleneck in a multi-machine, multi-card environment. An average end-to-end inference latency acceleration of about 1.24x was observed on different models, which verifies that the communication structure reconstruction effectively improves the overall inference latency in a multi-node environment by reducing communication overhead and synchronization overhead.

[0064] This application aims to address two core issues in traditional layered full-to-full communication: the efficiency bottleneck of cross-node full-switch and the inherent processing overhead of the layered architecture. Logical sparsity switching fundamentally overcomes the inefficiency of full-switch, while pipelined overlapping design effectively alleviates the inherent processing overhead of the layered architecture. The aforementioned experimental data has directly verified the practical effects of these two core optimizations. Traditional layered full-to-full solutions require the introduction of additional aggregation, relay, and secondary distribution logic. Different cluster deployment optimization strategies lead to significant differences in this overhead, making it impossible to establish a unified and reproducible performance benchmark. Therefore, no direct quantitative comparison data is provided. However, the aforementioned experiments have demonstrated, from both a principle and effect perspective, that this application systematically solves the two inherent defects of traditional layered solutions through the synergistic optimization of "sparseness + pipeline."

[0065] Figure 3 This is a schematic diagram of a logically sparse multi-node hierarchical communication device according to some embodiments of this application. Figure 3 As shown, the logical sparse multi-node hierarchical communication device 300 includes a node granularity aggregation module 310, an inter-node communication module 320, and an intra-node communication module 330. The logical sparse multi-node hierarchical communication function can be achieved by... Figure 1 The control module in the system shown executes the following: The node granularity aggregation module 310 is used to perform node granularity aggregation on data copies that belong to the computing unit itself and need to be sent to the same target remote node, and generate aggregated data packets to be sent to different target remote nodes. The inter-node communication module 320 is used to schedule the communication module to perform aggregated communication operations in a single global communication group. In this operation, the aggregated data packet is sent to a designated computing unit in the target remote node that has a predetermined mapping relationship with the computing unit. Invalid data is filled on the non-target communication path of the aggregated communication to achieve logically sparse data exchange. Each computing unit in the distributed computing system is directly connected to the single global communication group as an independent communication endpoint. The intra-node communication module 330 is used to determine the data distribution path within the node based on the routing information in the aggregated data packet after the computing unit receives the aggregated data packet as the designated computing unit, and distribute the data in the aggregated data packet to one or more target computing units within the node through the intra-node communication group within the node.

[0066] In summary, the logically sparse multi-node hierarchical communication method and apparatus provided in the embodiments of this application firstly construct a "single global communication group" and realize "logically sparse data exchange" under this architecture. Each computing unit prepares data locally through "node-level aggregation," and then, within this global communication group, sends only valid aggregated data packets to peer computing units in the target remote node, while filling all non-target communication paths with invalid data. This significantly reduces the total amount of valid data that must be transmitted on cross-node links from the perspectives of communication architecture and data exchange mode, directly alleviating the bandwidth pressure of low-speed links and overcoming the efficiency bottleneck of effective data transmission between nodes. Secondly, by pipelined overlapping execution of "inter-node data transmission" and "intra-node routing calculation," the intra-node data distribution path is determined in parallel while scheduling and executing long-term cross-node data transmission. This allows the additional computation and preparation overhead introduced by the layered design to be "hidden" within the unavoidable transmission latency, avoiding the significant latency burden that the existing "three-stage" design itself may bring, and effectively alleviating the inherent processing overhead of the layered architecture. Ultimately, the above technical features work together to achieve synergistic optimization of communication efficiency and processing overhead, providing a high-performance, low-latency communication solution for multi-machine, multi-card distributed computing systems, which can fully meet the high throughput and low latency requirements of MoE distributed inference scenarios.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0068] Although the subject matter described herein is provided in the general context of a computer system execution scenario combining an operating system and applications, those skilled in the art will recognize that other implementations can also be performed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside on both local and remote memory storage devices.

[0069] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0070] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A logically sparse multi-node hierarchical communication method, applied to a distributed computing system composed of multiple computing nodes, each computing node containing multiple computing units, wherein a control module and a communication module are deployed on the computing unit, characterized in that, The method is executed by the control module and includes: Node-level aggregation is performed on data copies belonging to the computing unit itself that need to be sent to the same target remote node to generate aggregated data packets to be sent to different target remote nodes; The communication module is scheduled to perform aggregated communication operations in a single global communication group. The aggregated data packet is sent to a designated computing unit in the target remote node that has a predetermined mapping relationship with the computing unit. Invalid data is filled on the non-target communication path of the aggregated communication to realize logically sparse data exchange in the aggregated communication. Each computing unit in the distributed computing system is directly connected to the single global communication group as an independent communication endpoint. When this computing unit receives the aggregated data packet as the designated computing unit, it determines the data distribution path within the node based on the routing information in the aggregated data packet, and distributes the data in the aggregated data packet to one or more target computing units within the node through the intra-node communication group within the node.

2. The method according to claim 1, characterized in that: Determining the data distribution path within this node includes determining, based on the routing information in the aggregated data packet, the data distribution path that needs to be distributed to one or more target computing units within this node in one go. When this computing unit receives a data packet from another computing unit within the same node as the target computing unit, it triggers the processing of the data in that data packet.

3. The method according to claim 2, characterized in that: The single global communication group and the communication groups within each node are pre-built and configured during the system initialization phase.

4. The method according to claim 3, characterized in that: The step of scheduling the communication module to send aggregated data packets to a designated computing unit of the target remote node and the step of determining the data distribution path within the local node are executed in time at least partially overlapping.

5. The method according to claim 4, characterized in that: The distributed computing system is a multi-machine, multi-card system; The computing unit is a graphics processor, and multiple graphics processors within the same computing node are interconnected through intra-node high-bandwidth interconnect links to perform data transmission of the intra-node communication group.

6. The method according to claim 5, characterized in that: Each computing node participating in the communication contains the same number of graphics processors, and each graphics processor has a unique local identifier within its own node; The designated computing unit in the target remote node that has a predetermined mapping relationship with this computing unit is the graphics processor in the target remote node that has the same local identifier as this graphics processor.

7. The method according to claim 6, characterized in that: The method is applied to distributed reasoning in sparse expert hybrid models, including the distribution of model input tokens and the aggregation of reasoning results.

8. The method according to claim 7, characterized in that: The data to be distributed is the input token of the MoE layer; For input tokens that are the same as those of the target remote nodes, node-level aggregation is performed to generate aggregated data packets. The graphics processors belonging to the target remote nodes and the target experts they carry are determined by the gated network of MoE based on the input tokens. The communication module is scheduled to perform aggregated communication operations in a single global communication group, route the aggregated data packets to the graphics processor that is equivalent to the graphics processor in the target remote node, and fill invalid data on non-target communication paths; The peer graphics processor is the graphics processor within the target remote node that has the same local identifier as this graphics processor; After receiving the aggregated data packet, the peer graphics processor determines the data distribution path within the target remote node, and distributes the input token to the graphics processor carrying the corresponding target expert within the local node through the intra-node communication group of the target remote node, so as to perform inference calculation.

9. The method according to claim 8, characterized in that, The step of aggregating input tokens with the same target remote node at the node level to generate an aggregated data packet includes: Multiple copies of the same input token that need to be sent to the same target remote node are merged into a single aggregate data packet.

10. A logically sparse multi-node hierarchical communication device, applied to a distributed computing system composed of multiple computing nodes, each computing node containing multiple computing units, wherein a control module and a communication module are deployed on the computing unit, characterized in that, The device is implemented by the control module and includes: The node granularity aggregation module is used to perform node granularity aggregation on data copies belonging to the computing unit itself that need to be sent to the same target remote node, and generate aggregated data packets to be sent to different target remote nodes. An inter-node communication module is used to schedule the communication module to perform aggregated communication operations in a single global communication group. The aggregated data packet is sent to a designated computing unit in the target remote node that has a predetermined mapping relationship with the computing unit. Invalid data is filled on the non-target communication path of the aggregated communication to achieve logically sparse data exchange. Each computing unit in the distributed computing system is directly connected to the single global communication group as an independent communication endpoint. The intra-node communication module is used to determine the data distribution path within the node based on the routing information in the aggregated data packet after the computing unit receives the aggregated data packet as the designated computing unit, and distribute the data in the aggregated data packet to one or more target computing units within the node through the intra-node communication group within the node.

Images