In-situ computing method and system for dynamic communication

Abstract

The application provides a dynamic communication-oriented switch-in computing method and system, and belongs to the technical field of computer interconnection communication, and comprises the following steps: a source device generates a communication request based on a logical data identifier and a dynamic target set list associated with a data unit; a network switch analyzes the target set list and performs a dynamic multicast or dynamic reduction operation on the data unit; and a destination device dynamically maps the logical data identifier to a local memory address and completes data access. In addition, through a pipeline scheduling mechanism, the fine-grained state of the data unit in the distribution, computing and merging stages is tracked, so that the merging stage of one data unit and the distribution stage of another data unit are concurrently executed. The application can fundamentally reduce redundant communication, natively support dynamic communication, and improve end-to-end system performance through balanced bidirectional link bandwidth utilization.

Description

Technical Field

[0001] This invention relates to the field of computer interconnection and communication technology, specifically to a method and system for in-switch computing for dynamic communication, particularly a dynamic in-switch computing method for dynamic communication between devices, and especially a communication acceleration method for in-switch computing in dynamic communication scenarios such as Mixture-of-Experts (MoE). Background Technology

[0002] Most current mainstream advanced AI models adopt a Mixture-of-Experts (MoE) structure to reduce computational requirements while maintaining model capacity scalability. MoE models split the feedforward network into multiple experts, ensuring each token activates and computes only a small number of experts. Due to the massive number of expert parameters, Expert Parallelism (EP) is typically used to distribute different experts across different accelerators for distributed deployment of large-scale MoE models within the system. When a token enters the MoE layer, it is first distributed to the accelerator of the target expert based on the gating network results. After expert-side computation is completed, the outputs of multiple experts are sent back to the original device where the token resides. This all-to-all communication phase between these two devices, commonly referred to as All-to-All Dispatch and All-to-All Combine, is one of the main performance bottlenecks for efficiently deploying MoE models. Figure 1 The communication mode in the EP scenario is shown.

[0003] Existing communication optimization schemes can be broadly categorized into two types: The first type is software communication optimization methods, represented by DeepEP, FasterMoE, Tutel, Comet, and CCFuser. These methods primarily improve performance by customizing communication libraries, reducing kernel startup through operator fusion, or achieving a certain degree of computation-communication overlap at various granularities. The second type is in-switch computation schemes, represented by NVLinkSHARP (NVLS). These schemes are designed for static set communication and can achieve multicast or reduction on the switch side through the multimem instruction. However, they rely on a fixed target set and symmetric address layout, and are designed for rule-based communication such as AllGather and Reduce-Scatter.

[0004] The dynamic communication mode in MoE contains a large amount of redundant data transmission, such as Figure 2As shown. During the distribution phase, the same token often needs to be sent to multiple accelerators, and the same data from the source device to the switch will be sent repeatedly. During the merging phase, multiple expert outputs need to be sent back and summed, which will also form multiple aggregateable but separately transmitted data on the link from the switch to the destination device. Theoretically, these redundant data transmissions can be eliminated through multicast or protocols within the switch, thereby reducing the communication load and significantly improving the communication speed. Figure 3 This demonstrates the enormous potential for accelerating sparse communication through computation within the switch. Existing software communication optimization methods, whether accelerating all-to-all communication itself or hiding communication time to some extent by computing communication overlap, fail to identify and utilize identical or aggregable redundant transmissions. These software solutions inherently transmit more unnecessary data, resulting in longer communication times and difficulty in hiding communication in the end-to-end.

[0005] Existing in-switch computing, such as NVLS, can only handle regular static set communication and cannot directly adapt to irregular dynamic communication in scenarios like MoE. Figure 4 As shown, (a) and (b) illustrate static and dynamic communication modes, respectively, while (c) and (d) demonstrate that existing in-switch computation is adapted to static communication mode but cannot be applied to dynamic communication mode. Forcibly rewriting dynamic communication into a static AllGather / Reduce-Scatter to utilize existing in-switch computation technology would introduce a large amount of invalid traffic, negating the benefits of in-switch computation. Furthermore, among other things... Figure 3 (a) Communication traffic breakdown of the DeepSeek-V3 model running on a 32-device architecture. Figure 3 (b) shows the theoretical speedup that can be achieved by removing redundant communication. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method and system for in-switch computing for dynamic communication.

[0007] According to the present invention, a switch-based computation method for dynamic communication is applied to a distributed system including a source device, a network switch, and at least one destination device, comprising the following steps: The source device generates a communication request based on a logical data identifier associated with a data unit and a target set list describing a dynamic target set of the data unit. The network switch parses the target set list contained in the communication request to perform a dynamic multicast operation or a dynamic reduction operation on the data unit, wherein the dynamic multicast operation constitutes a distribution phase, and the dynamic reduction operation constitutes a calculation phase and a merging phase. The destination device dynamically maps the logical data identifier in the communication request to its local memory address, and completes data access; and A pipeline scheduling mechanism is used to track the fine-grained state of the data unit in the distribution phase, the computation phase, and the merging phase, so as to achieve concurrent execution of the merging phase of one data unit and the distribution phase of another data unit.

[0008] Preferably, the step of dynamically mapping logical data identifiers to local memory addresses by the destination device is performed by a hardware memory manager within the destination device, which implements the mapping by maintaining a generational layout mapping table.

[0009] Preferably, the hardware memory manager further includes a cache module for caching the mapping relationship.

[0010] Preferably, the tracking is implemented using a token tracker to track the fine-grained state of the data unit.

[0011] Preferably, the token tracker includes: A status table is used to track the arrival and computation status of data on the destination device side; and An output ready table is used to track the ready status of multiple computation results on the source device side.

[0012] Preferably, the pipeline scheduling mechanism further includes a scheduler implemented with a persistent kernel, which polls the status of the token tracker to execute ready tasks.

[0013] Preferably, the step of generating a communication request includes: the source device executing a predefined dynamic multicast write instruction or a dynamic reduction read instruction.

[0014] Preferably, the communication request further includes weight information corresponding to the destination device, and before the network switch performs the dynamic reduction operation, the data units distributed to the destination device via the dynamic multicast operation are multiplied by the weight information corresponding to the destination device, so that the network switch performs an unweighted reduction operation.

[0015] The present invention also provides an in-switch computing system for dynamic communication, comprising a source device, a network switch, and at least one destination device, the system comprising: The source device is configured to generate a communication request based on a logical data identifier associated with a data unit and a target set list describing a dynamic target set of the data unit. The network switch is configured to parse the target set list contained in the communication request to perform a dynamic multicast operation or a dynamic reduction operation on the data unit, wherein the dynamic multicast operation constitutes a distribution phase and the dynamic reduction operation constitutes a calculation phase and a merging phase. The destination device is configured to dynamically map a logical data identifier to a local memory address of the destination device based on the logical data identifier in the communication request; and A pipeline scheduling module is configured to track the fine-grained state of the data unit in the distribution phase, the computation phase, and the merging phase, so as to enable concurrent execution of the merging phase of one data unit and the distribution phase of another data unit.

[0016] Preferably, the destination device includes a hardware memory manager configured to implement the mapping by maintaining a generational layout mapping table.

[0017] Preferably, the hardware memory manager further includes a cache module for caching the mapping relationship.

[0018] Preferably, the tracking is implemented using a token tracker to track the fine-grained state of the data unit.

[0019] Preferably, the token tracker includes: A status table is used to track the arrival and computation status of data on the destination device side; and An output ready table is used to track the ready status of multiple computation results on the source device side.

[0020] Preferably, the pipeline scheduling mechanism further includes a scheduler implemented with a persistent kernel, which polls the status of the token tracker to execute ready tasks.

[0021] Preferably, the process of generating the communication request is as follows: the source device executes a predefined dynamic multicast write instruction or dynamic revert read instruction.

[0022] Preferably, the communication request further includes weight information corresponding to the destination device, and before the network switch performs the dynamic reduction operation, the data units distributed to the destination device via the dynamic multicast operation are multiplied by the weight information corresponding to the destination device, so that the network switch performs an unweighted reduction operation.

[0023] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention fundamentally eliminates redundant data transmission in dynamic irregular communication by implementing multicast and reduction based on dynamic target lists within the switch, significantly reducing communication traffic.

[0024] 2. The solution of this invention is specifically designed for dynamic and irregular communication. It does not require the transformation of dynamic communication mode into static set communication, thus avoiding the large amount of invalid padding data and performance overhead introduced by it. It can efficiently handle complex communication scenarios with real-time changes in the target set and address asymmetry.

[0025] 3. By introducing a kernel fusion and pipeline scheduling mechanism of the token center, this invention enables the concurrent execution of two complementary communication modes, distribution and merging, at a fine-grained level, achieving balanced utilization of bidirectional link bandwidth and truly transforming the advantage of communication traffic reduction into end-to-end latency reduction and performance improvement.

[0026] 4. This invention offloads the complex remote address management, status tracking and synchronization work from software to dedicated hardware logic, which greatly simplifies the implementation logic of the upper-layer software and reduces synchronization overhead and additional computing resource consumption. Attached Figure Description

[0027] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A schematic diagram of the dynamic communication mode in a parallel scenario of MoE experts; Figure 2 A schematic diagram of redundant transmission and multicast / reduction opportunities within a switch in MoE dynamic communication. Figure 3 A schematic diagram showing the ratio of redundant traffic in MoE dynamic communication and the theoretical speedup ratio for implementing in-switch computation. Figure 4 This diagram illustrates static and dynamic communication modes, as well as the adaptation of existing in-switch computing schemes to static communication modes and their incompatibility with dynamic communication modes. Figure 5 A schematic diagram illustrating an example of the initialization process for a dynamically multi-target addressable memory region; Figure 6 A schematic diagram illustrating an example of an extension of the instruction set related to computation within a dynamic switch; Figure 7 A schematic diagram illustrating an example of a link-layer message format to support dynamic switching computation. Figure 8 This diagram illustrates the relationship between algebraic indexes and layout indexes, explaining the reason for introducing the mapping from algebraic indexes to memory layout indexes and the method of generating dense layouts at the destination. Figure 9A schematic diagram illustrating the overall hardware structure and data flow example for dynamic multi-target addressing, including the structure of the source device, switch, and destination device; Figure 10 This diagram illustrates the pipelined execution of the token center kernel fusion, including the token-level dependencies between the four main computation stages, the allocation of hardware resources, and a comparison of the pipelined execution of existing solutions and the solution proposed in this invention. Figure 11 A schematic diagram illustrating the main structure of a token tracker; Figure 12 This is a schematic diagram illustrating the end-to-end training speedup ratio of the proposed scheme (DySHARP) compared to other internationally advanced optimization schemes under different model configurations. Figure 13 This is a schematic diagram showing the end-to-end inference speedup ratio of the proposed scheme (DySHARP) compared to other internationally advanced optimization schemes under different model configurations. Figure 14 This is a schematic diagram showing the speedup of a single MoE layer of the proposed scheme (DySHARP) compared to other internationally advanced optimization schemes under different model configurations; Figure 15 This is a schematic diagram comparing the communication traffic of the proposed solution (DySHARP) and advanced software solutions or existing in-switch computing solutions. Figure 16 A flowchart illustrating a method for in-switch computation for dynamic communication according to an embodiment of the present invention; Figure 17 This is a timing diagram of intra-system signaling interaction provided in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0029] Example 1 This embodiment provides a basic implementation scheme for a switch-based computing method and system for dynamic communication. As an optional implementation, this scheme can be applied to a distributed system consisting of multiple computing devices (e.g., graphics processing units) interconnected through one or more network switches. Specifically, this embodiment uses a scenario of expert parallel deployment with hybrid expert models as an example to illustrate how to efficiently handle dynamic and irregular communication through hardware and software co-design.

[0030] The system in this embodiment mainly includes a source device as the communication initiator, one or more devices as communication targets, i.e., destination devices, and a network switch connecting these devices. The core of this application lies in enhancing and extending the functionality of the existing hardware architecture. This enhancement and extension mainly includes: extending the load / storage unit in the source device to enable it to handle new dynamic communication commands; enhancing the routing and computation logic of the network switch to enable it to parse dynamic target lists and perform corresponding multicast or reduction operations; and integrating a dedicated hardware memory manager into the network interface controller (usually called a Hub) of the destination device. This manager contains an algebraic layout mapping table and an algebraic layout mapping cache table for accelerating queries.

[0031] To better understand the technical solution of this application, the following will be combined with... Figure 16 The flowchart shown is Figure 17 The signaling interaction timing diagram shown illustrates the complete workflow of this embodiment. This workflow can be divided into several closely interconnected stages, including initialization, distribution, calculation and scheduling, and merging.

[0032] During the initialization phase (corresponding to) Figure 16 In step S1601, the upper-layer application or runtime library will call a predefined application programming interface to configure the dynamic communication environment. Specifically, this process may include calling a function such as cuDyMulticastCreate to create a dynamic multicast communication object, and calling the cuDyMulticastBindAddr function to bind memory regions for the subsequent distribution and merging phases. Understandably, this step logically establishes a multi-target address space and prepares the necessary hardware and software resources for communication operations within that space.

[0033] The distribution phase then begins, aiming to efficiently send a data unit from the source device to multiple dynamically determined destination devices. In a typical hybrid expert model application, the data unit can be a token. Assuming a data unit token A located on the source device GPU0, after computation by the gating network, it is determined that it needs to be sent to expert 1 deployed on destination device GPU1 and expert 2 deployed on destination device GPU2 for processing.

[0034] Accordingly, the distribution preparation and execution process (corresponding to) Figure 16Steps S1602 and S1603 are as follows: 1. The computing core of the source device GPU0 generates a target set list for the data unit token A based on the gating result. This list contains the identifiers of the target experts (expert 1, expert 2). Simultaneously, the system assigns a globally unique logical data identifier to the data unit token A within the distributed system, which can be referred to as an "algebraic index" in this application. This algebraic index will serve as the main part of the multi-target address. 2. The computing core of the source device GPU0 executes an extended dynamic multicast write instruction, such as the dymultimem.st instruction. The operands of this instruction include: a register pointing to the data to be sent, the multi-target address formed by the algebraic index, the number of targets for this communication (2 in this example), and the memory base address for storing the target set list. 3. The load / storage unit of the source device GPU0 is extended to recognize the dymultimem.st instruction. When executing this instruction, the load / storage unit not only reads the data to be sent but also retrieves the specific target set list (expert 1, expert 2) from memory based on the target list base address and the number of targets provided in the instruction. 4. Subsequently, the payload / storage unit constructs one or more extended link-layer packets. The header of the first flow control unit of this packet contains key control information, such as a 48-bit multi-destination address, a 1-bit stage identifier to distinguish between distribution and merging operations (set to "distribution" in this case), and a 15-bit target quantity (set to 2 in this case). The one or more target extended flow control units that follow carry specific target expert identifiers, while the data itself serves as the payload of the packet.

[0035] When the extended message reaches the network switch via the network link, the switch will perform a dynamic multicast operation (corresponding to...). Figure 16 Step S1604): 1. The routing logic of the network switch is enhanced to enable it to parse the extended packet format. It first reads the number of targets in the packet header, and then parses the subsequent target extended flow control unit to obtain the complete dynamic target set list. 2. The switch queries its internal routing table based on the target expert identifiers (expert 1, expert 2) to determine which output ports the physical devices (GPU1, GPU2) of these experts are connected to. 3. The switch copies the data payload of the packet and forwards it from the output ports leading to GPU1 and GPU2 respectively. It can be understood that this copying and forwarding based on the dynamic target list, completed within the switch, is called dynamic multicast. It fundamentally solves the redundant transmission problem caused by the source device needing to send the same data multiple times in traditional solutions, such as multicast opportunities within the switch. Experimental data also shows that communication traffic can be significantly reduced through this method. For example, the normalized communication traffic of the scheme in this application can be reduced by nearly 50% compared to the benchmark scheme.

[0036] After being forwarded by the switch, the message reaches its destination device (e.g., GPU1 and GPU2). On the destination device side, the integrated hardware memory manager performs efficient address mapping and data writing (corresponding to...). Figure 16 Step S1605): 1. After the Hub of the destination device GPU1 receives the message, its internal hardware memory manager parses the multi-target address in the message, which is the algebraic index of the data unit token A. 2. The algebraic index represents a global, logically potentially very sparse address space (algebraic tensor). To achieve efficient memory access and subsequent computation on the destination device, it needs to be mapped to a local, physically contiguous, dense memory space (layout tensor). 3. The hardware memory manager queries its internal algebraic layout mapping table. Assuming this is the first time the algebraic index of token A has arrived at GPU1, there is no mapping entry for this index in the algebraic layout mapping table. At this time, the manager allocates a new, dense memory block in local memory for this algebraic index and generates a locally unique layout index for it, for example, LIdx=0. Subsequently, the manager writes the mapping relationship "algebraic index -> layout index" into the algebraic layout mapping table. 4. To improve the efficiency of subsequent accesses, this mapping relationship can also be cached in the algebraic layout mapping cache table. The algebraic layout mapping cache table functions similarly to the address translation lookahead in a traditional processor, significantly accelerating the address translation process. 5. Once the layout index is obtained, the hardware memory manager can calculate the final local physical memory write address and instruct the data path to write the data payload from the message to that address.

[0037] After the data is written to the destination device, the system enters the computation and pipeline scheduling phase. This embodiment employs a token-centric kernel fusion mechanism, which uses a sophisticated token tracker and scheduler to achieve fine-grained pipelined concurrent execution of multiple stages such as distribution, computation, and merging.

[0038] The token tracker's structure mainly consists of a block status table (TS table) and a token sequence number table (TID table) deployed on the destination device side, and an output ready table (OR table) deployed on the source device side.

[0039] As an optional implementation method, pipeline scheduling and computation process (corresponding to...) Figure 16Steps S1606 and S1607 are as follows: 1. After the hardware memory manager on the destination device GPU1 completes the writing of the data for token A, it updates the local block status table. The TS table tracks the arrival status of the data required for each computation block, for example, a counter can record the number of data slices that have arrived. 2. A scheduler implemented using a persistent kernel runs within the system. This scheduler is a resident computation kernel that continuously polls the status of the TS table. 3. When the scheduler detects that the status of a computation block in the TS table has changed from not ready to fully ready, it recognizes a task ready boundary. At this time, the scheduler will immediately retrieve the computation task (e.g., matrix multiplication) corresponding to the computation block from the task queue and assign it to an available computation core for execution. 4. After the expert computation task is completed, the destination device GPU1 needs to notify the source device GPU0. To do this, it queries the token sequence number table. The TID table records the correspondence between each computation block and the original data unit (token). By querying the TID table, GPU1 can know that the computation block that has just completed the computation belongs to token A. 5. GPU1 sends a computation completion notification for token A to the source device GPU0.

[0040] The final stage of the process is the merging stage, which aims to efficiently aggregate the computation results scattered across multiple destination devices back to the source device.

[0041] Accordingly, the merge preparation and execution process (corresponding to) Figure 16Steps S1608, S1609, and S1610 are as follows: 1. After receiving a computation completion notification from the destination devices (GPU1 and GPU2), the source device GPU0 updates its local output ready table. The OR table maintains an entry for each active token in the system, which contains a ready counter to record how many expert computation results the token has received. For token A, its corresponding ready counter is incremented by one for each completion notification received. 2. The persistent kernel scheduler also monitors the OR table on the source device side. When it detects that the ready counter of token A has reached a preset threshold (e.g., top-k=2, indicating that the two expert computations required for it have been completed), it means that the merging operation of token A is ready. 3. The source device GPU0 then executes an extended dynamic reduction instruction, such as the dymultimem.ld_reduce instruction. The operands of this instruction are similar to those in the distribution phase, including the algebraic index of token A (as a multi-target address), the number of targets (2), and the base address pointing to the same target set list (expert 1, expert 2). 4. A merge request message is generated and sent to the network switch, but the stage identifier is set to "Merge". 5. Upon receiving the merge request, the network switch parses the target set list and creates a reduction context for the request. This context records the data that needs to be waited for from GPU1 and GPU2. 6. The switch forwards the data read request to GPU1 and GPU2. Upon receiving the request, these two destination devices read the previously computed result belonging to token A from their local memory and send it back to the switch. 7. The reduction logic unit within the switch dynamically detects and waits for the arrival of data from all participants (GPU1 and GPU2) within the reduction context. Once all data has arrived, the switch performs a reduction calculation internally (e.g., floating-point summation). 8. After reduction, the switch sends only this single final, aggregated result back to the source device GPU0. This achieves in-switch reduction opportunities, avoiding bandwidth waste caused by transmitting multiple partial results across the network. 9. The source device GPU0 receives the final result, completing a full distribution-computation-merge process.

[0042] A key benefit of this embodiment is that, through the fine-grained pipeline scheduling mechanism of the token center, concurrent execution of different data units at different processing stages can be achieved. In the execution timeline of this application, when a data unit (e.g., token A) has completed computation and entered the merging stage, its communication direction is from the destination device to the source device; simultaneously, another new data unit (e.g., token B) can begin its distribution stage, with its communication direction from the source device to the destination device. It is understood that these two communication modes complement each other in the data flow direction, and their concurrent execution allows the bidirectional network link connecting the computing device and the switch to be utilized simultaneously and evenly, which contrasts sharply with the asymmetrical use of link bandwidth in existing solutions. In this way, this solution truly transforms the advantage of reduced communication traffic into end-to-end latency reduction and throughput improvement. Experimental results show that the end-to-end training speed of this solution can be improved by up to approximately 2.31 times compared to existing advanced software solutions, and the pure communication performance can reach more than 90% of the theoretical ideal value.

[0043] The present invention also provides an in-switch computing system for dynamic communication. The in-switch computing system for dynamic communication can be implemented by executing the process steps of the in-switch computing method for dynamic communication. That is, those skilled in the art can understand the in-switch computing method for dynamic communication as a preferred embodiment of the in-switch computing system for dynamic communication.

[0044] Example 2 Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1.

[0045] This embodiment provides a method for accelerating in-switch computing communication in dynamic communication scenarios such as Mixture-of-Experts (MoE). In particular, it relates to a method and its supporting software and hardware system architecture for accelerating sparse communication processes in multi-device environments through dynamic multi-target addressing, in-switch multicast / reduction, and token center kernel fusion. It belongs to the fields of artificial intelligence, computer architecture, and multi-device interconnection communication technology.

[0046] This embodiment aims to provide a dynamic, programmable native switch-based computing solution that can flexibly support dynamic and irregular communication patterns. The core technical issues that need to be addressed are: a. Distributed systems support a full-stack architecture for dynamic switching computation. How to achieve dynamic multi-target addressing with low overhead, i.e., without requiring the source device to explicitly maintain a static remote memory layout, to achieve dynamic multi-target routing, destination address allocation and subsequent merged access. Specifically, this includes packet format, instruction set architecture, hardware microarchitecture of accelerators and switches, and software runtime. b. Efficient fusion operator data flow scheduling: How to truly transform the traffic reduction brought about by multicast / reduction within the switch into end-to-end performance improvement, rather than remaining at the level of unidirectional traffic optimization. Specifically, this includes realizing kernel fusion across different communication directions, realizing the pipelined execution process, enabling complementary traffic modes to execute concurrently, balancing bidirectional bandwidth, and shortening the end-to-end execution time of the model.

[0047] The specific technical solution of this embodiment is as follows: This embodiment proposes a dynamic in-switch computation method for dynamic communication modes. This method consists of two core components: a "dynamic multi-target addressing mechanism" and a "token center kernel fusion mechanism," and is jointly implemented by the source device, the switch, the destination device, and the runtime. The overall approach is as follows: the source device uses a single multi-target address to represent the algebraic index of a token and uses a lightweight target expert list to describe the dynamic target set; data packets carrying the algebraic index and the dynamic target set are multicast or reduced within the switch, with the multicast or reduction determining the participating set through the dynamic targets; the destination device locally performs the conversion from the algebraic index to the memory layout index and address allocation; through token-level dependency tracking and persistent thread block scheduling, fine-grained pipelined execution of distribution communication, expert computation, and merging communication is achieved, enabling the concurrent execution of the two complementary communication modes of distribution and merging.

[0048] The method in this embodiment specifically includes the following steps: Step S1: The runtime first initializes the memory regions. Based on the expert parallel mapping relationship, two sets of multi-target (multimem) memory regions are established for distribution and merging, and both regions share the same "algebraic index-layout index" mapping relationship. The process of initializing memory regions using the extended runtime API is as follows: Figure 5 As shown.

[0049] In step S2, the gating network first generates a target expert list for each token. When performing distribution, the source device does not directly calculate the physical or virtual destination address on each destination device, but instead generates a multi-destination address for each token to be sent. The offset portion of this multi-destination address represents the algebraic index of the token in the algebraic tensor; simultaneously, the source device stores the target expert list of this token in shared memory or global memory in contiguous storage.

[0050] Step S3: The source device initiates data communication using extended instructions that support dynamic switching computation. Define the dymultimem.st and dymultimem.ld_reduce instructions, as follows: Figure 6 As shown. The former is used for dynamic multicast writing in the distribution phase, and the latter is used for dynamic reduction reading in the merging phase. In addition to carrying the data register and the multi-target address register (multimem register), the instruction also carries the target quantity register and the target list base address register.

[0051] Step S4: The data packet is transmitted in the Internet in an extended link-layer message format, such as... Figure 7 As shown, the data packet carries a 48-bit multitimem address, a 1-bit stage identifier, and a 15-bit target quantity in the first flit; subsequent target extension flits are used to encode the target expert identifier.

[0052] In step S5, the switch receives data packets and, based on the target expert identifier and the mapping relationship between experts and devices, copies or trims requests according to the output port. For the dymultimem.ld_reduce request, the switch also maintains a target quantity count and completes the internal reduction within the switch after collecting all relevant data.

[0053] Step S6: On the Hub side of the destination device, a hardware memory manager is set up to perform algebraic index to memory layout index mapping and multi-target multitimem address to local virtual address translation. When a distribution request arrives, if a certain algebraic block appears for the first time, the hardware memory manager allocates a new memory layout block for it and writes the mapping into the algebraic layout mapping table (AL table); when subsequent merge requests arrive, the mapping is directly reused to read data from the corresponding memory layout location. Figure 8 The layout transformation process is shown, where (a) is the token layout before routing. The dense token layout shown in (b) can be obtained through dense AllGather communication. Since expert routing selects a subset of experts for each token, the actual communication volume is less than that under the AllGather implementation. The algebraic layout obtained according to the expert routing result is actually a sparse layout (b), as shown in (c). This sparse layout needs to be further transformed into a dense memory layout. This layout transformation is accomplished by performing a mapping from algebraic index to memory layout index.

[0054] Step S7: To improve address translation efficiency, an Algebraic Layout Mapping Cache Table (AL-TLB) is further set up in the destination device. The AL-TLB caches AL table entries with "expert identifier + algebraic block index" as tags; when a hit occurs, the memory layout block index is obtained directly; when a miss occurs, the AL table is accessed and filled back. Figure 9This paper illustrates a possible overall hardware architecture for supporting dynamic multi-target addressing proposed in this invention, with an example using a GPU as an accelerator. The blue section represents the main system architecture proposed in this invention, including runtime support on the host side, support for computation instructions within the extended switch on the source GPU, computation logic and routing extension within the network switch, and enhanced memory management and scheduling table management on the hub side of the destination GPU.

[0055] Step S8: For the weighted summation requirement in the merged communication stage, the present invention preferably does not implement weighted reduction within the switch. Instead, the expert output is first multiplied by the gate weight in the epilogue of the preceding matrix multiplication, and then the unweighted reduction within the switch is completed using dymultimem.ld_reduce, thereby reducing the complexity of the switch.

[0056] Step S9: To address the bandwidth imbalance caused by the isolated execution of distribution and merging communication, a token-centric kernel fusion mechanism is introduced. This mechanism treats the MoE layer as a token-driven pipeline consisting of the main computational operations Dispatch—GEMM-1—GEMM-2—Combine, rather than four independent operators, and implements fine-grained pipeline through a token tracker and a token-centric scheduler. Figure 10 (a) shows the four main stages that make up the pipeline; (c) and (d) show the computing and communication pipelines implemented by existing international advanced software solutions; (e), (f), (g), and (h) show the computing and communication pipelines of the proposed solution under different combinations of technologies, wherein (h) is the computing and communication pipeline implemented by the proposed solution using all technologies.

[0057] Step S10, the token tracker detects fine-grained ready boundaries. Preferably, the token tracker includes, for example, Figure 11 The three tables shown are: a block status table (TS table) for tracking tile-level write arrivals and GEMM row completion status; a token sequence number table (TID table) for recording the token identifiers contained in each tile; and an output ready table (OR table) for accumulating the completion count of the top k expert outputs for each token on the source device side. When the TS table detects that tsize tokens have been distributed and written on a certain expert, it immediately triggers the corresponding expert to calculate the GEMM-1 thread block row and enter the schedulable state; when all corresponding GEMM-1 lines are completed, the GEMM-2 thread block becomes schedulable; when GEMM-2 calculation is completed and the source device is notified through the TID table, the source device updates the nReady count of the token in the OR table, and can initiate dymultimem.ld_reduce when the topk count is reached.

[0058] Step S11: At the scheduler level, a persistent kernel software implementation is preferred. The original thread blocks are abstracted as tasks, and the persistent thread blocks continuously retrieve ready tasks from the task list for execution; simultaneously... Figure 10 As shown in (b), all parallel computing cores are logically divided into four groups: Dispatch, GEMM-1, GEMM-2, and Combine. Alternatively, GEMM-1 and GEMM-2 can share computing resources when no tasks are ready. All state updates are preferably performed after the written data is visible to all parallel computing cores, i.e., the TS table, AL table, or OR table is updated only after a store acknowledgement is detected, to ensure the correctness of subsequent reads and reductions.

[0059] Step S12, further, the present invention can also be implemented as a system or device, including: a source device load / storage unit with dynamic target list capture capability; switch logic with dynamic target awareness forwarding and dynamic reduction capability; a destination device hub with AL table, AL-TLB and address allocation counter; and a runtime interface for creating multimem objects, binding communication-related memory regions and initiating execution.

[0060] The technical solution proposed in this embodiment brings significant performance gains to large model training and inference, including: This embodiment fundamentally reduces redundant transmissions in dynamic communication. In the MoE model, for distribution communication, the same token only needs to be sent from the source device to the switch once, and then the switch completes the multicast; for merging communication, the switch completes the reduction of the multi-path partial results and returns the final result only once to the destination device.

[0061] This embodiment natively supports dynamic communication with changes in the target set and address asymmetry, eliminating the need to degrade dynamic communication to static set communication simulation and avoiding invalid traffic.

[0062] This embodiment avoids the source device explicitly maintaining the remote destination address, counter, or complex memory layout state, reducing software synchronization overhead and additional computing resource consumption.

[0063] This embodiment transforms the original one-way optimized traffic advantage into a two-way bandwidth balanced end-to-end latency advantage through the fine-grained pipeline mechanism of token center kernel fusion, thereby improving the overall bandwidth utilization of the system.

[0064] The solution in this embodiment has low intrusion into existing hardware and software stacks. In the preferred embodiment, the switch expansion mainly manifests as target-aware control logic and reduction logic; the source device expands its support for new instructions based on the existing internal computing scheme of the switch; the destination device adds a lightweight hub management structure; at runtime, only the creation and address binding interfaces for dynamic multi-destination addressing objects need to be expanded. The hardware area overhead is small, making it feasible for engineering implementation.

[0065] Based on the experimental results in the paper, the proposed scheme in this embodiment shows significant performance advantages compared with existing representative schemes: the end-to-end training speed can be improved by up to about 1.79 times compared with internationally advanced software computing and communication overlap optimization schemes (such as Comet), and by up to about 2.31 times compared with internationally advanced communication library implementations (such as DeepEP); dynamic multi-target addressing can reduce traffic by nearly 50% compared with the original scheme, and the pure communication performance can reach more than 90% of the theoretical ideal value. Figure 12 The end-to-end training speedup of the proposed scheme (DySHARP) compared to other internationally advanced optimization schemes is shown under different model configurations; Figure 13 The end-to-end inference speedup of the proposed solution (DySHARP) compared to other internationally advanced optimization solutions is shown under different model configurations; Figure 14 The results show the speedup of the proposed solution (DySHARP) for a single MoE layer compared to other internationally advanced optimization solutions under different model configurations, excluding cross-layer optimizations such as DualPipe; Figure 15 The diagram shows a comparison of communication traffic between the proposed solution (DySHARP) and advanced software solutions or existing in-switch computing solutions.

[0066] The aforementioned effects are directly brought about by the technical solution of this embodiment, especially by the core technologies such as dynamic multi-target addressing, dynamic multicast / reduction within the switch, destination memory layout mapping management, and token center pipelined scheduling.

[0067] The core idea of ​​this embodiment lies in the support for in-switch computation under dynamic communication mode. The target identifier in the target list can be a direct expert identifier, a combination of "device identifier + device internal local number," or an index in other encoding formats. The target list can be stored in shared memory, global memory, or a dedicated buffer; the source device's memory access unit can retrieve the list through one or more memory accesses. The algebraic layout mapping table can be located in accelerator DRAM, on-chip SRAM, or adopt a hierarchical structure of on-chip cache and off-chip main table; its cache organization can be fully associative, set-associative, or multi-level cache. The various state tables in the token tracker can be located in accelerator DRAM, on-chip SRAM, or adopt a hierarchical structure of on-chip cache and off-chip main table; its cache organization can be fully associative, set-associative, or multi-level cache; the entire token tracker can be fully implemented in hardware or implemented using a hybrid approach of "hardware state registers combined with software polling." The scheduler can be implemented using a persistent software kernel, a dedicated hardware thread block scheduler, or a firmware scheduler. Intra-switch reduction or broadcast is unweighted reduction or broadcast. In implementations with stronger intra-switch computing capabilities, weighted reduction or broadcast, other associative functions, or more generalized multi-purpose computing operations can also be supported.

[0068] This embodiment provides an addressing mechanism that uses a single multi-target (multitimem) address combined with a dynamic target set list to represent multi-target dynamic operations.

[0069] This embodiment provides a command extension and message extension method for communication within a dynamic switch, including dymultimem.st, dymultimem.ld_reduce, and a link layer message format carrying multiple destination addresses, stage identifiers, number of destinations, and a destination list.

[0070] This embodiment provides a forwarding and reduction mechanism within a switch for dynamic multi-target requests, including output port calculation based on the target list, copying / pruning, and dynamic reduction completion detection.

[0071] This embodiment provides a hardware memory management mechanism that performs algebraic index to memory layout index mapping and multi-target address to virtual address translation on the target device hub side, including an AL table (AL-TLB) and an allocation counter.

[0072] This embodiment provides a token center kernel fusion mechanism, including a token tracker composed of TS / TID / OR tables, and a persistent kernel scheduling method based on ready boundaries.

[0073] This embodiment provides a method for executing Dispatch, GEMM-1, GEMM-2, and Combine at the token / tile granularity in the MoE model, which is used to achieve concurrent complementary communication modes and transform the advantages of link traffic into end-to-end performance improvement.

[0074] Example 3 Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1.

[0075] This embodiment uses a multi-GPU system interconnected with NVLink / NVSwitch as an example, where multiple experts are distributed across different GPUs in a parallel manner. Each token is used by a gating network to select the top k target experts. The following description assumes a typical case where top k is greater than or equal to 2.

[0076] (1) Runtime initialization phase: Create dynamic multimem objects, bind the multimem region and virtual address region to Dispatch and Combine respectively; set the number of tokens that each expert can receive, nactive[expert]; establish the mapping relationship between expert identifiers and GPU indexes; reserve storage space for AL tables, etc.

[0077] (2) Dispatch preparation phase: The source GPU constructs a target expert list for each token based on the output of the gating network; writes the target list into shared memory or global memory; generates a multitimem address for the token, where the address offset is equal to the algebraic index of the token in the algebraic tensor.

[0078] (3) Dispatch execution phase: The source GPU executes the dymultimem.st instruction. The Memory Access Unit (LSU) first reads the target quantity and the base address of the target list, and then retrieves the target list into the buffer structure associated with MultimemQ. After the target list is in place, the LSU generates a request message containing the multimem address, phase number, target quantity and target list and sends it to the switch.

[0079] (4) Switch processing stage: The switch calculates the corresponding output port based on the target expert list, copies or trims the request and sends it to the corresponding target GPU; if it is a Combine request, the switch simultaneously establishes a reduction context and records the number of targets that have not yet arrived.

[0080] (5) Target GPU address allocation stage: When a Dispatch request accesses a certain generation block for the first time, the hardware memory manager in the Hub checks the valid bit of the corresponding entry in the AL table; if invalid, a new layout block LIdx is allocated for it through the allocation counter and LIdx is written into the AL table; then the local virtual address is calculated according to VAddr=VBase+LIdx×bsize+intra-block-offset.

[0081] (6) Expert calculation phase: After the data is written to the local layout tensor, the token tracker determines whether tsize tokens of a tile have arrived based on the DAcc count in the TS table; if they have arrived, the corresponding GEMM-1 row is triggered to enter the ready state. After GEMM-1 is completed, TBCnt1 is updated, and GEMM-2 is triggered after all corresponding rows are completed; GEMM-2 is preferentially multiplied by the gating weight in the epilogue first.

[0082] (7) Completion Notification Phase: When the corresponding row of GEMM-2 is completed, the token tracker finds the identifiers of all tokens in the tile through the TID table and sends a completion notification to the source GPU. The source GPU accumulates the nReady count in the OR table; when the nReady of a token reaches topk, it means that all its expert outputs are ready and can be used for merging.

[0083] (8) Combine execution phase: The source GPU executes dymultimem.ld_reduce for tokens that meet the conditions. The destination GPU remaps the algebraic index to the layout index based on the previously established AL table and then reads the expert output; the switch performs reduction after receiving all the required data and returns the final result to the source GPU only once.

[0084] (9) Scheduling Implementation: The persistent thread block continuously polls the ready state in the token tracker and initiates the corresponding task only when both resources and dependencies are satisfied. As a result, Dispatch and Combin can occupy the link in different directions at the same time, so that the bandwidth in both the GPU→switch and switch→GPU directions can be utilized more evenly, and the bandwidth advantage brought by the computation within the switch can be converted into an end-to-end latency advantage.

[0085] (10) Consistency guarantee: To prevent subsequent reads from being triggered before the data is visible, updates to the states of TS table, OR table, AL table, etc. are preferably performed only after a write confirmation signal is detected.

[0086] (11) Link layer message format: In the preferred implementation, the first flit of the link layer message contains a 48-bit multitimem address, a 1-bit stage identifier, and a 15-bit target number; the target expert identifier is encoded in 16 bits, and each target extension flit can accommodate multiple target identifiers.

[0087] (12) Mapping table implementation: In the preferred implementation, the AL table entries can adopt the form of "1 valid bit + 31-bit layout index".

[0088] The above bit width is a preferred implementation parameter, and those skilled in the art can also adjust it according to the address space size, number of experts and chip implementation constraints.

[0089] This invention is applicable to both the training and inference phases. During training, longer sequences and larger top-k values ​​lead to a higher communication ratio; this invention can significantly reduce the overhead of the MoE layer. During inference, although decoding is more memory-intensive, due to latency sensitivity, this invention can still yield significant benefits by reducing software control and synchronization boundaries.

[0090] This invention provides a method and system for in-switch computation oriented towards dynamic communication, belonging to the field of computer interconnection and communication technology. To address the problems of redundant data transmission and performance bottlenecks in existing technologies when handling dynamic and irregular communication, this invention proposes a hardware-software collaborative scheme. The method includes: a source device generating a communication request based on a logical data identifier associated with a data unit and a dynamic target set list; a network switch parsing the target set list and performing dynamic multicast or dynamic reduction operations on the data unit; and a destination device dynamically mapping the data to a local memory address based on the logical data identifier and completing data access. Furthermore, a pipeline scheduling mechanism tracks the fine-grained state of data units during the distribution, computation, and merging stages to achieve concurrent execution of the merging stage of one data unit and the distribution stage of another.

[0091] This invention can fundamentally reduce redundant communication, natively support dynamic communication, and improve end-to-end system performance by balancing the utilization of bidirectional link bandwidth.

[0092] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0093] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A calculation method within a switch for dynamic communication, characterized in that, Applied to a distributed system including a source device, a network switch, and at least one destination device, the following steps are included: The source device generates a communication request based on a logical data identifier associated with a data unit and a target set list describing a dynamic target set of the data unit. The network switch parses the target set list contained in the communication request to perform a dynamic multicast operation or a dynamic reduction operation on the data unit, wherein the dynamic multicast operation constitutes a distribution phase, and the dynamic reduction operation constitutes a calculation phase and a merging phase. The destination device dynamically maps the logical data identifier in the communication request to its local memory address, and completes data access; and A pipeline scheduling mechanism is used to track the fine-grained state of the data unit in the distribution phase, the computation phase, and the merging phase, so as to achieve concurrent execution of the merging phase of one data unit and the distribution phase of another data unit.

2. The in-switch computation method for dynamic communication according to claim 1, characterized in that, The step of dynamically mapping logical data identifiers to local memory addresses by the destination device is performed by a hardware memory manager within the destination device, which implements the mapping by maintaining a generational layout mapping table.

3. The in-switch computation method for dynamic communication according to claim 2, characterized in that, The hardware memory manager also includes a cache module for caching the mapping relationship.

4. The in-switch computation method for dynamic communication according to claim 1, characterized in that, The tracking is achieved through a token tracker, which is used to track the fine-grained state of the data unit.

5. The in-switch computation method for dynamic communication according to claim 4, characterized in that, The token tracker includes: A status table is used to track the arrival and computation status of data on the destination device side; and An output ready table is used to track the ready status of multiple computation results on the source device side.

6. The in-switch computation method for dynamic communication according to claim 4 or 5, characterized in that, The pipeline scheduling mechanism also includes a scheduler implemented with a persistent kernel, which polls the status of the token tracker to execute ready tasks.

7. The in-switch computation method for dynamic communication according to claim 1, characterized in that, The step of generating a communication request includes: the source device executing a predefined dynamic multicast write instruction or a dynamic reduction read instruction.

8. The in-switch computation method for dynamic communication according to claim 1, characterized in that, The communication request also includes weight information corresponding to the destination device, and before the network switch performs the dynamic reduction operation, the data units distributed to the destination device via the dynamic multicast operation are multiplied by the weight information corresponding to the destination device, so that the network switch performs an unweighted reduction operation.

9. A computing system within a switch for dynamic communication, characterized in that, The system includes a source device, a network switch, and at least one destination device. The source device is configured to generate a communication request based on a logical data identifier associated with a data unit and a target set list describing a dynamic target set of the data unit. The network switch is configured to parse the target set list contained in the communication request to perform a dynamic multicast operation or a dynamic reduction operation on the data unit, wherein the dynamic multicast operation constitutes a distribution phase and the dynamic reduction operation constitutes a calculation phase and a merging phase. The destination device is configured to dynamically map a logical data identifier to a local memory address of the destination device based on the logical data identifier in the communication request; and A pipeline scheduling module is configured to track the fine-grained state of the data unit in the distribution phase, the computation phase, and the merging phase, so as to enable concurrent execution of the merging phase of one data unit and the distribution phase of another data unit.

10. The in-switch computing system for dynamic communication according to claim 9, characterized in that, The destination device includes a hardware memory manager configured to implement the mapping by maintaining a generational layout mapping table; And / or, the hardware memory manager further includes a cache module for caching the mapping relationship; And / or, the tracking is implemented through a token tracker for tracking the fine-grained state of the data unit; The token tracker includes: A status table is used to track the arrival and computation status of data on the destination device side; and An output ready list is used to track the ready status of multiple calculation results on the source device side; And / or, the pipeline scheduling mechanism further includes a scheduler implemented with a persistent kernel, the scheduler polling the status of the token tracker to execute ready tasks; And / or, the execution process of generating the communication request is as follows: the source device executes a predefined dynamic multicast write instruction or dynamic revert read instruction; And / or, the communication request further includes weight information corresponding to the destination device, and before the network switch performs the dynamic reduction operation, the data units distributed to the destination device via the dynamic multicast operation are multiplied by the weight information corresponding to the destination device, so that the network switch performs an unweighted reduction operation.

Images