Dynamic multicast communication method, system and switch based on in-network computing

Abstract

This application relates to a dynamic multicast communication method, system, and switch based on on-network computing. The method includes: receiving a multicast request message sent by a source computing unit and parsing the multicast request message; copying the multicast request message into the same number of message copies based on the number of destination computing unit identifiers in the dynamic multicast information; for each message copy, determining the destination computing unit identifier and destination expert identifier corresponding to the message copy based on the dynamic multicast information, and determining the destination memory address corresponding to the message copy based on the destination expert identifier and a pre-configured address dense list table; associating the destination memory address with the message copy, and forwarding the message copy to the destination computing unit corresponding to the destination computing unit identifier. This method can reduce the network transmission traffic of the source computing unit and the computing resources used for performing multicast communication operations.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a dynamic multicast communication method, system and switch based on on-network computing. Background Technology

[0002] With the rapid growth in the scale of large-scale language and vision models, the Mixture of Experts (MoE) architecture has been widely adopted. In this architecture, the model's feedforward network layer is split into multiple parallel expert networks, and each input token only needs to be computed by at least a few expert networks. In practical deployments, due to the enormous number of expert network parameters, a parallel expert strategy is commonly adopted, distributing different expert networks across multiple computing units.

[0003] In this context, after expert routing is performed, the token data on each source computing unit needs to be distributed to target experts located on other destination computing units for processing. After processing, the computation results are then aggregated back to the source computing unit. This "distribution" and "aggregation" communication triggers Alltoallv (non-uniform full exchange) type network traffic, meaning that each source computing unit may send data to all other destination computing units, and the amount of data sent to each destination computing unit is unequal. This traffic becomes the main communication bottleneck in the training and inference process of the MoE model.

[0004] Currently, the industry standard for expert-parallel Alltoallv communication typically involves the source computing unit preparing a separate data copy for each target expert, filling in the destination memory address on each destination computing unit, and initiating network transmission separately. This approach results in an excessive computational load on the source computing unit, consuming processing resources intended for model calculations. Furthermore, the need to generate multiple copies of each token data and transmit them separately leads to significant network traffic and increased communication latency. Summary of the Invention

[0005] Therefore, it is necessary to address the technical problems of high network traffic and high computing resource consumption caused by the dynamic distribution of token data from the source computing unit to multiple destination computing units when the MoE model adopts an expert parallel strategy in the existing technology. To provide a dynamic multicast communication method, system and switch based on on-network computing that can reduce the network transmission traffic of the source computing unit and the computing resources for performing multicast communication operations, it is necessary to provide a dynamic multicast communication method, system and switch based on on-network computing that can reduce the computing resources of the source computing unit for performing multicast communication operations.

[0006] In a first aspect, this application provides a dynamic multicast communication method based on on-network computing, executed by a switch communicatively connected to multiple computing units, the method comprising:

[0007] The system receives a multicast request message sent by a source computing unit and parses the multicast request message, which carries payload data and dynamic multicast information. The source computing unit is the computing unit among the plurality of computing units that outputs the multicast request message to the switch.

[0008] Based on the number of destination calculation unit identifiers in the dynamic multicast information, the multicast request message is copied into the same number of message copies;

[0009] For each message copy, the destination computing unit identifier and destination expert identifier corresponding to the message copy are determined according to the dynamic multicast information, and the destination memory address corresponding to the message copy is determined according to the destination expert identifier and the pre-configured address dense list table; wherein, the address dense list table is used to store the base address and expert identifier of the memory block allocated to the expert network deployed on each computing unit, and the destination memory address is the base address of the memory block corresponding to the destination expert identifier in the address dense list table;

[0010] The destination memory address is associated with the message copy, and the message copy is forwarded to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

[0011] In one embodiment, the pre-configuration steps of the address dense array table are as follows:

[0012] Obtain the base address of the memory block allocated on each computing unit to each expert network deployed thereon, which corresponds to each switch port of the switch, and obtain the expert identifier of all expert networks.

[0013] For each switch port, the expert identifier of its corresponding expert network and the base address of its memory block are associated and stored in the address dense table of the switch port.

[0014] In one embodiment, the dynamic multicast information includes a set of destination computing units and a set of destination experts corresponding to the set of destination computing units. The set of destination computing units includes multiple destination computing unit identifiers, and the set of destination experts includes destination expert identifiers of the destination expert network deployed on the destination computing unit corresponding to each destination computing unit identifier.

[0015] After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes:

[0016] For each message replica, a multicast index is generated for that message replica.

[0017] For each message copy, determining the destination computing unit identifier and destination expert identifier corresponding to the message copy based on the dynamic multicast information includes:

[0018] For each message replica, the destination computing unit identifier in the destination computing unit set that corresponds to the multicast index of the message replica is used as the destination computing unit identifier corresponding to the message replica.

[0019] The destination expert identifier in the destination expert set that corresponds to the multicast index is used as the destination expert identifier corresponding to the message copy.

[0020] In one embodiment, the multicast request message further carries a source identifier field, a destination identifier field, and a destination memory address field. The initial value of the source identifier field is the source computing unit identifier of the source computing unit, and the initial values ​​of the destination identifier field and the destination memory address field are empty. After parsing the multicast request message, the method further includes:

[0021] Replace the value of the source identifier field with the switch port identifier of the switch;

[0022] After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes:

[0023] The value of the target identifier field is set to the target computing unit identifier, and the value of the target memory address field is set to the target memory address.

[0024] In one embodiment, after the destination computing unit writes the payload data from the message copy into the memory block indicated by the destination memory address, the method further includes:

[0025] Receive a response message returned by the destination computing unit, the response message carrying a source identifier field, a target identifier field, and a target memory address field;

[0026] The value of the source identifier field is modified to the destination computing unit identifier, the value of the target identifier field is modified to the source computing unit identifier, and the value of the target memory address field is modified to the copy write-back address corresponding to the message copy;

[0027] The response message is sent to the source computing unit based on the target identifier field.

[0028] In one embodiment, the multicast request message further carries a message sequence number, which is initially bound to the original message sequence number; after parsing the multicast request message, the method further includes:

[0029] Assign an internal sequence number to the multicast request message and establish a mapping relationship between the original message sequence number and the internal sequence number;

[0030] Replace the original message sequence number bound to the message sequence number with the internal sequence number;

[0031] After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes:

[0032] For each message copy, an outgoing sequence number is assigned to the message copy, and a mapping relationship is established between the internal sequence number and the outgoing sequence number;

[0033] Replace the internal sequence number bound to the message sequence number with the outgoing sequence number.

[0034] In one embodiment, after the destination computing unit writes the payload data from the message copy into the memory block indicated by the destination memory address, the method further includes:

[0035] Receive a response message returned by the destination computing unit, the response message carrying a message sequence number;

[0036] Based on the mapping relationship between the internal sequence number and the outgoing sequence number, the message sequence number is rebound to the internal sequence number;

[0037] Based on the mapping relationship between the original message sequence number and the internal sequence number, the message sequence number is rebound to the original message sequence number;

[0038] The system obtains the number of times the message sequence number is rebound to the internal sequence number or the original message sequence number. If the number of times is equal to the number of destination calculation unit identifiers in the dynamic multicast information, the system sends all response messages to the source calculation unit according to the target identifier field.

[0039] In one embodiment, the dynamic multicast information further includes a write-back address list, which stores the write-back addresses of the replica memory blocks pre-allocated on the source computing unit for all message replicas; after copying the multicast request message into the same number of message replicas according to the number of destination computing unit identifiers in the dynamic multicast information, the method further includes:

[0040] For each message copy, obtain the copy write-back address corresponding to the message copy from the write-back address list;

[0041] The copy write-back address is written into the message copy so that after the message copy is forwarded to the corresponding destination computing unit, the destination computing unit writes the expert calculation result corresponding to the message copy back to the copy memory block indicated by the copy write-back address.

[0042] In one embodiment, before receiving the multicast request message sent by the source computing unit, the method further includes:

[0043] Receive a connection establishment request sent by the source computing unit, the connection establishment request carrying the source computing unit identifier and the switch port identifier of the source computing unit;

[0044] Based on the switch port corresponding to the switch port identifier, a connection establishment response is sent to the source computing unit according to the source computing unit identifier to establish a connection between the switch port and the source computing unit.

[0045] Secondly, this application also provides a dynamic multicast communication system based on on-network computing, including multiple computing units and a switch communicatively connected to the multiple computing units; the multiple computing units include a source computing unit and a destination computing unit corresponding to the source computing unit; wherein, the source computing unit is the computing unit among the multiple computing units that outputs a multicast request message to the switch, and the destination computing unit is the computing unit among the multiple computing units that is indicated by the multicast request message;

[0046] The source computing unit is used to generate and send the multicast request message to the switch. The multicast request message carries payload data and dynamic multicast information.

[0047] The switch is used to perform the following operations:

[0048] Receive the multicast request message sent by the source computing unit and parse the multicast request message;

[0049] Based on the number of destination calculation unit identifiers in the dynamic multicast information, the multicast request message is copied into the same number of message copies;

[0050] For each message copy, the destination computing unit identifier and destination expert identifier corresponding to the message copy are determined according to the dynamic multicast information, and the destination memory address corresponding to the message copy is determined according to the destination expert identifier and the pre-configured address dense list table; wherein, the address dense list table is used to store the base address and expert identifier of the memory block allocated to the expert network deployed on each computing unit, and the destination memory address is the base address of the memory block corresponding to the destination expert identifier in the address dense list table;

[0051] The destination memory address is associated with the message copy, and the message copy is forwarded to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

[0052] In one embodiment, the switch includes a switch port and a centralized replication engine communicatively connected to the switch port. The switch port is divided into an inbound processing module communicatively connected to the source computing unit and an outbound processing module communicatively connected to the destination computing unit.

[0053] The incoming processing module is used to receive multicast request messages sent by the source computing unit and parse the multicast request messages.

[0054] The centralized replication engine is used to replicate the multicast request message into the same number of message copies based on the number of destination computing unit identifiers in the dynamic multicast information.

[0055] The centralized replication engine is also used to determine the destination computing unit identifier and destination expert identifier corresponding to each message copy based on the dynamic multicast information, and to determine the destination memory address corresponding to the message copy based on the destination expert identifier and the address dense array table, and then associate the destination memory address with the message copy.

[0056] The outgoing processing module is used to forward each message copy to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

[0057] In one embodiment, the system further includes a manager;

[0058] The manager is used to obtain the base address and corresponding expert identifier of the memory block allocated by each computing unit to the expert network it deploys and corresponding to each switch port of the switch, and for each switch port, associates the expert identifier and the base address of the memory block of the corresponding expert network and sends them to the switch port.

[0059] The switch port is used to associate and store the expert identifier of the received expert network with the base address of the memory block in the address dense table of the switch port.

[0060] Thirdly, this application also provides a switch for performing the method as described in any of the preceding claims.

[0061] The aforementioned dynamic multicast communication method, system, and switch based on on-network computing, in its implementation, firstly, receives multicast request messages sent by the source computing unit through the switch and parses the multicast request messages; then, based on the number of destination computing unit identifiers in the dynamic multicast information, the multicast request message is copied into the same number of message copies; next, for each message copy, the destination computing unit identifier and destination expert identifier corresponding to the message copy are determined according to the dynamic multicast information, and the destination memory address corresponding to the message copy is determined according to the destination expert identifier and a pre-configured address dense array table; finally, the destination memory address is associated with the message copy, and the message copy is forwarded to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address. Based on this invention, the source computing unit only needs to generate and send an original message carrying dynamic multicast information. The message copying and address arrangement are offloaded to the switch. The switch generates a destination memory address independently for each message copy by querying a pre-configured address dense table, and writes the payload data in the message copy directly to the correct location of the corresponding expert memory block on the destination computing unit. This significantly reduces the network transmission traffic of the source computing unit and the computing resources used to perform multicast communication operations, thereby improving the overall training and inference efficiency of the MoE model. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a schematic diagram of the structure of dense models and MoE models in the prior art;

[0064] Figure 2 A diagram illustrating two strategies used when training or inferring MoE models in the prior art;

[0065] Figure 3 This diagram illustrates the Alltoallv traffic generated when the MoE model in the prior art employs an expert parallel strategy.

[0066] Figure 4 This diagram illustrates the distribution of operational overhead during the Dispatch phase in expert-parallel Alltoallv communication in existing technologies.

[0067] Figure 5 This is an application environment diagram of a dynamic multicast communication method based on on-network computing in one embodiment;

[0068] Figure 6 This is a flowchart illustrating a dynamic multicast communication method based on on-network computing in one embodiment;

[0069] Figure 7 A schematic diagram illustrating the entire process of expert parallel communication as an example;

[0070] Figure 8 This is a schematic diagram illustrating the physical connection between a computing unit and eight switches, serving as an example.

[0071] Figure 9 This is a schematic diagram illustrating memory block allocation for a computing unit and address dense table mapping for switch ports, serving as an example.

[0072] Figure 10 A schematic diagram illustrating the allocation of replica memory blocks for an example computing unit;

[0073] Figure 11 A schematic diagram illustrating the connection between the switch and the computing unit;

[0074] Figure 12 This is a schematic diagram illustrating the message structure of a connection request and a connection response, serving as an example.

[0075] Figure 13 A diagram illustrating proxy and deproxy operations, as well as retag and unretag operations, performed on a switch;

[0076] Figure 14 This is a timing diagram for dynamic multicast communication and multicast protocol communication in one embodiment;

[0077] Figure 15 This is a flowchart of dynamic multicast communication and multicast protocol communication in one embodiment.

[0078] Figure 16 for Figure 15 A schematic diagram of the message structure sent at each stage;

[0079] Figure 17 Another flowchart illustrating dynamic multicast communication and multicast protocol communication is provided as an example.

[0080] Figure 18 This is a schematic diagram of another message structure as an example;

[0081] Figure 19 This is a schematic diagram illustrating the benefits of dynamic multicast communication based on on-network computing in an AF-coupled scenario.

[0082] Figure 20 This is a schematic diagram illustrating the benefits of dynamic multicast communication based on on-network computing in an AF separation scenario.

[0083] Figure 21 This is a schematic diagram of the switch structure in one embodiment. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0085] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0086] To facilitate understanding of the technical solution of this application, the following content is now described:

[0087] I. MoE Model

[0088] The vast majority of existing large models (language and vision types) are built based on the Transformer module. For example... Figure 1 As shown in the middle left figure, a Transformer module (or layer) mainly consists of an attention module and a linear feed-forward network (FFN) module, with a normalization module following each module. A large model consists of several Transformer modules.

[0089] In a standard dense model (dense means that each layer of the FFN module is fully connected), when the model becomes large (e.g., 100B), the FFN module becomes very large, and its storage and computation account for the vast majority (e.g., 95%) of the entire model, which consumes a lot of storage and computing power.

[0090] In the MoE model (such as...) Figure 1 As shown in the middle right figure, the FFN module is broken down into E (e.g., 256) small FFNs, which are small linear feedforward networks, also known as expert networks or experts. Before each token data enters the FFN module for computation, it is processed by a gating network (which uses a small linear network) to select K (e.g., 8) experts for computation. After computation, the results of multiple experts are combined (averaged). The process of selecting experts through gating network computation is usually called expert routing (analogous to the routing of data packets in a network). Since K is much smaller than E, even if the model is very large, the actual computational load of the linear network part is much smaller than that of the large network in a dense model (e.g., one-tenth). However, in the MoE model, the parameters of the linear network part still account for the vast majority of the entire model (e.g., over 98% in DeepSeek V3).

[0091] Therefore, in the MoE model, each token data only selects a small number of small FFNs for computation, and the routing of token data is achieved through a gating network. The gating network actually implements a specialized selection function; the routing computation of the gating network can be understood as routing computation based on the token data type, and the small FFNs can be understood as expert networks capable of handling specific types of token data. This is the origin of the name "Hybrid Expert Model." For example, the gating network can be understood as routing animal-related token data to FFN1, plant-related token data to FFN2, and so on. That is, FFN1 can be understood as an expert specializing in animal-related token data, FFN2 as an expert specializing in plant-related token data, and so on.

[0092] II. Expert Parallel Strategy

[0093] As mentioned above, the expert parameters in the MoE model account for a high proportion. During training or inference, if each data parallelism (DP) plane, which includes at least one data unit (the computing unit is the hardware that performs AI (Artificial Intelligence) computation and ensemble communication; for simplicity in the accompanying drawings and subsequent description, it is referred to as XPU in this document, where XPU stands for Accelerated Processing Unit), uses an independent set of expert parameters (such as...) Figure 2As shown in the middle left figure, this would occupy too much space and require too many computing units. Therefore, in the training or inference scenarios of MoE models, the industry's common practice is that each DP uses an independent non-expert part, while multiple DPs share a set of expert parts (such as...). Figure 2 As shown in the middle right figure, this approach of distributing a set of experts across multiple DPs is called the expert parallel strategy.

[0094] The advantages of using an expert parallel strategy are as follows:

[0095] (1) There is only one set of expert data, with no other copies. The parameters of the experts do not need to be synchronized.

[0096] (2) The shared experts are distributed across multiple DPs, rather than having a set on each DP, which significantly reduces memory usage and thus reduces the number of XPUs required.

[0097] In a parallel expert strategy, since a set of experts is distributed across multiple (or all) Data Processors (DPs), the token data of one DP, after route calculation, may need to be sent to multiple XPUs of other DPs for expert computation. After the expert computation is completed, the token data needs to be sent back to the source DP for subsequent computations (such as normalization, attention calculation for the next layer, etc.). To address this, the communication of the expert part can be divided into two segments: the stage from the source DP to the expert on the destination DP is called the Dispatch stage, and the stage from the expert on the destination DP completing the computation and sending the token data back to the source DP is called the Combine stage.

[0098] III. Alltoallv Traffic

[0099] As mentioned above, the expert parallel strategy has significant advantages, but it also introduces some new technical problems. Specifically, such as Figure 3 As shown, because a DP's token data may be sent to any expert, and most experts reside on other DPs, the expert parallel strategy introduces cross-DP traffic. In detail, a token data is sent to K experts, who are distributed across only a few XPUs or DPs. However, a large batch of token data will macroscopically (statistically) trigger traffic to all DPs, i.e., an all-to-all communication. Therefore, the communication in the Dispatch and Combine phases is of the all-to-all type. Since the traffic from each source XPU to other DPs may be different, the traffic in the Dispatch and Combine phases of expert parallelism is called Alltoallv (where v stands for variable) traffic.

[0100] The traffic of two Alltoallv operations (one Dispatch and one Combine) when using an expert parallel strategy is as follows: Figure 3 As shown, the specific description is as follows:

[0101] (1) Dispatch traffic: Each DP has a batch of independent token data (without overlap). Figure 3 In the diagram, the number below the token data block represents the expert sequence number calculated by the gating network (assuming each token data is sent to two experts). For example, token data 'a' from DP1 is sent to expert 2 (located on XPU2) and expert 4 (located on XPU4). Each DP's XPU calculates the expert sequence number to which all token data in the current batch should be routed at once, and then sends each token data to the corresponding expert. This actually creates a traffic inflation effect, where each token data generates K replicas sent to K different experts (e.g., K=8 in DeepSeek V3). From the token data granularity perspective, this is a network multicast.

[0102] (2) Combine Traffic: After the expert network completes its calculations, the expert sends a copy of the token data back to the source DP (XPU). At the source XPU, each token data receives K copies. These copies need to be reduced, i.e., averaged. For example, if two copies of token a are received at XPU1, then the two copies are averaged (for simplicity, see attached diagram). Figure 3 The text only shows the sum, not the division by 2.

[0103] As shown above, the characteristics of Alltoallv traffic are that the amount of traffic sent from each source computing unit to different target computing units is inconsistent, and from the perspective of token data granularity, each token data needs to be multicast to multiple specific target experts. This dynamic multicast requirement cannot be efficiently supported by traditional aggregated communication modes. This makes Alltoallv traffic the main communication bottleneck in the training and inference process of MoE models.

[0104] Currently, the industry standard for Alltoallv communication in expert parallel strategies typically involves the source computing unit preparing a separate data copy for each target expert, filling in the destination memory address on each destination computing unit, and then initiating network transmission separately. As an example, Figure 4This paper presents a schematic diagram illustrating the overhead distribution of the Dispatch phase in Alltoallv communication using an expert parallel strategy in existing technologies. Before the Dispatch phase, the source computing unit needs to copy the data to be sent from its original location according to the differences between the destination computing unit and the experts, and then regroup and organize it into a continuous transmission buffer for each destination computing unit in memory. This is equivalent to data pre-sorting by the source computing unit. After the Dispatch phase, the destination computing unit needs to copy the data scattered in the memory blocks of each expert and reorganize it into a continuous format suitable for computation. This is also equivalent to data rearrangement by the destination computing unit. This data pre-sorting and rearrangement step accounts for more than 60% of the total time consumed by the same computing unit in the entire communication process. Similarly, after the Combine phase, when the source computing unit receives multiple expert computation results returned by the destination computing unit, it also needs to organize them into the correct format. This approach results in an excessive computational load on the source computing unit, consuming its processing resources used for model computation. Furthermore, the need to generate multiple copies of each token data and transmit them separately leads to a large amount of network traffic and increases communication latency.

[0105] As an example, the existing technology includes the FUSCO communication scheme (a communication library designed for MoE model structures and expert parallel strategy scenarios). It pipelining data operations and network communication through metadata synchronization and orchestration, hiding the overhead of data operations within network communication. This means that data processing and network communication are executed concurrently; while transmitting the previous batch of data, the background prepares the next batch, significantly reducing the impact of data operations. However, FUSCO communication only utilizes XPU-level token data aggregation and does not fundamentally reduce the total amount of data transmitted over the network or the number of transmissions, therefore, there is still room for performance improvement.

[0106] In existing technologies, there are on-network computing solutions for Allreduce-type operations (a type of aggregated communication operation used to aggregate and broadcast data from multiple devices to achieve data synchronization), such as NVIDIA's NVLink-SHARP and Hygon's HSY-SHARP. These solutions offload the aggregated communication operation from the end-side computing unit to the network path device (such as a network interface card or switch), reducing the computational load of the end-side computing unit and the amount of data that needs to be transmitted, thus improving data transmission efficiency. However, the on-network computing solutions for Allreduce-type operations cannot be used for Alltoallv communication in the expert parallel scenario of the MoE model in this application because the multicast group in the Allreduce-type on-network computing solution is static, meaning that all data from an XPU uses the same multicast table. However, as mentioned above, the multicast target set for each token data in the Alltoallv traffic is dynamically changing and cannot be covered by a static multicast group configuration. Therefore, the expert parallel communication of the MoE model cannot use the on-network computing solutions for Allreduce-type operations in existing technologies.

[0107] Therefore, the current industry practices for Alltoallv communication in expert parallel scenarios of MoE models either suffer from huge network traffic and latency overhead, or can only perform data orchestration optimization on the computing unit side, resulting in a significant bottleneck in communication efficiency and severely restricting the overall business performance of large-scale MoE models.

[0108] Therefore, embodiments of this application provide a dynamic multicast communication method, system, switch, medium, and product based on on-network computing, which can be applied to, for example... Figure 5 In the application environment shown, switch 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be integrated onto server 102, or it can be located in the cloud or on other network servers. Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. It can be equipped with multiple computing units, and expert networks are deployed on these computing units. The expert networks perform expert calculations on the token data output by the current computing unit or other computing units.

[0109] In this embodiment, switch 101 refers to a network device responsible for data packet exchange and forwarding in the network. It can forward communication signals between multiple computing units on server 102. In this embodiment, it can refer to any network interconnection device with packet parsing, processing, copying, and forwarding capabilities, and its form can include box switches, chassis switches, top-of-rack switches, leaf / spine switches, etc. In particular, switch 101 in this embodiment integrates in-network computing (INC) functionality, which can understand and process the information carried in packets. From a functional implementation perspective, switch 101 can be a single physical device or a logical switching system composed of multiple physical devices.

[0110] In this embodiment, the message bearer protocol processed by switch 101 can be an Ethernet-like protocol (such as the SUE protocol defined by the Super Ethernet Alliance), or a standard Ethernet, InfiniBand (IB), InfiniBand, or any other link layer or network layer protocol that supports message semantics.

[0111] The computing unit mounted on server 102 refers to any hardware processor capable of performing AI computing tasks. The MoE model running on it employs an expert parallel strategy, with the expert network distributed across different computing units. In this embodiment, the computing unit can encompass a Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), Neural Network Processing Unit (NPU), Intelligence Processing Unit (IPU), or any other type of dedicated or general-purpose AI accelerator. As an example, the computing unit may use an NVIDIA GPU or a Huawei NPU. Functionally, a computing unit represents the unit that bears the computational load in the MoE model training or inference task; any processing unit that performs similar functions and completes model computation tasks can be considered a computing unit.

[0112] In one exemplary embodiment, such as Figure 6 As shown, a dynamic multicast communication method based on on-network computing is provided. This embodiment applies the method to the Dispatch stage of an expert parallel communication scenario in the MoE model, such as... Figure 7As shown, a schematic diagram of the entire process of expert parallel communication is further provided, that is, a schematic diagram of the overall process including the method in this embodiment, as well as the global synchronization, expert calculation and Combine stage communication executed after all steps in the method. In this diagram, XPU1 refers to the source computing unit, XPU2, XPU3 and XPU4 refer to the destination computing units, INC1IP refers to the switch port in the switch that receives the multicast request message sent by the source computing unit, INC2EP, INC3EP and INC4EP refer to the switch ports that send message copies to each destination computing unit, and CRE refers to the centralized replication engine in the switch.

[0113] The method in this embodiment is applied to Figure 1 Taking switch 101 as an example, the explanation includes the following steps 202 to 204. Wherein:

[0114] Step 201: Receive the multicast request message sent by the source computing unit and parse the multicast request message. The multicast request message carries payload data and dynamic multicast information. The source computing unit is the computing unit that outputs the multicast request message to the switch among multiple computing units.

[0115] In this context, the source computing unit refers to the computing unit that needs to send token data to the expert network on other computing units for processing during the Dispatch phase of expert parallel communication; it is the initiator of multicast communication. In this embodiment, the source computing unit determines the destination expert network and the computing unit where that expert network is located for each token data based on the routing results calculated by the gating network, and then generates a multicast request message.

[0116] Among them, the multicast request message is a data packet generated by the source computing unit after the AI ​​application initiates a collective communication operation, which is used to request the switch to perform dynamic multicast operation; the AI ​​application is also an AI-related business application, such as a training framework (DeepSpeed) or an inference framework (vLLM).

[0117] The payload data is the actual user data that needs to be transmitted carried in the multicast request message, which is the core content of the communication. In this embodiment, it is token data. A multicast request message can carry one or more token data.

[0118] Dynamic multicast information is a set of control information appended by the source computing unit to the multicast request message, instructing the switch how to multicast replicate and route the message. The term "dynamic" here refers to the configuration of the multicast group in existing Allreduce-type on-grid computing schemes.

[0119] In this embodiment, the routing result for each token data is different, causing its multicast target set (i.e., the combination of destination computation units and destination experts) to change dynamically. Therefore, dynamic information needs to be carried in the message. Accordingly, the dynamic multicast information in this embodiment is set independently for each or each group of multicast request messages, rather than being statically configured. It includes at least the destination computation unit set and the destination expert set corresponding to the payload data to be transmitted in the current message. The destination computation unit set and the destination expert set can be an explicit field containing a list of destination computation unit identifiers and a list of destination expert identifiers, or they can be some kind of identifier or index. The switch can use this to look up the multicast objects and address mapping rules related to the context of the message.

[0120] In this context, parsing refers to the switch extracting the field content of dynamic multicast information from a specific offset position in the multicast request message. This includes a set of destination computation units in list form and their effective length, which determines the number of message copies subsequently replicated. In this embodiment, the switch can trigger the parsing operation by recognizing a specific identifier (such as Ethernet type or opcode) in the multicast request message.

[0121] Step 202: Based on the number of destination calculation unit identifiers in the dynamic multicast information, copy the multicast request message into the same number of message copies.

[0122] The destination computation unit identifier is the identifier of the destination computation unit within the set of destination computation units in the dynamic multicast information. Each destination computation unit identifier indicates a destination computation unit that needs to receive a copy of the message. The computation unit identifier is an identity label indicating a computation unit in the network. It can be the computation unit's physical ID (identification number), logical number (such as DP plane number), or network address, which is not restricted here.

[0123] The message copy is a message generated by the switch based on the number of destination computation unit identifiers in the dynamic multicast information (i.e., the effective length of the destination computation unit set parsed in step 201). It has the same payload data as the original multicast request message, but the destination address may be different. Each message copy corresponds to one destination computation unit.

[0124] Step 203: For each message copy, determine the destination computing unit identifier and destination expert identifier corresponding to the message copy based on the dynamic multicast information, and determine the destination memory address corresponding to the message copy based on the destination expert identifier and the pre-configured address dense list table; wherein, the address dense list table is used to store the base address and expert identifier of the memory block allocated to the expert network deployed on each computing unit, and the destination memory address is the base address of the memory block corresponding to the destination expert identifier in the address dense list table.

[0125] The address dense array is a mapping table that the switch receives and stores on each switch port before formal communication. It is used to record the mapping relationship between the base address of the memory block allocated by each computing unit for its deployed expert network and the expert identifier. The memory block is also the pre-allocated buffer corresponding to the expert network for receiving input data.

[0126] The destination expert identifier is an identity label corresponding to the destination computing unit identifier, used to indicate the target expert network responsible for processing message copies on the corresponding destination computing unit. The destination expert identifier is also a local index of the corresponding expert network on the destination computing unit, rather than a globally unique ID. This design simplifies the address calculation of the destination computing unit.

[0127] The destination memory address is the base address of the physical memory block where the payload data of the current packet copy is ultimately stored on the destination computing unit. The switch obtains the base address of the memory block corresponding to the destination expert identifier by querying the address dense packing table, and can generate an accurate destination memory address based on the base address and the internally maintained offset pointer. The offset pointer is automatically shifted backward by the size of the data written each time data is written, to ensure that the data is compactly arranged in the memory block.

[0128] Step 204: Associate the destination memory address with the message copy and forward the message copy to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

[0129] The way to associate the destination memory address with the packet replica can be by writing the destination memory address into the corresponding address field in the packet replica, or by using the switch's internal metadata (that is, attaching the destination memory address to the internal descriptor of the packet replica) or the port-side mapping table (that is, setting a mapping table and directly mapping the packet replica to the corresponding destination memory address by looking up the table). As long as the binding between the packet replica and the destination memory address can be achieved, it is acceptable.

[0130] Specifically, the switch modifies the protocol header of the message copy, filling the destination memory address determined in step 203 into the corresponding address field (such as the "Addr." field) to replace the initial value of this field (such as an empty or invalid value) when the source computing unit sends the request message. Then, the modified message copy is sent from the switch port directly connected to the destination computing unit and delivered to the destination computing unit so that the destination computing unit can perform the write operation on the payload data in the message copy.

[0131] In the above-described dynamic multicast communication method based on on-network computing, firstly, the switch receives multicast request messages sent by the source computing unit and parses them; then, based on the number of destination computing unit identifiers in the dynamic multicast information, the multicast request message is copied into the same number of message copies; next, for each message copy, the destination computing unit identifier and destination expert identifier corresponding to the message copy are determined according to the dynamic multicast information, and the destination memory address corresponding to the message copy is determined according to the destination expert identifier and a pre-configured address dense array table; finally, the destination memory address is associated with the message copy, and the message copy is forwarded to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address. In this embodiment, the source computing unit only needs to generate and send one original message carrying dynamic multicast information. The message copying and address orchestration are offloaded to the switch. The switch generates a destination memory address independently for each message copy by querying a pre-configured address dense table, and writes the payload data in the message copy directly to the correct location of the corresponding expert memory block on the destination computing unit. Compared to the traditional method where the source computing unit needs to organize and send K message copies for K expert networks, in this embodiment, the source computing unit only needs to send one data copy. Theoretically, its computing resource consumption can be reduced to 1 / K of the original, which significantly reduces the network transmission traffic of the source computing unit and the computing resources used to perform multicast communication operations, thereby improving the overall training and inference efficiency of the MoE model.

[0132] In one exemplary embodiment, the pre-configuration steps for the address dense table are as follows:

[0133] Step a1: Obtain the base address of the memory block allocated to each expert network deployed on each computing unit, which corresponds to each switch port of the switch, and obtain the expert identifier of all expert networks.

[0134] In this parallel expert communication process, each computing unit can act as either a source or a destination computing unit. Therefore, before the communication begins, each computing unit pre-allocates a memory block in its local memory for each expert network deployed on it. This memory block is used to receive token data from the source computing unit during the Dispatch phase when the computing unit is acting as the destination computing unit. Additionally, before the communication, each computing unit also pre-allocates a copy memory block in its local memory. This copy memory block is used to receive expert computation results sent back from the destination computing unit during the Combine phase when the computing unit is acting as the source computing unit.

[0135] The base address is the starting physical memory address of each allocated memory block.

[0136] Specifically, when using a multi-plane, multi-track topology (i.e., multiple switches are set up, and each switch has multiple switch ports), a computing unit allocates an independent memory block for each track port of each switch to one of its expert networks to ensure the isolation and parallel processing of data flows.

[0137] Step a2: For each switch port, associate and store the expert identifier of its corresponding expert network and the base address of the memory block in the address dense table of the switch port.

[0138] The address dense array establishes a mapping record from expert identifiers to memory block base addresses. This storage method allows the switch to quickly obtain the corresponding target memory address by performing a local lookup based on the expert identifier carried in the packet when forwarding packets, without needing to query across networks.

[0139] As an example, in an 8-plane, 2-track scenario, the physical connection diagram between the computing unit XPU1 and the 8 switches is shown below. Figure 8 As shown, "Port" refers to the switch port of the corresponding switch. In this case, the computing unit XPU1 allocates one memory block to each of the two switch ports of the eight expert networks deployed on it (e.g., eight in this example). That is, each expert network is allocated a total of 16 memory blocks (e.g., the memory blocks allocated to expert network E1 are shown in the diagram as: Plane 1 Track 1, Plane 2 Track 2, ..., Plane 8 Track 1, Plane 8 Track 2). The base addresses of these 16 memory blocks are managed by the two switch ports of the eight switches. A schematic diagram showing how each switch port associates and stores the expert identifier of the corresponding expert network with the base address of the memory block is shown below. Figure 9 As shown, the computing unit can send the base addresses of all its memory blocks to the corresponding switch via the network manager or INC manager. The address-mapping table for each switch port in the switch is in the form of [EID: addr], where EID represents the expert identifier and addr represents the base address of the memory block corresponding to the expert network. Correspondingly, as... Figure 10 As shown, the computing unit XPU1 pre-allocates a copy memory block in its local memory, which is illustrated in the figure as a Tensor Buffer.

[0140] In this embodiment, the mapping relationship between the base address of the expert memory block on each computing unit and the expert identifier is fixed on the corresponding port of the switch before the formal data communication is initiated. Based on this, in the subsequent data communication stage, when the switch performs address translation for each packet copy, it only needs to perform a local table lookup operation with extremely low latency to immediately obtain the destination memory address. This avoids remote address lookup or negotiation across the network during expert parallel communication, further reducing communication latency and improving communication efficiency.

[0141] In an exemplary embodiment, before receiving the multicast request message sent by the source computing unit, the method further includes:

[0142] Receive a link establishment request sent by the source computing unit. The link establishment request carries the source computing unit identifier and the switch port identifier of the active computing unit.

[0143] Based on the switch port corresponding to the switch port identifier, a connection establishment response is sent to the source computing unit according to the source computing unit identifier to establish a connection between the switch port and the source computing unit.

[0144] The chain establishment request is a handshake message in which the two communicating parties exchange initial parameters (such as the initial message sequence number PSN, communication buffer, etc.) before conducting data communication based on message semantics (similar to TCP or RoCEv2's QP (QueuePair) connection).

[0145] As an example, a diagram illustrating the connection between a switch and different computing units is shown below. Figure 11 As shown, the four computing units establish communication connections with different switch ports of the switch. The message structures for connection requests and responses are as follows: Figure 12 As shown, the chain establishment request is represented as syn, and the chain establishment response is represented as syn ack. Both include ETH Hdr (similar to the SUE-like Ethereum message header), ETH Type (Ethereum type), and Protocol Hdr (transmission header, which contains control fields related to reliable transmission). The Protocol Hdr includes the OpCode (opcode, indicating the operation type) identifier, Addr. (message operation address, such as the peer write address of a write operation message, or the peer read address of a read message), and PSN (message sequence number, used for reliable control of transmission).

[0146] In this embodiment, the ETH Hdr in the chain establishment request message includes a source computing unit identifier field and a switch port identifier field. The source computing unit identifier field is represented as S-XPU, and the switch port identifier field is represented as D-XPU. The switch port identifier D-XPU can be an ID of a specific switch port directly connected to the source computing unit, pre-configured by the INC manager, or it can be a pre-agreed ID representing all ports in the entire switch (such as 0xFFFF). In this case, all computing units use a common ID as the D-XPU to initiate the chain establishment request. The INC module in the directly connected switch port can identify the chain establishment request after detecting this ID.

[0147] In this embodiment, by establishing a connection between the source computing unit and the switch port before data communication, a foundation can be laid for reliable transmission of multicast request messages in subsequent steps. During this process, the source computing unit and the switch can negotiate and synchronize transmission parameters such as the initial sequence number, thereby achieving reliable end-to-end transmission in the subsequent dynamic multicast process.

[0148] In an exemplary embodiment, the dynamic multicast information includes a set of target computing units and a set of target experts corresponding to the set of target computing units. The set of target computing units includes multiple target computing unit identifiers, and the set of target experts includes the target expert identifiers of the target expert network deployed on the target computing unit corresponding to each target computing unit identifier.

[0149] Both the target calculation unit set and the target expert set can be ordered lists, and there is a one-to-one mapping relationship between the two lists.

[0150] After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes:

[0151] For each message replica, a multicast index is generated for that message replica.

[0152] The multicast index is a sequential number assigned to each packet replica by the switch when generating it. For example, when K replicas are generated, the multicast indexes are 0, 1, 2...K-1.

[0153] Correspondingly, for each message copy, the destination computing unit identifier and destination expert identifier corresponding to the message copy are determined based on the dynamic multicast information, including:

[0154] For each message replica, the destination computation unit identifier in the destination computation unit set that corresponds to the multicast index of the message replica is used as the destination computation unit identifier for the message replica.

[0155] The destination expert identifier in the destination expert set that corresponds to the multicast index is used as the destination expert identifier for the message copy.

[0156] It should be noted that since the destination computation unit set and the destination expert set are ordered and correspond one-to-one, for each packet copy, the switch only needs to use the multicast index as the offset to accurately obtain the destination computation unit identifier and destination expert identifier that uniquely correspond to the packet copy from the two sets.

[0157] In this embodiment, by introducing a multicast index, dynamic multicast information is organized into an ordered set and accessed through the multicast index. That is, it is not necessary to parse the destination computing unit set and the destination expert set separately for each message copy. It is only necessary to allocate a simple multicast index to each message copy and use the multicast index to extract the corresponding identification information from the destination computing unit set and the destination expert set. Based on this, the message copying and address allocation process can be simplified.

[0158] In an exemplary embodiment, the multicast request message further carries a source identifier field (S-XPU), a destination identifier field (D-XPU), and a destination memory address field (Addr.). The initial value of the source identifier field is the source compute unit identifier of the source compute unit, and the initial values ​​of the destination identifier field and the destination memory address field are empty or invalid. After parsing the multicast request message, the method further includes:

[0159] Replace the value of the source identifier field with the switch port identifier of the switch.

[0160] It should be noted that this step, also known as the proxy step, is used to guide the response message of the multicast request message back to the current switch port.

[0161] After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes:

[0162] Set the value of the target identifier field to the target computation unit identifier, and set the value of the target memory address field to the target memory address.

[0163] It should be noted that this step is used to complete the message copy so that it becomes a standard unicast write request that can be directly delivered to a specific memory address of the destination computing unit.

[0164] In an exemplary embodiment, after the destination computing unit writes the payload data from the message copy into the memory block indicated by the destination memory address, the method further includes:

[0165] Receive the response message returned by the destination computing unit. The response message carries the source identifier field, the destination identifier field, and the destination memory address field.

[0166] Modify the value of the source identifier field to the destination compute unit identifier, modify the value of the target identifier field to the source compute unit identifier, and modify the value of the target memory address field to the copy write-back address corresponding to the packet copy;

[0167] The response message is sent to the source computing unit based on the target identifier field.

[0168] It should be noted that this step is the reverse proxy step, which is used to guide the response message to be accurately sent back to the source computing unit.

[0169] This embodiment discloses a technical solution for proxy replacement of the source and destination identifier fields in the transmission and reception process of the switch. For the source computing unit, it only sends a request and ultimately receives a response, just like performing a standard unicast communication, without needing to perceive the complexity of multicast. For the destination computing unit, it receives and replies with a standard unicast message. In this embodiment, the switch completely encapsulates the complex multicast response relationship within itself by performing identity replacement in the inbound and outbound directions, which can avoid the high context management complexity caused by multicast replication. At the same time, after the destination computing unit completes the expert calculation, it can directly use the write-back address carried in the response message to initiate the result write-back without renegotiating the address in the Combine phase, which helps to reduce the total communication latency.

[0170] In one exemplary embodiment, the multicast request message also carries a message sequence number, which is initially bound to the original message sequence number oPSN.

[0171] Among them, the original message sequence number (oPSN) is the sequence number carried in the message sent by the source computing unit, which is used for reliable connection between the source computing unit and the switch.

[0172] After parsing the multicast request message, the method also includes:

[0173] Assign an internal sequence number (iPSN) to the multicast request message and establish a mapping relationship between the original message sequence number (oPSN) and the internal sequence number (iPSN).

[0174] The Internal Sequence Number (iPSN) is a unique internal sequence number assigned by the switch to each incoming request packet, used for managing multicast tasks within the switch.

[0175] The mapping relationships are maintained through mapping table entries (such as iPSN mapping table and ePSN mapping table), which are the basis for sequence number translation and response aggregation.

[0176] In this embodiment, the mapping relationship between the original message sequence number oPSN and the internal sequence number iPSN can be maintained through an iPSN mapping table, which is in the form of [iPSN, oPSN, S-XPU, Rsp_cnt], where Rsp_cnt is equal to the effective length of the destination computing unit set (i.e., the number of destination XPUs, such as K).

[0177] Replace the original message sequence number oPSN bound to the message sequence number with the internal sequence number iPSN;

[0178] After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes:

[0179] For each message copy, an outgoing sequence number (ePSN) is assigned to the message copy, and a mapping relationship is established between the internal sequence number (iPSN) and the outgoing sequence number (ePSN).

[0180] The outgoing sequence number (ePSN) is a sequence number assigned by the switch to each copy of a message sent to the destination computing unit, used for reliable connection between the switch port and the destination computing unit.

[0181] In this embodiment, the mapping relationship between the internal serial number iPSN and the outgoing serial number ePSN can be maintained through an ePSN mapping table, which is in the form of [ePSN, iPSN, S-XPU].

[0182] Replace the internal sequence number iPSN bound to the message sequence number with the outbound sequence number ePSN.

[0183] In this embodiment, the processing steps for the message sequence number in the multicast request message are disclosed, which can be called the Retag operation (i.e., re-numbering the message). Based on this, a sequence number management rule that is completely independent of the source end (source computing unit) and the destination end (destination computing unit) is built inside the switch. This allows the inbound processing module and outbound processing module of the switch to perform sequence number management independently without the need to achieve cross-module synchronization with the computing unit, which helps to simplify the hardware design of large-scale switch chips.

[0184] In an exemplary embodiment, after the destination computing unit writes the payload data from the message copy into the memory block indicated by the destination memory address, the method further includes:

[0185] Receive the response message returned by the destination computing unit. The response message carries a message sequence number.

[0186] Based on the mapping relationship between the internal sequence number iPSN and the outgoing sequence number ePSN, the message sequence number is rebound to the internal sequence number iPSN;

[0187] Based on the mapping relationship between the original message sequence number oPSN and the internal sequence number iPSN, the message sequence number is rebound to the original message sequence number oPSN;

[0188] The system retrieves the number of times the message sequence number is rebound to the internal sequence number iPSN or the original message sequence number oPSN. If this number equals the number of destination calculation unit identifiers in the dynamic multicast information (e.g., K), it combines (or aggregates) all response messages into a final response and sends them to the source calculation unit based on the target identifier field.

[0189] In this embodiment, a reverse processing step for the message sequence number in the response message is disclosed, which can be called the reverse Retag operation (i.e., re-numbering the message in reverse). Based on this, the switch can not only correctly restore the sequence numbers of multiple response messages to the original original message sequence number oPSN, but also accurately determine when the response messages of all K message replicas have been received through a preset response counter (Rsp_cnt). Only when all the response messages of the message replicas have been successfully returned will the switch reply to the source computing unit with a final aggregated response. This enables the counting and management of multiple asynchronously arriving responses. For the source computing unit, it only needs to initiate a request once and wait for a response, just like performing standard unicast communication.

[0190] In this embodiment, a schematic diagram illustrating the switch performing proxy and deproxy operations, as well as retag and unretag operations, is shown below. Figure 13 As shown, CRE represents the centralized replication engine in the switch, INC_IP represents the inbound processing module in the switch, and INC_EP represents the outbound processing module in the switch.

[0191] In an exemplary embodiment, the dynamic multicast information further includes a write-back address list, which stores the write-back addresses of the replica memory blocks pre-allocated on the source computing unit for all message replicas; after replicating the multicast request message into the same number of message replicas according to the number of destination computing unit identifiers in the dynamic multicast information, the method further includes:

[0192] For each message copy, retrieve the copy write-back address corresponding to the message copy from the write-back address list;

[0193] Write the copy write-back address into the message copy so that after forwarding the message copy to the corresponding destination computing unit, the destination computing unit writes the expert calculation result corresponding to the message copy back to the copy memory block indicated by the copy write-back address.

[0194] Based on this embodiment, after forwarding a copy of a message to the corresponding destination computing unit, the destination computing unit can store the write-back address of the copy, and after completing the expert calculation, write the expert calculation result corresponding to the copy of the message back to the copy memory block indicated by the copy write-back address.

[0195] In this embodiment, after the destination computing unit completes the expert calculation, it can directly initiate a data write-back operation using the pre-set write-back address in the message copy. This eliminates the need for the destination to re-query, negotiate, or calculate the return path and destination address for each token data during the Combine phase, and also eliminates the need for the source to perform complex address rearrangement of the returned out-of-order data during the Combine phase. Based on this, the two independent Alltoallv communications, the Dispatch phase and the Combine phase, can be chained together, avoiding the additional address resolution and data orchestration overhead of the Combine phase, thereby optimizing the entire Alltoallv communication process.

[0196] In this embodiment, the timing diagram for dynamic multicast communication and multicast protocol communication is as follows: Figure 14 As shown, the thick arrow at the top represents dynamic multicast communication, and the thin arrow at the bottom represents multicast protocol communication. The flowcharts for dynamic multicast communication and multicast protocol communication are as follows: Figure 15 As shown, Figure 15 In dynamic multicast communication and multicast protocol communication, the message structure diagrams for each stage are shown below. Figure 16 As shown, in the Alltoallv field, XPU ID list represents a list of target computational units, Exp.list represents a list of target experts, WB Addr.list represents the write-back address field, and Payload represents the payload data.

[0197] Furthermore, in practical applications, if the link layer has robust reliability (e.g., Link Level Retry (LLR) and Credit-Based Flow Control (CBFC)), the response process can be omitted. That is, after sending a dispatch request message, the source automatically generates a response (without waiting for a response from the network or network computing module), marking the request as complete. The flowchart for dynamic multicast communication and multicast protocol communication in this case is as follows: Figure 17 As shown.

[0198] In addition, in this embodiment, the message carrying protocol and format may be changed to other protocols and formats, that is, this embodiment does not have a mandatory requirement for the way effective information is carried. Figure 16The Ether header in the packet is described according to an Ether-like format such as SUE. In fact, as... Figure 18 As shown, the outermost header can also be a standard Ethernet header, or other link-layer headers (such as the IB or UB link-layer headers), while routing-related information (such as source / destination XPU IDs) can be placed in a network-layer header (such as...). Figure 18 NET Hdr).

[0199] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0200] Based on the same inventive concept, this application also provides a dynamic multicast communication system based on on-network computing. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more system embodiments provided below can be found in the limitations of the dynamic multicast communication method based on on-network computing described above, and will not be repeated here.

[0201] In one embodiment, such as Figure 11 As shown, a dynamic multicast communication system based on on-network computing is provided, including multiple computing units and a switch communicatively connected to the multiple computing units; the multiple computing units include a source computing unit and a destination computing unit corresponding to the source computing unit; wherein, the source computing unit is the computing unit among the multiple computing units that outputs a multicast request message to the switch, and the destination computing unit is the computing unit among the multiple computing units that is indicated by the multicast request message;

[0202] The source computing unit is used to generate and send multicast request messages to the switch. The multicast request messages carry payload data and dynamic multicast information.

[0203] A switch is used to perform the following operations:

[0204] Receive multicast request messages sent by the source computing unit and parse the multicast request messages;

[0205] Based on the number of destination calculation unit identifiers in the dynamic multicast information, the multicast request message is copied into the same number of message copies;

[0206] For each message copy, the destination computing unit identifier and destination expert identifier corresponding to the message copy are determined according to the dynamic multicast information. Then, the destination memory address corresponding to the message copy is determined according to the destination expert identifier and the pre-configured address dense list table. The address dense list table is used to store the base address and expert identifier of the memory block allocated to the expert network deployed on each computing unit. The destination memory address is the base address of the memory block corresponding to the destination expert identifier in the address dense list table.

[0207] The destination memory address is associated with the message copy, and the message copy is forwarded to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

[0208] As an example, Figure 19 A schematic diagram illustrating the benefits of dynamic multicast communication based on on-network computing in an AF-coupled scenario is presented. Figure 20 A schematic diagram illustrating the benefits of dynamic multicast communication based on on-network computing in an AF separation scenario is presented.

[0209] Among them, for example Figure 2 The AF coupling scenario shown indicates that the computation of the attention module and the expert part are performed on the same set of computational units. As described above, the Alltoallv communication in the Dispatch phase actually consists of a large number of token-level multicasts, where each token data is multicast to K experts. The Alltoallv communication in the Combine phase actually consists of a large number of token-level multi-receives (overlaying reduction operations), where multi-receives can be implemented as responses to multicast read requests. Ignoring the implementation scheme, the network traffic gains (corresponding to network transmission and reception time, which is the lower bound of the operation time) are as follows: Figure 19 As shown in the figure, the circles represent computing units (including computing units 1 to 3). It can be seen that after applying the dynamic multicast communication scheme based on on-network computing in this embodiment, the transmit and receive traffic of Alltoallv communication decreased from 2KN to (K+1)N, where N is the number of token data in a batch for each DP, and the ratio is... In other words, applying this embodiment in an AF-coupled scenario can reduce network communication time by approximately 50%.

[0210] In actual large-scale model training / inference, since parameters are concentrated in the expert part, and experts are distributed across multiple dynamic programming (DP) nodes, parallel communication among experts accounts for the majority (approximately 60%-80%+) of the total network communication. After on-network computation in this embodiment, the overall service time can be reduced by up to 30%-40%. However, in actual training, communication and computation are mutually hidden, so simply reducing communication time cannot directly achieve the theoretical benefits. Considering other optimization methods and overhead, the final service time may be reduced by approximately 15-20%.

[0211] In scenarios where autofocus (AF) is separated, the computational characteristics of the attention module and the expert part differ significantly. The attention module is memory-intensive (requiring substantial memory for storing the key-value (K) and value (V) caches, but with relatively low computational cost), while the expert part is purely computational. Therefore, many inference scenarios have adopted an optimized architecture of AF separation, where the attention module and expert part are computed on separate sets of computational units, such as... Figure 20 As shown, since the computational load of experts is relatively light, the computational power of the expert group can be fully utilized by having one expert group undertake the computational tasks of multiple attention modules. In this scenario, a single Dispatch or Combine phase can bring communication benefits (unlike AF coupled scenarios, which require the Dispatch and Combine phases to be combined to achieve benefits).

[0212] In this embodiment, a system with on-network computing capabilities is disclosed. In this system, when the MoE model running on multiple computing units triggers Alltoallv communication, a large number of token data distribution and multicast replication tasks are offloaded to the switches connecting the computing units. This makes the entire system appear as follows: the computing resources of the computing units are more concentrated on model computation, while the switches are responsible not only for data transmission but also for communication mode conversion and address orchestration. Based on this, it helps to reduce the network transmission traffic of the source computing units and the computing resources used to perform multicast communication operations.

[0213] It should be noted that the working process, working details and technical effects of the dynamic multicast communication system based on on-network computing provided in this embodiment can be found in the embodiment of the dynamic multicast communication method based on on-network computing, and will not be repeated here.

[0214] In one exemplary embodiment, such as Figure 21 As shown, the switch includes switch ports and a centralized replication engine that communicates with the switch ports. The switch ports are divided into an inbound processing module that communicates with the source computing unit and an outbound processing module that communicates with the destination computing unit; wherein,

[0215] The inbound processing module is used to receive multicast request messages sent by the source computing unit and parse the multicast request messages;

[0216] The centralized replication engine is used to replicate multicast request messages into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information.

[0217] The centralized replication engine is also used to determine the destination computing unit identifier and destination expert identifier corresponding to each message copy based on the dynamic multicast information, and to determine the destination memory address corresponding to the message copy based on the destination expert identifier and the address dense array table, and then associate the destination memory address with the message copy.

[0218] The outgoing processing module is used to forward each message copy to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

[0219] In this embodiment, both the inbound and outbound processing modules integrate an INC module, which can be implemented using a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a network processor (NP) to enable on-network computing.

[0220] In one exemplary embodiment, the system further includes a manager;

[0221] The manager is used to obtain the base address and corresponding expert identifier of the memory block allocated by each computing unit to the expert network it deploys, which corresponds to each switch port of the switch. For each switch port, the manager associates the expert identifier and the base address of the memory block of the corresponding expert network with the switch port and sends them to the switch port.

[0222] The switch port is used to associate and store the expert identifier of the received expert network with the base address of the memory block in the address dense table of the switch port.

[0223] Based on the same inventive concept, this application also provides a switch. The solution provided by this switch is similar to the solution described in the above method. Therefore, the specific limitations of the switch provided below can be found in the limitations of the dynamic multicast communication method based on on-network computing described above, and will not be repeated here.

[0224] In one embodiment, a switch is provided for performing any of the methods described above.

[0225] It should be noted that the working process, working details and technical effects of the dynamic multicast communication system based on on-network computing provided in this embodiment can be found in the embodiment of the dynamic multicast communication method based on on-network computing, and will not be repeated here.

[0226] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0227] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0228] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0229] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0230] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A dynamic multicast communication method based on on-network computing, characterized in that, Performed by a switch communicatively connected to multiple computing units, the method includes: The system receives a multicast request message sent by a source computing unit and parses the multicast request message, which carries payload data and dynamic multicast information. The source computing unit is the computing unit among the plurality of computing units that outputs the multicast request message to the switch. Based on the number of destination calculation unit identifiers in the dynamic multicast information, the multicast request message is copied into the same number of message copies; For each message copy, the destination computing unit identifier and destination expert identifier corresponding to the message copy are determined according to the dynamic multicast information, and the destination memory address corresponding to the message copy is determined according to the destination expert identifier and the pre-configured address dense list table; wherein, the address dense list table is used to store the base address and expert identifier of the memory block allocated to the expert network deployed on each computing unit, and the destination memory address is the base address of the memory block corresponding to the destination expert identifier in the address dense list table; The destination memory address is associated with the message copy, and the message copy is forwarded to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

2. The method according to claim 1, characterized in that, The pre-configuration steps for the address dense array table are as follows: Obtain the base address of the memory block allocated on each computing unit to each expert network deployed thereon, which corresponds to each switch port of the switch, and obtain the expert identifier of all expert networks. For each switch port, the expert identifier of its corresponding expert network and the base address of its memory block are associated and stored in the address dense table of the switch port.

3. The method according to claim 1, characterized in that, The dynamic multicast information includes a set of destination computing units and a set of destination experts corresponding to the set of destination computing units. The set of destination computing units includes multiple destination computing unit identifiers, and the set of destination experts includes destination expert identifiers of the destination expert network deployed on the destination computing unit corresponding to each destination computing unit identifier. After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes: For each message replica, a multicast index is generated for that message replica. For each message copy, determining the destination computing unit identifier and destination expert identifier corresponding to the message copy based on the dynamic multicast information includes: For each message replica, the destination computing unit identifier in the destination computing unit set that corresponds to the multicast index of the message replica is used as the destination computing unit identifier corresponding to the message replica. The destination expert identifier in the destination expert set that corresponds to the multicast index is used as the destination expert identifier corresponding to the message copy.

4. The method according to claim 1, characterized in that, The multicast request message also carries a source identifier field, a destination identifier field, and a destination memory address field. The initial value of the source identifier field is the source computing unit identifier of the source computing unit, and the initial values ​​of the destination identifier field and the destination memory address field are empty. After parsing the multicast request message, the method further includes: Replace the value of the source identifier field with the switch port identifier of the switch; After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes: The value of the target identifier field is set to the target computing unit identifier, and the value of the target memory address field is set to the target memory address.

5. The method according to claim 4, characterized in that, After the destination computing unit writes the payload data from the message copy into the memory block indicated by the destination memory address, the method further includes: Receive a response message returned by the destination computing unit, the response message carrying a source identifier field, a target identifier field, and a target memory address field; The value of the source identifier field is modified to the destination computing unit identifier, the value of the target identifier field is modified to the source computing unit identifier, and the value of the target memory address field is modified to the copy write-back address corresponding to the message copy; The response message is sent to the source computing unit based on the target identifier field.

6. The method according to claim 5, characterized in that, The multicast request message also carries a message sequence number, which is initially bound to the original message sequence number; after parsing the multicast request message, the method further includes: Assign an internal sequence number to the multicast request message and establish a mapping relationship between the original message sequence number and the internal sequence number; Replace the original message sequence number bound to the message sequence number with the internal sequence number; After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes: For each message copy, an outgoing sequence number is assigned to the message copy, and a mapping relationship is established between the internal sequence number and the outgoing sequence number; Replace the internal sequence number bound to the message sequence number with the outgoing sequence number.

7. The method according to claim 6, characterized in that, After the destination computing unit writes the payload data from the message copy into the memory block indicated by the destination memory address, the method further includes: Receive a response message returned by the destination computing unit, the response message carrying a message sequence number; Based on the mapping relationship between the internal sequence number and the outgoing sequence number, the message sequence number is rebound to the internal sequence number; Based on the mapping relationship between the original message sequence number and the internal sequence number, the message sequence number is rebound to the original message sequence number; The system obtains the number of times the message sequence number is rebound to the internal sequence number or the original message sequence number. If the number of times is equal to the number of destination calculation unit identifiers in the dynamic multicast information, the system sends all response messages to the source calculation unit according to the target identifier field.

8. The method according to claim 1, characterized in that, The dynamic multicast information also includes a write-back address list, which is used to store the write-back addresses of the replica memory blocks pre-allocated for all message replicas on the source computing unit. After copying the multicast request message into the same number of message copies based on the number of destination calculation unit identifiers in the dynamic multicast information, the method further includes: For each message copy, obtain the copy write-back address corresponding to the message copy from the write-back address list; The copy write-back address is written into the message copy so that after the message copy is forwarded to the corresponding destination computing unit, the destination computing unit writes the expert calculation result corresponding to the message copy back to the copy memory block indicated by the copy write-back address.

9. The method according to claim 1, characterized in that, Before receiving the multicast request message sent by the source computing unit, the method further includes: Receive a connection establishment request sent by the source computing unit, the connection establishment request carrying the source computing unit identifier and the switch port identifier of the source computing unit; Based on the switch port corresponding to the switch port identifier, a connection establishment response is sent to the source computing unit according to the source computing unit identifier to establish a connection between the switch port and the source computing unit.

10. A dynamic multicast communication system based on on-network computing, comprising multiple computing units and a switch communicatively connected to the multiple computing units; characterized in that, The plurality of computing units include a source computing unit and a destination computing unit corresponding to the source computing unit; wherein, the source computing unit is the computing unit that outputs a multicast request message to the switch among the plurality of computing units, and the destination computing unit is the computing unit indicated by the multicast request message among the plurality of computing units; The source computing unit is used to generate and send the multicast request message to the switch. The multicast request message carries payload data and dynamic multicast information. The switch is used to perform the following operations: Receive the multicast request message sent by the source computing unit and parse the multicast request message; Based on the number of destination calculation unit identifiers in the dynamic multicast information, the multicast request message is copied into the same number of message copies; For each message copy, the destination computing unit identifier and destination expert identifier corresponding to the message copy are determined according to the dynamic multicast information, and the destination memory address corresponding to the message copy is determined according to the destination expert identifier and the pre-configured address dense list table; wherein, the address dense list table is used to store the base address and expert identifier of the memory block allocated to the expert network deployed on each computing unit, and the destination memory address is the base address of the memory block corresponding to the destination expert identifier in the address dense list table; The destination memory address is associated with the message copy, and the message copy is forwarded to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

11. The system according to claim 10, characterized in that, The switch includes switch ports and a centralized replication engine communicatively connected to the switch ports. Each switch port is divided into an inbound processing module communicatively connected to the source computing unit and an outbound processing module communicatively connected to the destination computing unit. The incoming processing module is used to receive multicast request messages sent by the source computing unit and parse the multicast request messages. The centralized replication engine is used to replicate the multicast request message into the same number of message copies based on the number of destination computing unit identifiers in the dynamic multicast information. The centralized replication engine is also used to determine the destination computing unit identifier and destination expert identifier corresponding to each message copy based on the dynamic multicast information, and to determine the destination memory address corresponding to the message copy based on the destination expert identifier and the address dense array table, and then associate the destination memory address with the message copy. The outgoing processing module is used to forward each message copy to the destination computing unit corresponding to the destination computing unit identifier, so that the destination computing unit writes the payload data in the message copy into the memory block indicated by the destination memory address.

12. The system according to claim 11, characterized in that, The system also includes a manager; The manager is used to obtain the base address and corresponding expert identifier of the memory block allocated by each computing unit to the expert network it deploys and corresponding to each switch port of the switch, and for each switch port, associates the expert identifier and the base address of the memory block of the corresponding expert network and sends them to the switch port. The switch port is used to associate and store the expert identifier of the received expert network with the base address of the memory block in the address dense table of the switch port.

13. A switch, characterized in that, The switch is used to perform the method as described in any one of claims 1 to 9.

Images