Method, system and apparatus for group communication

Abstract

The present disclosure provides a set communication method, system, device, electronic equipment, storage medium and computer program product, relates to the technical field of artificial intelligence, in particular to the technical field of large model, agent, distributed computing and cluster communication, and can be applied to a set communication scene. The specific implementation scheme is as follows: a source computing unit generates a set communication task, and transmits the set communication task to a source network card, the set communication task including to-be-processed data and object representation data of a plurality of processing objects of the to-be-processed data; the source network card determines the destination nodes where the plurality of processing objects are respectively located according to the object representation data and a routing mapping relationship; and the source network card transmits the to-be-processed data and the distribution identifier of the processing object to the destination nodes where the processing objects are located. The present disclosure reduces the size of the physical transmission path while having multicast capability, and saves the network bandwidth across nodes.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to the technical fields of large models, intelligent agents, distributed computing and cluster communication, and in particular to a cluster communication method, device, electronic device, storage medium and computer program product that can be applied to cluster communication scenarios. Background Technology

[0002] In aggregated communication scenarios (such as large-scale distributed inference scenarios based on hybrid expert models), both intra-node and inter-node aggregated communication exhibit three characteristics distinct from traditional dense models: extremely small message size, high fan-out and highly dynamic communication, and extremely high latency sensitivity. This small message size, high fan-out, and high latency sensitivity of many-to-many communication patterns place stringent demands on both intra-node and inter-node communication paths, but there is a significant asymmetry in the hardware support capabilities for these two aspects. For intra-node communication, GPUs (Graphics Processing Units) and Streaming Multiprocessors (SMs) can directly trigger hardware multicast within the NVSwitch (high-speed switching chip) via Multimem load / store instructions, natively broadcasting a token data set to multiple GPUs within the NVLink (high-speed interconnect bus) domain. This native hardware multicast capability effectively matches the high fan-out characteristic. However, for inter-node communication, cross-node links lack equivalent native hardware multicast capabilities. Summary of the Invention

[0003] This disclosure provides a collection of communication methods, systems, devices, electronic devices, storage media, and computer program products.

[0004] According to the first aspect, a collective communication method is provided, applied to a source node, comprising: generating a collective communication task through a source computing unit in the source node, and transmitting the collective communication task to a source network interface card (NIC) of the source node, wherein the collective communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed; determining the destination node of each of the multiple processing objects based on the object representation data and a routing mapping relationship through the source NIC, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination node of the processing object; and transmitting the data to be processed and the distribution identifier of the processing object to the destination node of the processing object through the source NIC.

[0005] According to the second aspect, a collection communication method is provided, applied to a destination node, comprising: obtaining, through the destination network interface card (NIC) of the destination node, the data to be processed sent by the source NIC of the source node and the distribution identifier of the processing object of the data to be processed; determining, based on the distribution identifier and the distribution mapping relationship, the storage address of the target computing unit where the processing object is located in the destination node, wherein the distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the corresponding processing object is located; transmitting the data to be processed to the storage area indicated by the storage address through the destination NIC, and processing the data to be processed through the processing object.

[0006] According to a third aspect, a collective communication system is provided, comprising: a source node, configured to: generate a collective communication task through a source computing unit in the source node, and transmit the collective communication task to a source network interface card (NIC) of the source node, wherein the collective communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed; determine the destination node of each of the multiple processing objects through the source NIC based on the object representation data and a routing mapping relationship, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination node of the processing object; and transmit the data to be processed and the distribution identifier of the processing object to the destination node of the processing object through the source NIC; and a destination node, configured to: obtain the data to be processed and the distribution identifier through the destination NIC; determine the storage address of the target computing unit of the processing object in the destination node based on the distribution identifier and the distribution mapping relationship, wherein the distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit of the processing object corresponding to the distribution identifier; transmit the data to be processed to the storage area indicated by the storage address through the destination NIC, and process the data to be processed through the processing object.

[0007] According to a fourth aspect, a collective communication device is provided, deployed at a source node, comprising: a task generation unit configured to generate a collective communication task through a source computing unit in the source node, and transmit the collective communication task to a source network interface card (NIC) of the source node, wherein the collective communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed; a node determination unit configured to determine the destination node of each of the multiple processing objects according to the object representation data and a routing mapping relationship through the source NIC, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination node where the processing object is located; and a data transmission unit configured to transmit the data to be processed and the distribution identifier of the processing object to the destination node where the processing object is located through the source NIC.

[0008] According to a fifth aspect, a collection communication device is provided, deployed at a destination node, comprising: a data acquisition unit configured to acquire, via the destination network interface card (NIC) of the destination node, data to be processed sent by the source NIC of the source node and a distribution identifier of the processing object of the data to be processed; an address determination unit configured to determine, based on the distribution identifier and a distribution mapping relationship, the storage address of the target computing unit where the processing object is located in the destination node; and a data processing unit configured to transmit the data to be processed to the storage area indicated by the storage address via the destination NIC, and process the data to be processed via the processing object.

[0009] According to a sixth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first or second aspect.

[0010] According to a seventh aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first or second aspect.

[0011] According to the eighth aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in any implementation of the first or second aspect.

[0012] According to the technology disclosed herein, a collective communication method, system, and apparatus are provided. A source node generates a collective communication task through a source computing unit and transmits the collective communication task to a source network interface card (NIC). The collective communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed. Through the source NIC, the destination nodes of each of the multiple processing objects are determined based on the object representation data and a routing mapping relationship, where the routing mapping relationship represents the correspondence between the object representation data and the destination nodes of the processing objects. Through the source NIC, the data to be processed and the distribution identifiers of the processing objects are transmitted to the destination nodes of the processing objects. The destination nodes obtain the data to be processed and the distribution identifiers of the processing objects through the destination NIC. The system identifies the target computing unit where the processing object is located in the destination node based on the distribution identifier and the distribution mapping relationship. The distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the processing object is located. The system transmits the data to be processed to the storage area indicated by the storage address through the destination network card, and processes the data through the processing object. This pushes the multicast control plane down to the network card hardware pipeline. By issuing a single aggregate communication task, the network card hardware is triggered to realize the multicast process of the data to be processed across nodes. This enables the inter-node communication to have multicast capabilities while reducing the physical transmission path size and saving cross-node network bandwidth.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure; Figure 2 This is a schematic diagram of intra-node aggregate communication and inter-node aggregate communication; Figure 3 This is a timing diagram of the communication process under the CPU Proxy mode; Figure 4 This is a timing diagram of the communication process under the GIN / GDAKI mode; Figure 5 This is a system framework diagram of the publicly disclosed collective communication system; Figure 6 This is a flowchart of one embodiment of the collection communication method according to the present disclosure; Figure 7 This is a schematic diagram of the routing mapping relationship according to this embodiment; Figure 8This is a schematic diagram of the node connection mapping relationship according to this embodiment; Figure 9 A flowchart of an embodiment of the collection communication method according to this disclosure; Figure 10 This is a schematic diagram of the distribution mapping relationship according to this embodiment; Figure 11 This is a schematic diagram of processed data records according to this embodiment; Figure 12 This is a schematic diagram illustrating an application scenario of the collection communication method according to this embodiment; Figure 13 This is a structural diagram of an embodiment of the collective communication system according to the present disclosure; Figure 14 This is a structural diagram of one embodiment of the collective communication device according to the present disclosure; Figure 15 This is a structural diagram of one embodiment of the collective communication device according to the present disclosure; Figure 16 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] The technical solutions disclosed herein involve the collection, storage, use, processing, transmission, provision, and disclosure of various types of information, such as user personal information, in accordance with relevant laws and regulations and do not violate public order and good morals.

[0017] Figure 1 An exemplary architecture 100 is shown that can be applied to the collection communication methods and apparatus disclosed herein.

[0018] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0019] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data acquisition, interaction, and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0020] Server 105 can be a server that provides various services, such as receiving data processing requests from terminal devices 101, 102, and 103, and, during the aggregated communication process, pushing the multicast control plane down to the network interface card (NIC) hardware pipeline, triggering the NIC hardware to implement the multicast process of data to be processed across nodes by issuing a single aggregated communication task. As an example, server 105 can be a cloud server.

[0021] It should be noted that a server can be either hardware or software. When a server is hardware, it is generally implemented as a distributed server cluster consisting of multiple servers, but the possibility of implementing it as a single server cannot be ruled out. When a server is software, it is generally implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), but the possibility of implementing it as a single software program or software module cannot be ruled out. No specific limitations are made here.

[0022] It should also be noted that the collective communication method provided in the embodiments of this disclosure is generally executed by a server, but the possibility of it being executed by the server and terminal devices in cooperation with each other is not excluded. Accordingly, the various parts (e.g., various units) included in the collective communication device can be all located in the server, or they can be located separately in the server and the terminal device.

[0023] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. When the electronic devices running the aggregated communication method do not need to transmit data with other electronic devices, the system architecture may only include the electronic devices running the aggregated communication method (e.g., servers).

[0024] Continue to refer to Figure 2 This shows a schematic diagram of intra-node aggregate communication and inter-node aggregate communication.

[0025] In multi-to-multi communication modes between nodes characterized by small messages, high fan-out, and high latency sensitivity, cross-node links lack native hardware multicast support equivalent to NVLink-SHARP (Scalable Hierarchical Aggregation and Reduction Protocol). Industry-leading upper-layer application libraries such as DeepEP / NCCL (NCCL_EP) degenerate into the same software Top-K cyclic unicast abstraction at the application layer. This means that during Top-K routing, sending a single token still requires constructing K independent WQEs (Work Queue Elements) and ringing the doorbell K times within a warp (thread bundle). This results in the same token data being repeatedly retrieved K times across the PCIe (Peripheral Component Interconnect Express) bus, severely wasting PCIe bus and cross-node network bandwidth.

[0026] Under the unified application layer abstraction between nodes, there are currently two different underlying approaches to cross-node communication in the industry: CPU Proxy: Host API (Application Programming Interface) submission based on CPU (Central Processing Unit) Proxy. The CPU thread handles RDMA (Remote Direct Memory Access) control plane logic through busy polling. In high-concurrency, high-frequency small packet submission scenarios, the multi-stage synchronization overhead and latency across devices (GPU, Host CPU, RNIC (RDMA Network Interface Controller)) account for a larger proportion than the data transmission time on the physical network itself, resulting in low communication efficiency. Simultaneously, the CPU Proxy cannot perceive the SQ (Send Queue) cache status within the network card, leading to cache jitter, which manifests as transmission jitter and tail latency inflation on the physical link.

[0027] GIN / GDAKI: Device API commits initiated directly by the GPU. To reduce control plane latency, the GPU SM directly constructs the WQE (Write-Free Execution Environment) to bypass CPU intervention. Although this increases latency, it erodes GPU computing power.

[0028] Continue to refer to Figure 3The diagram shows the communication flow timing under the CPU Proxy mode. The specific communication process is as follows: Phase 1: GPU prepares data Step 1: The GPU calculates the token data to be transmitted across nodes and writes it into its own video memory HBM (High Bandwidth Memory). Step 2: The GPU sends a communication request to the CPU proxy and places it in the host memory's circular cache queue (ProxyBuffer).

[0029] Phase 2: CPU agent processes packet sending instructions; Step 3: The CPU thread continuously polls the queue to capture communication requests from the GPU; Step 4: The CPU assembles the network card packet sending task order (WQE) in the host memory queue; Step 5: Update the network card doorbell record (the CPU sends doorbell signals to the network card) and mark it as a new task; Step 6: Send a doorbell DB signal to the network card via the PCIe bus to notify the network card that there is a new packet sending task.

[0030] Phase 3: RNIC network card transmits data via RDMA

[0031] Step 7: The network card receives the DB signal and reads the WQE task list from the host memory across PCIe (the high-speed bus channel on the motherboard is the physical channel for communication between the GPU, CPU, and RNIC network card. When one device (CPU / GPU) wants to interact with another piece of hardware (network card / GPU), the data / signals must be transmitted through the PCIe bus, which is called cross-PCIe). Step 8: Relying on GPUDirectRDMA, the raw business data in the GPU memory is read directly without going through the CPU copy. Step 9: Send the data to the remote machine via the IB / RoCE network.

[0032] Phase 4: Transmission Tail-End Synchronization State

[0033] Step 10: After the remote end returns an ACK, the network card writes the transmission completion record (CQE) into the completion queue. Step 11: The CPU continuously polls or receives an interrupt notification, detects a queue update, and continues until transmission is complete; Step 12: The CPU modifies the status flag in the GPU's video memory across PCIe to notify the GPU that this cross-machine communication has ended.

[0034] Continue to refer to Figure 4The diagram shows the communication flow timing under the GIN / GDAKI mode. The specific communication process is as follows: Phase 1: The GPU kernel handles all network card packet sending commands. Step 1: The GPU calculates the token data to be transmitted and stores it in its own video memory (HBM). Step 2: The GPU computing core SM directly generates a packet sending task order WQE and writes it into the network card sending queue WQ. Step 3: The GPU updates the doorbell record DBR and marks it as a new task; Step 4: The GPU sends a doorbell DB signal directly to the network card via PCIe hardware address BAR2 to notify the network card to do its work, without going through the CPU.

[0035] Phase 2: The network card directly pulls data and sends it to the remote end (5-7 steps)

[0036] Step 5: After the network card receives the doorbell, it uses PCIe to read the WQE task from the video memory; Step 6: Relying on GPUDirectDMA, business data is directly retrieved from GPU memory without copying it to CPU memory; Step 7: The network card encapsulates the data packet and sends it to the remote node across machines via IB / RoCE.

[0037] Phase 3: Transmission complete, GPU performs self-detection and finalization (Step 8)

[0038] Step 8: After the remote processing is completed, an ACK is sent to the source. After the source network card receives the ACK, it writes the transmission completion record (CQE) to the GPU memory. The GPU kernel asynchronously detects the completion queue status directly, without CPU polling / interrupt notification, and immediately continues to the next round of model calculation.

[0039] The aforementioned mainstream upper-layer aggregation communication libraries (DeepEP, NCCL / NCCL_EP, etc.) in the MoE (Mixture of Experts) Top-K routing scenario, although nominally intended to deliver the same token to the remote destination nodes where K experts reside, all degenerate into a "software Top-K cyclic unicast" abstraction at the application layer: each GPUWarp, when sending a token, needs to loop K times within the Warp—constructing K independent WQEs and ringing the doorbell K times to the network card's Doorbell register. This leads to: 1. The same token data is repeatedly retrieved by the network card K times across the PCIe bus (each WQE triggers a DMA Read once).

[0040] 2. K independent messages are sent across the network nodes, and the network bandwidth is linearly amplified by a factor of K.

[0041] 3. When K is large (such as Top-8 under EP (Expert Parallelism), both PCIe and RDMA networks become serious bottlenecks, and Doorbell knocking and CQE (Completion Queue Element) processing also bring additional CPU / GPU synchronization overhead.

[0042] Specifically, the architectural limitations of the CPU Proxy cross-node underlying communication solution are mainly reflected in the following three points: (1) The control flow path is lengthy, and the latency accounts for a high proportion in small message scenarios. Taking the distributed inference scenario of MoE large model as an example, in the Top-K dynamic routing (All-to-All) scenario of MoE inference, each GPU needs to communicate frequently with hundreds or thousands of remote peers (nodes) with extremely small packets (1~16KB). The control flow takes a long path of "GPU→host memory→CPU→PCIe→NIC". There is cross-device synchronization. Each commit has inherent overhead at the microsecond level. The latency of cross-device synchronization is much greater than the data transmission time on the physical network itself, which becomes the communication latency bottleneck.

[0043] (2) CPU computing power is eroded, and serial processing has high overhead.

[0044] The CPU Proxy thread maintains the communication state machine by polling 100% of one core, continuously consuming CPU computing power. At the same time, it serially and cyclically constructs WQE for each Peer. Its microsecond-level serial overhead will be amplified exponentially as the cluster size increases.

[0045] (3) The internal SQ Cache state of the network card is not visible, causing link transmission jitter.

[0046] In high-concurrency, high-frequency small packet submission scenarios, the CPU's indiscriminate issuance of WQE (Wide-Queued Escape) can easily cause frequent thrashing of the network card's internal SQ Cache. This frequent thrashing of the hardware-level queue cache directly translates into transmission jitter and latency uncertainty on the physical network link.

[0047] 2. GIN / GDAKI, as a representative solution for squeezing out the lowest latency in large-scale, high-performance communication, (GPU-Initiated Networking / GPU Direct Async Kernel-Initiated) reduces end-to-end communication latency to the microsecond level by allowing the GPU to directly initiate RDMA operations, completely bypassing the host CPU. However, when this solution is deployed in industrial-scale large clusters and faces high-frequency, dynamic, small-message communication scenarios such as MoE (Hybrid Expert Model), its architectural limitations are exposed, mainly in the following two aspects: (1) GPU SM computing power is being continuously eroded The GPU's Streaming Multiprocessor (SM) was originally designed as a dedicated unit for performing large-scale, regularized parallel mathematical operations. However, under the GIN / GDAKI path, GPU threads are forced to divert resources from computational tasks to undertake the full network protocol stack control plane work, including but not limited to: Construct and package the WQE (Work Queue Element), and complete field filling and byte order conversion; Maintain and update communication status flags (PSN, credit, ring index, etc.); Execute MMIO to write to the doorbell register of the network card; Insert system-level memory barriers to ensure PCIe end-to-end visibility; Poll the CQE to confirm remote completion.

[0048] These task instruction patterns are irregular, with multiple control flow branches and a severe misalignment between memory access patterns and computation, which runs counter to the SIMT data parallelism mode that SM excels at. In scenarios like MoE with small messages and Top-K high fan-out, each token needs to loop K times to construct WQE and trigger the doorbell. The proportion of control plane instructions increases sharply, and the SM computing power that should be used for Attention / FFN / Expert computation is continuously eroded by the communication protocol stack, resulting in a significant decrease in overall computational throughput.

[0049] (2) The CUDA (Compute Unified Device Architecture) kernel lacks network anomaly fault tolerance.

[0050] Unlike host drivers with full interrupt and scheduling support, the CUDA kernel cannot handle network anomalies such as packet loss, timeouts, or connection failures. In the GIN scheme, GPU threads that directly poll the network card's CQE cannot trigger soft interrupts for anomaly recovery when encountering network anomalies, nor can they actively relinquish SM resources. This makes them highly susceptible to indefinite busy-waiting, which can cause the entire warp and its associated CUDA stream to hang permanently.

[0051] Continue to refer to Figure 5 The diagram shows the system framework of the collective communication system. Please refer to it. Figure 6 , Figure 6 A flowchart illustrating a collection communication method provided in this embodiment of the disclosure. The process 600, applied to the source node, includes the following steps: Step 601: Generate a collection communication task through the source computing unit in the source node, and transmit the collection communication task to the source network card of the source node.

[0052] In this embodiment, the execution subject of the collection communication method (e.g., Figure 1 The server in the process can generate a collection communication task through the source computing unit in the source node, and then transmit the collection communication task to the source network card of the source node. The collection communication task includes the data to be processed and the object representation data of multiple processing objects of the data to be processed.

[0053] The distinction between source and destination nodes is not an inherent attribute of the computing nodes themselves, but a logical distinction based on the role played by the data flow in the collective communication process. Specifically, the source node is the node responsible for initiating communication and sending data to be processed in the current collective communication task; the destination node is the node responsible for receiving data to be processed and performing corresponding processing on that data in the current collective communication task.

[0054] It is important to note that the distinction between source and destination nodes is relative and dynamic. In different communication stages or different ensemble communication tasks within the same distributed system, the same physical node may act as both a source and a destination node. For example, in the All-Reduce operation of distributed training, each node needs to both send local gradient data to other nodes (in this case, the node is the source node) and receive gradient data from other nodes and reduce it (in this case, the node is the destination node). Therefore, source and destination nodes are merely logical designations used to describe the data flow during ensemble communication and should not be interpreted as fixed limitations on node hardware or functionality.

[0055] Collective communication is a communication pattern in distributed computing where multiple computing nodes coordinate data exchange. Its key characteristic is that the participants are a group of nodes, rather than simple point-to-point communication. Besides the MoE distributed inference scenario, collective communication tasks are also widely present in the following scenarios: All-Reduce in distributed deep learning training: In data-parallel training, each computing node independently completes a mini-batch of forward and backward propagation. After obtaining the local gradient, the All-Reduce operation is used to sum or average the gradients of all nodes, so that each node finally obtains a globally consistent gradient for parameter updates. In this process, each node is both a data sender (source node) and a data receiver (destination node).

[0056] All-Gather in distributed training: In model-parallel or pipeline-parallel scenarios, each node holds a portion of the model's parameters or intermediate activation values. All-Gather operations are needed to collect the data from all nodes and distribute it to all participating nodes to complete the full forward or backward computation.

[0057] Broadcasting in distributed training: During the training initialization phase, it is typically necessary to broadcast the model parameters or optimizer state from one node (such as the master node) to all other worker nodes, ensuring that all nodes receive consistent initial parameters. In this case, the master node is the source node, and the remaining worker nodes are the destination nodes.

[0058] Reduce-Scatter in distributed training: Each node first reduces its local data (e.g., by summing or finding the maximum value), then distributes the reduced results in shards to other nodes, with each node receiving only a portion of the result. This pattern is often used for gradient sharding and reduction in training large models.

[0059] Parallel numerical simulation in high-performance computing: In scientific computing scenarios such as weather forecasting and fluid dynamics simulation, the computational domain is divided into multiple subdomains and assigned to different nodes. Each node needs to exchange boundary data through aggregate communication at each time step to ensure the consistency of global computation.

[0060] A network interface card (NIC), also known as a network adapter or network interface controller, is a hardware component in a computing node responsible for connecting the computing unit to an external network. Its core functions include: Data encapsulation and decapsulation: Encapsulate the data output by the computing unit into network packets according to network protocols (such as Ethernet protocol, InfiniBand protocol, etc.) and send them, or decapsulate the received network packets and deliver them to the computing unit.

[0061] Data transmission control: manages the data sending and receiving process, including flow control, congestion control, error detection and retransmission.

[0062] Address resolution and routing: Determine the next-hop path of data based on the target address to ensure the correct delivery of data in the network.

[0063] In this embodiment, the network card involved is a smart network card that supports RDMA (Remote Direct Memory Access) capability, namely an RDMA network interface controller (RNIC).

[0064] The source computing unit and the source network interface card (NIC) are physically connected via a system bus (such as a PCIe bus), and logically constitute a collaborative relationship between task producer and task executor. The source computing unit is the initiator and orchestrator of collective communication tasks. It is responsible for generating collective communication tasks based on the needs of upper-layer applications (such as communication primitive calls in distributed training, token routing decisions in MoE inference, etc.), specifying the content of the data to be processed, the object representation data of the objects to be processed, and the communication semantics involved in the communication task.

[0065] The source network interface card (NIC) is the executor and offloading engine for aggregated communication tasks. After receiving aggregated communication tasks from the source computing unit, it is responsible for parsing the task content, determining the destination node of the data, reading the data to be processed from the memory of the source computing unit, and sending it to the corresponding destination node through the network.

[0066] The source computing unit writes the collection communication task into the memory area accessible by the source network card (e.g., through memory-mapped I / O) and notifies the source network card of the arrival of a new task through a doorbell mechanism. After receiving the notification, the source network card reads the task content from memory and executes it autonomously. After execution, it reports the completion status to the source computing unit through an interrupt or polling mechanism.

[0067] As an example, in the All-Reduce communication scenario based on distributed training, the source node is a worker node in the distributed deep learning training cluster, the source computing unit is the GPU in the worker node, and the source network card is the RNIC that supports RDMA capability in the host where the GPU is located.

[0068] After performing backpropagation, the source computation unit obtains its local gradient data, which is the data to be processed. Simultaneously, based on the distributed training topology, the source computation unit determines the identity information of each reduction computation unit participating in this All-Reduce operation. This identity information is the object representation data, and each identity corresponds to a processing object (i.e., a reduction computation unit responsible for performing reduction operations on the gradient data, such as the GPUs on participating nodes). The source computation unit organizes the data to be processed (local gradients) and the object representation data (the identity information of each reduction computation unit) into a collective communication task.

[0069] The source computing unit writes the aforementioned aggregate communication task into the task submission area of ​​the system memory accessible by the source network card via the PCIe bus, and notifies the source network card of the arrival of a new task by writing to the doorbell register. After receiving the notification, the source network card reads the aggregate communication task from the system memory, obtains the data to be processed and the object representation data, and prepares for the subsequent transmission of gradient data to the destination nodes where the processing objects are located.

[0070] In this example, after the source computation unit completes the generation and submission of the set communication task, it can continue to perform other computational work (such as the forward computation of the next iteration) without waiting for the gradient data transmission and reduction to complete. The source network card autonomously completes the network transmission of data in the background, thus achieving the overlap of communication and computation.

[0071] As another example, in the Broadcast communication scenario based on distributed inference, in this embodiment, the source node is the parameter server node in the distributed inference system, the source computing unit is the HostCPU in the parameter server, and the source network card is the RNIC that supports RDMA capability in the server where the Host CPU is located.

[0072] During the inference service startup phase, the source compute unit loads the parameter file of the large language model from persistent storage; these model parameters constitute the data to be processed. Simultaneously, based on the deployment configuration of the inference cluster, the source compute unit determines the identity information of all inference compute units that need to receive and load the model parameters. This identity information represents the object representation data, and each identity corresponds to a processing object (i.e., an inference compute unit responsible for loading and running the model parameters to perform inference computation, such as the GPUs on each worker node). The source compute unit organizes the data to be processed (model parameters) and the object representation data (the identity information of each inference compute unit) into a set communication task.

[0073] The source computing unit writes the aforementioned collective communication task into the memory area corresponding to the submission queue of the source network interface card (NIC) via memory-mapped I / O, and notifies the source NIC of the arrival of a new task by updating the producer pointer. Upon receiving the notification, the source NIC reads the collective communication task from system memory, parses the storage address of the data to be processed in system memory and the object representation data of each processing object, preparing for the subsequent broadcast of model parameters to the destination nodes where each processing object resides.

[0074] In this example, after the source computing unit completes the generation and submission of the ensemble communication task, it can continue to perform other management tasks (such as receiving inference requests from clients and scheduling inference tasks) without blocking and waiting for the broadcast of model parameters to complete. The source network card autonomously completes the network transmission of data in the background, thereby realizing the communication offloading during the parameter loading phase.

[0075] In some optional implementations of this embodiment, the collective communication task is stored in the task cache queue of the source node. The task cache queue is a circular buffer in the memory of the source network card used to temporarily store collective communication tasks, corresponding to the circular buffer of the CCD (Collective Communication Descriptor) in the hardware architecture. This circular buffer is shared by the source computing unit and the source network card. Through the coordinated movement of consumer and producer pointers, lock-free asynchronous handover of collective communication tasks between the computing and communication sides is achieved, serving as the source of tasks for the source network card to autonomously execute data transmission.

[0076] In this implementation, the aforementioned execution entity collects communication tasks in the following manner: in response to the fact that the difference between the consumer pointer and the producer pointer corresponding to the task cache queue does not reach the length value of the task cache queue, a collection communication task is generated through the source computing unit, wherein the consumer pointer is a pointer to the processed tasks in the task cache queue, and the producer pointer is a pointer to the unprocessed tasks in the task cache queue.

[0077] In this implementation, the aforementioned execution entity can also perform the following operations: add the collection communication task to the task cache queue and update the producer pointer.

[0078] As an example, before generating a set communication task, the source computing unit first reads the current consumer pointer and producer pointer of the task cache queue and calculates the difference between them. When the difference does not reach the length of the task cache queue, it indicates that there is a free slot in the task cache queue. Based on this, the source computing unit generates a set communication task and writes the set communication task into the task cache queue slot currently pointed to by the producer pointer.

[0079] After the collection communication task is written, the source computing unit performs an update operation on the producer pointer. Specifically, the source computing unit moves the producer pointer one slot backward along the circular direction of the task cache queue, so that the producer pointer points to the next free slot, thereby marking that the collection communication task has been submitted to the task cache queue. If the producer pointer moves to the end of the task cache queue, it is wrapped back to the beginning of the task cache queue to maintain the continuous writing capability of the circular buffer.

[0080] The consumer pointer on the source network interface card (NIC) side is maintained independently by the source NIC. After the source NIC completes processing a set communication task in the task buffer queue, it moves the consumer pointer one slot backward along the same circular direction to mark that the set communication task has been consumed. By using the difference between the producer and consumer pointers, the source computing unit and the source NIC can lock-free determine the idle state of the task buffer queue and the number of tasks to be processed, thus enabling asynchronous submission and processing of set communication tasks.

[0081] In this implementation, the source computing unit determines the idle status of the task cache queue by the difference between the producer pointer and the consumer pointer, thereby realizing lock-free asynchronous task handover between the computing side and the communication side. This avoids lock contention and blocking waiting during the task submission process, reduces the communication synchronization overhead of the source computing unit, and improves the submission efficiency of the collection communication task and the autonomous processing capability of the source network card.

[0082] In some optional implementations of this embodiment, the execution entity may also perform the following operation: in response to receiving feedback information from the destination nodes of all objects to be processed in the aggregate communication task through the source network card, update the consumer pointer.

[0083] Feedback information indicates that the data to be processed has been transmitted to the storage area of ​​the target computing unit corresponding to the feedback information. The feedback information is a communication primitive-level acknowledgment signal sent by the destination node to the source network interface card (NIC) after confirming that the data to be processed has been securely written to the target computing unit's storage area. This feedback information not only includes a completion status identifier indicating that the transmission is complete, but also carries a task identifier or sequence number corresponding to a specific set of communication tasks in the source node's task cache queue, as well as verification information confirming that the data has been successfully written to the target computing unit's storage area. By parsing the task identifier in this feedback information, the source NIC can accurately locate the corresponding processed set of communication tasks in the task cache queue, thereby providing accurate addressing basis for subsequent updates to the consumer pointer and ensuring reliable synchronous updates of task consumption status and data consistency.

[0084] As an example, during the continuous reception of feedback information from various destination nodes, the source network interface card (NIC) parses each received feedback message and extracts the task identifier and completion status identifier carried within it. Based on the task identifier, the source NIC locates the corresponding aggregate communication task slot in the task cache queue and marks the aggregate communication task as confirmed as completed.

[0085] The source network interface card (NIC) continuously monitors the reception status of feedback information from the destination nodes of all objects corresponding to the current aggregated communication task. When it is detected that all destination nodes corresponding to the aggregated communication task have returned feedback information, and the completion status flags in all feedback information indicate that the data to be processed has been successfully transmitted to the storage area of ​​the corresponding target computing unit, the source NIC determines that the aggregated communication task has been completely processed.

[0086] Subsequently, the source NIC performs a consumer pointer update operation. The source NIC reads the task buffer queue slot currently pointed to by the consumer pointer and confirms that the corresponding aggregate communication task is one that has been fully processed. The source NIC moves the consumer pointer one slot backward along the circular direction of the task buffer queue, making the consumer pointer point to the next pending aggregate communication task slot, thus marking that the fully processed aggregate communication task has been consumed. If the consumer pointer moves to the end of the task buffer queue, it wraps back to the beginning of the task buffer queue to maintain the continuous consumption capability of the circular buffer. Through this consumer pointer update mechanism based on feedback information, the source NIC achieves accurate release of processed tasks in the task buffer queue and timely reclamation of idle slots.

[0087] In this implementation, the source network card accurately updates the consumer pointer based on the feedback information from the destination node, which realizes the timely release of processed tasks and the reclamation of idle slots in the task cache queue. This avoids queue overflow and blocking caused by task backlog, reduces the storage occupation of the task cache queue and the waiting delay of the source computing unit, and improves the processing throughput of the collective communication task and the overall operating efficiency of the system.

[0088] Step 602: Using the source network interface card, determine the destination node of each of the multiple processing objects based on the object representation data and routing mapping relationship.

[0089] In this embodiment, the aforementioned execution entity generates a collective communication task through the source computing unit in the source node and transmits the collective communication task to the source network interface card of the source node. The routing mapping relationship represents the correspondence between the data and the destination node where the processing object resides.

[0090] Continue to refer to Figure 7The diagram illustrates the routing mapping relationship. In the MoE distributed inference scenario, the routing mapping relationship is represented by an expert forwarding table stored in the static random access memory on the source network card. The expert model identifier is the object representation data, uniquely representing a processing object, which is an expert model in the global expert set. The forwarding table entry contains the destination node identifier (number), representing the destination node where the expert model is deployed. The number of entries in the expert forwarding table equals the total number of global experts, ensuring that any expert identifier can be used to find the corresponding destination node identifier. The expert forwarding table is searched using the Top-K routing results output by the gating network. The source network card performs a search using the selected expert identifiers as keys. Each search returns K forwarding table entries equal to the number of Top-K entries, and the destination node identifier is read from the matched entries. Therefore, the object representation data, mapped through the routing mapping relationship, represents the destination node where the processing object resides, thus distinguishing the processing object from the destination node.

[0091] As an example, in a scenario based on MoE expert parallel routing distribution, the source node is the computing node that deploys the MoE model, the source computing unit is the graphics processor that runs the gated network and some expert models, and the source network card is a network card with remote direct memory access capability.

[0092] After completing the gating calculation, the source computing unit obtains the Top-K routing results. The expert identifiers of the selected K expert models constitute the object representation data. Each expert identifier corresponds to a processing object, which is the selected expert model for expert computation. The source computing unit organizes the data to be processed and the object representation data into a set communication task and transmits it to the source network interface card (NIC). The source NIC parses the object representation data and uses the K expert identifiers as keys to perform a lookup of the routing mapping relationship. Each lookup returns K forwarding table entries that correspond one-to-one with the K expert identifiers. The source NIC reads the destination node identifier from each matched entry. This destination node identifier is the destination node where the corresponding processing object resides, thus determining the destination node for each of the K processing objects. When two or more expert identifiers in the object representation data correspond to the same destination node identifier, the source NIC determines that multiple processing objects are deployed on that destination node. Simultaneously with reading the destination node identifier, the source NIC also reads the local expert index from each forwarding table entry, which, along with the destination node identifier, serves as information for subsequent addressing to the destination node.

[0093] As another example, in the ensemble communication scenario based on distributed training gradient reduction, the source node is the worker node in the distributed training cluster, the source computing unit is the graphics processor that performs model training, and the source network card is a network card with remote direct memory access capability. Its local memory maintains a routing mapping relationship, which is represented as a communication group member table, recording the correspondence between the identity of each computing unit participating in this gradient reduction and the network address of the node where it is located.

[0094] After backpropagation, the source computing unit obtains local gradient data as the data to be processed. Based on the training topology configuration, it determines the identity identifiers of each computing unit participating in this reduction as object representation data. These two are organized into a collective communication task and transmitted to the source network interface card (NIC). The source NIC parses the object representation data, where each identity identifier corresponds to a processing object, which is the computing unit responsible for performing the reduction operation. Using each identity identifier in the object representation data as a key, the source NIC performs a lookup on the routing mapping relationship represented by the communication group member table, retrieving matching member table entries and reading their network addresses. The node pointed to by this network address is the destination node where the corresponding processing object resides. Through a sequential lookup of all identity identifiers, the source NIC obtains a set of correspondences from processing objects to destination nodes, with each destination node uniquely identified by its network address. When multiple identity identifiers in the object representation data correspond to the same network address, the source NIC determines that multiple processing objects reside on that destination node. Therefore, the source NIC determines the destination nodes of multiple processing objects based on the object representation data and the routing mapping relationship.

[0095] Step 603: Transmit the data to be processed and the distribution identifier of the object to be processed to the destination node where the object is located via the source network card.

[0096] In this embodiment, the aforementioned execution entity transmits the data to be processed and the distribution identifier of the processing object to the destination node where the processing object is located through the source network card.

[0097] The distribution identifier is a lightweight metadata that serves as a hardware-level distribution instruction for the destination node, instructing the destination network interface card to distribute the received data to which specific computing units (such as specific expert models) within the node for processing.

[0098] A mask is an efficient and concrete implementation of distribution identifiers. Assuming a destination node deploys 8 expert models (numbered 0-7), the model mask can be an 8-bit binary number, with each bit corresponding to one expert model. For example, a mask 00001010 indicates that received data needs to be distributed to expert models 1 and 3 for processing; a mask 10000001 indicates that data needs to be distributed to expert models 0 and 7.

[0099] As an example, in a scenario based on MoE expert parallel routing distribution, for each processing object represented by the object representation data, the source NIC reads the data to be processed corresponding to that processing object from the local memory of the destination node where the processing object is located. This data to be processed is the token data that needs to be passed to the processing object for expert calculation after gating computation. The source NIC writes the data to be processed and the distribution identifier corresponding to the processing object to the pre-allocated receiving storage area of ​​the destination node through a remote direct memory access write operation. The data to be processed is written to the data area of ​​the receiving storage area, and the distribution identifier is written to the description area associated with the data area. The two maintain a correspondence in the receiving storage area.

[0100] After the network interface card (NIC) of the destination node completes the write operation, it delivers the data to be processed in the receive storage area to the processing object (expert model) represented by the distribution identifier within the destination node. Based on this, the processing object retrieves the data to be processed from the receive storage area and performs expert calculations on the data.

[0101] As another example, in a ensemble communication scenario based on distributed training gradient reduction, for the processing object represented by the object representation data, the source network interface card (NIC) encapsulates the data to be processed corresponding to the processing object and the distribution identifier of the processing object into a message according to the destination node where the processing object is located. The data to be processed is the local gradient data generated by the source computing unit after backpropagation computation. The source NIC delivers this message to the receive queue of the destination node NIC through a remote direct memory access (RDBMI) send operation.

[0102] The destination node's network interface card (NIC) retrieves the message from the receive queue, writes the data to be processed into the destination node's local memory, and generates a receive completion event to notify the corresponding processing object within the destination node. In response to the receive completion event, the processing object retrieves the data to be processed from the destination node's local memory and uses it, along with the distribution identifier, for subsequent reduction operations.

[0103] In some optional implementations of this embodiment, the execution entity can perform step 603 in the following manner to complete the data transmission process: The first step is to determine the connection identifier of the network connection between the source computing unit and the destination node based on the node connection mapping relationship for each destination node. The node connection mapping relationship represents the connection identifier of the network connection between the source computing unit and the destination node.

[0104] Continue to refer to Figure 8This diagram illustrates the node connection mapping relationship. A network connection refers to an end-to-end communication channel established between a source node and a destination node based on the Remote Direct Memory Access Protocol (RDP). This communication channel is carried at the network interface card (NIC) hardware level by a pair of related queue pairs (QPs). The source queue pair and the destination queue pair are paired together. The source queue pair contains a work queue for submitting send requests, and the destination queue pair contains a work queue for receiving receive requests. Together, they constitute a one-way data delivery path from the source node to the destination node.

[0105] Accordingly, the connection identifier is a queue pair number used to uniquely identify the queue pair within the source network interface card (NIC). The source NIC can locate the corresponding queue pair and the network connection it represents using this queue pair number. The work queue is the work queue contained within the queue pair that carries requests from the sending side; that is, the send work queue (SQ) in the source queue pair. Work queue entries are WQEs (Work Queue Elements) submitted to the work queue. Each work queue entry describes a data transmission operation to be executed, and at least records the connection identifier and the storage address of the data to be processed.

[0106] The source network card uses the destination node as an index to look up the table in the node connection mapping relationship to obtain the connection identifier of the network connection between the source computing unit and the destination node, that is, to obtain the queue pair number of the queue pair carrying the network connection.

[0107] The second step involves combining the connection identifier and the storage address of the data to be processed in the source computing unit to generate and add work queue entries to the work queue corresponding to the network connection.

[0108] The source network interface card (NIC) obtains the storage address of the data to be processed, which needs to be transmitted to the destination node, within the source computing unit. This storage address points to the starting position of the data to be processed in the local storage area of ​​the source computing unit and may include the length information of the data. The source NIC combines this storage address with the connection identifier obtained in the previous step to generate a work queue entry, ensuring that the work queue entry carries both the connection identifier and the storage address. Subsequently, the source NIC appends this work queue entry to the work queue of the network connection corresponding to the connection identifier, i.e., appends it to the tail of the send work queue of that queue pair, and updates the producer index of the work queue to indicate to the source NIC's sending engine the existence of a work queue entry to be processed.

[0109] The third step involves using the network connection corresponding to the connection identifier in the work queue entry to transmit the data to be processed at the storage address in the work queue entry and the distribution identifier of the processing object of the data to be processed in the destination node to the destination node.

[0110] The source network interface card's (NIC) sending engine sequentially retrieves work queue entries from the work queue, reads the connection identifier within each entry, and locates the corresponding network connection—that is, the corresponding queue pair—based on the connection identifier. The source NIC then reads the data to be processed from the corresponding storage location of the source computing unit according to the storage address in the work queue entry. Simultaneously, the source NIC obtains a distribution identifier corresponding to the processing object of the data to be processed in the destination node. Using the network connection corresponding to the connection identifier, the source NIC encapsulates the read data to be processed and the obtained distribution identifier into a network packet, and sends it to the destination node via the network connection represented by the queue pair. Upon receiving the network packet, the destination node's NIC delivers the data to be processed to the processing object represented by the distribution identifier within the destination node, so that the processing object can perform the corresponding processing on the data.

[0111] In this implementation, the source network card can autonomously complete data transmission based on the connection identifier and storage address in the work queue entry, without the source computing unit needing to intervene in the transmission process step by step. This reduces the communication overhead of the source computing unit and improves data transmission efficiency.

[0112] In some optional implementations of this embodiment, the execution entity can perform the third step described above in the following manner to transmit the data to be processed to the target node: in response to the network connection in the work queue being non-empty including an active connection whose connection context is cached by the source network card, the active connection is used to transmit the data to be processed at the storage address in the work queue entry of the active connection and the distribution identifier of the processing object of the data to be processed in the target node corresponding to the active connection to the target node of the active connection.

[0113] A connection context is a set of key parameters and resources cached by the source network interface card (NIC) to maintain the network connection with the destination node. It forms the foundation of the NIC hardware's operational environment for data transmission. In the Remote Direct Memory Access (RDA) protocol, the connection context specifically includes queue pair numbers, local and remote memory registration keys, sequence numbers, flow control window size, retransmission timer status, and key exchange information. For example, when a source NIC establishes a RDA connection with a destination node, both parties need to complete memory registration, key exchange, and parameter negotiation. The source NIC caches the negotiated queue pair numbers, memory keys, and flow control status in its on-chip static random access memory (SRAM). This cached information set constitutes the connection context. Subsequently, when the source NIC sends data to the destination node, it can directly invoke this connection context to locate the remote memory address, verify access permissions, and manage transmission flow control without renegotiation.

[0114] In this implementation, active connections whose connection contexts are cached by the source network interface card (NIC) are prioritized. The core reason for this is to avoid the hardware overhead and latency caused by connection context switching. If the source NIC does not currently cache the connection context of a network connection, it must load that context from external memory or RAM into the on-chip cache before sending data. This loading process involves additional memory access operations and state recovery, significantly increasing first-byte latency and consuming on-chip cache bandwidth. However, the connection context of active connections already resides in the source NIC's on-chip cache, allowing the NIC to directly read and execute the transmission operation without additional context loading and switching overhead. In aggregated communication tasks, multiple work queue entries often correspond to a few destination nodes. Prioritizing the reuse of active connections allows the source NIC to maintain context resident during continuous transmission, reducing the number of context switches and thus improving the throughput and response speed of the NIC's transmission engine.

[0115] In this implementation, after determining each destination node and its corresponding network connection, the source network interface card (NIC) first iterates through all non-empty network connections in the work queue to identify active connections whose connection contexts have been cached by the source NIC. For each active connection, the source NIC sequentially reads the work queue entries from the work queue corresponding to that active connection, obtaining the storage address and connection identifier recorded in the entry. Based on the connection identifier, the source NIC confirms the active connection to which the entry belongs and directly calls the cached connection context for that active connection, without needing to perform a context loading operation.

[0116] The source network interface card (NIC) reads the data to be processed from the corresponding storage location of the source computing unit based on the storage address in the work queue entry. Simultaneously, the source NIC obtains the distribution identifier corresponding to the processing object within the destination node, based on the destination node corresponding to the active connection. The source NIC encapsulates the read data to be processed and the distribution identifier into a network packet and sends the network packet to the destination node corresponding to the active connection using the network connection represented by the active connection. Upon receiving the packet, the destination node NIC delivers the data to be processed and the distribution identifier to the corresponding processing object.

[0117] The source network interface card (NIC) continuously reads subsequent work queue entries from the work queue of the active connection and repeats the process of reading data, obtaining the distribution identifier, encapsulating packets, and sending them until the work queue corresponding to the active connection is empty, or the source NIC determines that it needs to switch to another active connection. For the remaining active connections, the source NIC performs the same operation until all work queues corresponding to active connections have been processed. During this process, the source NIC always prioritizes processing work queue entries corresponding to active connections; it generally only considers processing inactive connections whose connection contexts are not cached when the active connection work queue is empty.

[0118] In this implementation, the source network card prioritizes sending data using the active connection with the cached connection context, avoiding the additional overhead of context loading and switching, reducing first-byte latency and on-chip cache bandwidth usage, and improving the data transmission throughput efficiency of the collective communication task.

[0119] In some optional implementations of this embodiment, the execution entity performs the transmission process of the data to be processed in the following manner: First, it generates a sending message by combining the data to be processed at the storage address in the work queue entry corresponding to the active connection and the distribution identifier of each processing object of the data to be processed in the destination node; then, it transmits the sending message to the destination node using the active connection.

[0120] As an example, firstly, based on the storage address recorded in the work queue entry corresponding to the active connection, the data to be processed is read from the corresponding storage location of the source computing unit. Simultaneously, the source network interface card (NIC) obtains the distribution identifier corresponding to each processing object of the data to be processed in the destination node, based on the destination node corresponding to the active connection.

[0121] Subsequently, the source network interface card (NIC) combines and encapsulates the read data to be processed with the distribution identifiers of each processing object to generate a transmission packet. Specifically, the source NIC places the data to be processed in the data payload area of ​​the transmission packet and places the distribution identifiers of each processing object in the header description area or dedicated metadata area of ​​the transmission packet. This ensures a clear correspondence between the data to be processed and the distribution identifiers of each processing object in the transmission packet. The NIC also adds necessary header information according to the network protocol format corresponding to the active connection, completing the generation of the transmission packet. In some implementations, the distribution identifiers of each processing object in the data to be processed in the destination node can be represented by the mask or index of each processing object.

[0122] After generating the transmission packet, the source network interface card (NIC) invokes the cached connection context of the active connection and delivers the transmission packet to the network connection represented by the active connection, transmitting it to the destination node corresponding to the active connection. Upon receiving the transmission packet, the destination node NIC extracts the data to be processed from the packet's payload area, extracts the distribution identifier of each processing object from the packet's description area or metadata area, and determines and transmits the data to be processed to the corresponding processing object within the destination node based on the distribution identifier, so that the processing object can perform the appropriate processing on the data.

[0123] In this implementation, the source network card encapsulates the data to be processed and the distribution identifier into a transmission message and transmits them in a unified manner. This avoids the matching overhead and out-of-order risk at the receiving end caused by sending the two separately, reduces the data reassembly complexity at the destination node, and improves the data transmission reliability and processing efficiency of the collective communication task.

[0124] In some optional implementations of this embodiment, the execution entity can transmit data based on the active connection in the following way: using the active connection, continuously transmitting the sending messages corresponding to each work queue entry of the active connection's work queue to the destination node.

[0125] After determining that an active connection will be used for data transmission, the source network interface card (NIC) performs a continuous transmission operation for the work queue corresponding to that active connection. Starting from the head of the work queue, the source NIC sequentially reads the first work queue entry, reads the data to be processed according to the storage address in the entry, and combines it with the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the active connection to generate a transmission packet. Subsequently, the source NIC uses the active connection to transmit the transmission packet to the destination node.

[0126] After sending the first work queue entry, the source network interface card (NIC) does not interrupt the sending process of the active connection. It immediately reads the second work queue entry from the work queue and repeats the steps of reading the data to be processed, generating a send packet by combining the distribution identifier, and transmitting the send packet using the active connection. In this manner, the source NIC continues to sequentially read all subsequent work queue entries in the work queue of the active connection and performs the same sending operation for each work queue entry until all work queue entries in the work queue have been processed.

[0127] Throughout the continuous transmission process, the source network interface card (NIC) always uses the same active connection, without switching to other network connections or interrupting the transmission pipeline of that active connection. While sending the transmission packet corresponding to the previous work queue entry, the source NIC can simultaneously read the storage address of the next work queue entry and prefetch the data to be processed. This allows the reading of the data to be processed, the generation of the transmission packet, and the transmission of the network packet to overlap in time, achieving continuous and uninterrupted transmission of the transmission packets corresponding to each work queue entry in the active connection's work queue.

[0128] In this implementation, the source network card continuously sends each packet in the work queue in the same active connection order, avoiding pipeline pauses caused by connection switching and transmission interruption, reducing context switching overhead and latency of active connections, and improving the data transmission throughput efficiency of the collective communication task.

[0129] In some optional implementations of this embodiment, the execution entity may also perform the following operations: First, in response to the fact that the number of active connections with cached connection contexts in the source network card does not reach a preset threshold, the connection context of inactive connections with non-empty work queues is loaded and cached; then, using inactive connections, the data to be processed at the storage address in the work queue entry of the inactive connection's work queue, and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the inactive connection are transmitted to the destination node of the inactive connection.

[0130] The preset quantity threshold refers to the maximum number of connection contexts that are allowed to reside simultaneously in the cache on the source network card. This threshold is determined by the on-chip storage capacity and concurrent processing capability of the source network card and is used to limit the number of active connections to prevent excessive occupation of on-chip resources by connection contexts, which could lead to cache overflow or performance degradation.

[0131] As an example, the source network interface card (NIC) continuously monitors the number of active connections with cached connection contexts in real time while processing work queue entries corresponding to active connections. When it detects that the number of active connections has not reached a preset threshold and there are inactive connections in the work queue that are not empty, the source NIC triggers the connection context loading mechanism.

[0132] The source network interface card (NIC) reads the connection context of the inactive connection (which is not empty) from external memory or system memory and loads it into the NIC's on-chip cache, thus turning the inactive connection into an active connection. After loading, the source NIC updates the active connection count and immediately uses the newly active network connection to read the work queue entry from its work queue. Based on the storage address in the entry, it retrieves the data to be processed, combines this data with the distribution identifier of the processing object in the destination node corresponding to the connection, generates a transmission packet, and transmits the transmission packet to the corresponding destination node.

[0133] The source network interface card (NIC) continuously performs the aforementioned monitoring and loading operations. Whenever the number of active connections falls below a preset threshold and there are pending inactive connections, a new connection context is loaded and data transmission for the corresponding connection is initiated. When the number of active connections reaches the preset threshold, the source NIC stops loading new connection contexts and only maintains data transmission for active connections with currently cached connection contexts until the work queue for active connections is completed and the connection context is released. Then, based on the remaining capacity, it decides whether to load new connection contexts for inactive connections.

[0134] In this implementation, the source network card dynamically loads the connection context of inactive connections when the number of active connections has not reached the threshold, so that more pending connections can be transmitted in parallel, avoiding transmission blocking caused by inactive connections waiting, and improving the overall concurrent transmission efficiency and bandwidth utilization of the collective communication task.

[0135] In some implementations, this hardware architecture possesses real-time awareness of the producer index and GFD (GetFrom Device) queue status. The system can predict the next queue pair (QP) to be served based on scheduling logic and proactively initiate a prefetch operation for the context data of the next QP before the current QP's processing task is completed. This mechanism aims to utilize the window period during which the computing unit executes the current task to perform data loading in parallel, thereby effectively masking the access latency caused by PCIe bus round trips.

[0136] In some optional implementations of this embodiment, the processing object is the expert model in the hybrid expert model, and the source node is the computing node where the gating network in the hybrid expert model is located.

[0137] The source node is configured to host a gating network within the hybrid expert model. This gating network receives input data and calculates routing weights based on the features of the input data to determine which expert model should be activated for the current input data. Correspondingly, the processing objects are specifically defined as the individual expert models within the hybrid expert model, which are distributed across different destination nodes to perform specific forward computation tasks on the input data.

[0138] When the source computing unit generates a collective communication task, this collective communication task is specifically an expert route distribution task triggered by a gating network. The source computing unit takes the input data as the data to be processed and identifies one or more expert models pointed to by the route weights calculated by the gating network as the objects to be processed. The source computing unit encapsulates the identification information of these expert models into distribution identifiers and submits the collective communication task containing the data to be processed and the distribution identifiers to the task cache queue.

[0139] After retrieving the set of communication tasks from the task cache queue, the source network interface card (NIC) parses the expert model identifier contained in the distribution identifier, and then determines the destination node carrying these expert models. The source NIC establishes active connections with each destination node, distributing and transmitting the data to be processed to the corresponding destination node where the expert model resides. Upon receiving the data, the expert models on each destination node independently process the data using their respective model parameters, thereby achieving distributed parallel inference of hybrid expert models.

[0140] In this implementation, the processing object is concretized as an expert model in the hybrid expert model, and the source node where the gating network is located initiates the collection communication task. This achieves efficient decoupling and data distribution between gating routing decision and expert computation execution, reduces cross-node communication latency during model inference, and improves the inference throughput and computing resource utilization of the hybrid expert model in a distributed environment.

[0141] Continue to refer to Figure 9 , Figure 9 A flowchart of a collection communication method provided in an embodiment of this disclosure. The flowchart 900, applied to the destination node, includes the following steps: Step 901: Obtain the data to be processed sent by the source network card of the source node and the distribution identifier of the processing object of the data to be processed through the destination network card of the destination node.

[0142] In this embodiment, the execution subject of the collection communication method (e.g.) Figure 1 The server in the process obtains the data to be processed and the distribution identifier of the processing object of the data to be processed from the source network card of the source node through the destination network card of the destination node.

[0143] The destination network interface card (NIC) refers to an intelligent network interface device deployed on the destination node. It has remote direct memory access capabilities and can independently parse received network packets and execute data transmission tasks. In this embodiment, the destination NIC is responsible for receiving packets containing data to be processed and a distribution identifier sent by the source NIC, parsing the distribution identifier to determine the target storage location of the data to be processed within the destination node, and directly writing the data to be processed to that target storage location. This eliminates the need for the destination computing unit in the destination node to participate in data transfer, thereby achieving decoupling between communication and computing.

[0144] As an example, in a scenario based on MoE expert parallel routing distribution, the processing object is specifically the expert model distributed across different computing nodes in a hybrid expert model, and the destination node is the computing node carrying a specific expert model. The destination network interface card (NIC) receives data packets sent by the source NIC of the source node through an active connection. These data packets encapsulate the data to be processed (i.e., the input token feature vector selected by the gating network) and the distribution identifier (i.e., the identifier of the target expert model and its corresponding memory offset).

[0145] The destination network interface card (NIC) parses the distribution identifier, identifies the target expert model to which the data to be processed needs to be delivered, and determines the input buffer address of the expert model in the destination node's storage area based on the video memory offset. Subsequently, the destination NIC performs a remote direct memory access (RDBMI) write operation, directly writing the data to be processed to this input buffer address without requiring intervention from the destination computing unit. After the write operation is complete, the destination NIC triggers an interrupt signal to notify the destination computing unit to invoke the corresponding expert model to perform forward computation on the data to be processed in the input buffer.

[0146] As another example, in a distributed training gradient reduction-based ensemble communication scenario, the processing object is the model parameter block that needs gradient synchronization during distributed training, and the destination node is the parameter server node or rank node responsible for aggregating the gradients of a specific parameter block. The destination network interface card (NIC) receives the data to be processed (i.e., the local gradient data calculated by the source node) and the distribution identifier (i.e., the identifier of the target parameter block to be reduced and the address of the aggregation buffer) sent by the source NIC of the source node.

[0147] The destination network interface card (NIC) directly transmits the data to be processed to the corresponding aggregation buffer in the storage area of ​​the destination node via remote direct memory access, based on the aggregation buffer address in the distribution identifier. After the data is written, the destination NIC further triggers the destination computing unit to perform gradient accumulation and reduction operations, or performs hardware-level atomic accumulation of the data to be processed with the existing gradient data in the aggregation buffer.

[0148] Step 902: Determine the storage address of the target computing unit where the processing object is located in the destination node based on the distribution identifier and distribution mapping relationship.

[0149] In this embodiment, the execution entity determines the storage address of the target computing unit where the processing object is located in the destination node based on the distribution identifier and the distribution mapping relationship.

[0150] Continue to refer to Figure 10 The diagram illustrates the distribution mapping relationship. The distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the corresponding processing object resides. Specifically, it is represented by an index table stored in the target network interface card's local memory (such as SRAM). This distribution mapping relationship establishes a direct mapping between the processing object and the storage address of the target computing unit, using the distribution identifier of the processing object (e.g., the expert index `expert_index`) as the key. The storage address is composed of the target virtual address (vaddr), the remote access key (rkey), the GPU identifier, and the slot offset (`expert_slot`), used to precisely locate the physical write position of data in the target computing unit's video memory (HBM).

[0151] As an example, in a scenario of parallel routing distribution based on MoE experts, the processing object is the expert model in the hybrid expert model. The destination network interface card has a pre-defined group table as the distribution mapping relationship, which records the binding relationship between the index of each expert model and the corresponding GPU memory address information.

[0152] After receiving the distribution identifier sent by the source node, the destination NIC extracts the expert index as the lookup key. The destination NIC uses this expert index to perform a fixed-width direct index lookup on the group table, finds the corresponding entry, and reads the stored destination virtual address, remote access key, and slot offset. Subsequently, the destination NIC calculates the final storage address of the target computing unit based on the read destination virtual address and slot offset, and initiates a remote direct memory access write operation using the remote access key to accurately write the data to be processed into the GPU memory region corresponding to that storage address.

[0153] As another example, in a scenario based on distributed training gradient reduction, the processing object is the computational unit block responsible for gradient aggregation in distributed training. The destination network interface card maintains a group table containing the correspondence between parameter block identifiers and aggregation buffer addresses as a distribution mapping relationship.

[0154] The destination network interface card (NIC) parses the received distribution identifier to obtain the parameter block identifier of the gradient to be reduced. Using this parameter block identifier as an index, the destination NIC queries the group table to obtain the target virtual address, remote access key, and aggregation slot offset corresponding to the block. Based on this information, the destination NIC determines the storage address of the target computation unit used to perform gradient accumulation and directly transmits the data to be processed to this storage address via remote direct memory access (RDM) for subsequent gradient reduction operations by the target computation unit.

[0155] In some optional implementations of this embodiment, the execution entity determines the storage address of the target computing unit where the processing object is located in the target node in the following manner: First step, through the target network card, determine whether the data to be processed is received data to be processed based on the processed data record, wherein the processed data record represents the data to be processed that has been received by the target network card; Second step, in response to the determination of no, determine the storage address based on the distribution identifier and the distribution mapping relationship.

[0156] The destination network interface card maintains a retransmission deduplication table in its local memory as records of processed data. This table is updated by rolling replacement using a first-in-first-out or least recently used strategy.

[0157] After receiving the data to be processed from the source network interface card (NIC) of the source node, the destination NIC parses the packet header to obtain unique characteristic information such as the source node identifier as a lookup key. The destination NIC uses this lookup key to perform a lookup operation on the retransmission deduplication table. If a table entry is matched and the "written to disk" flag in the entry is true, the data to be processed is determined to be received data, and the destination NIC discards the packet directly; if no table entry is matched or the "written to disk" flag is false, the data to be processed is determined to be non-received data.

[0158] After determining that the data is not already received, the destination network interface card (NIC) determines the storage address based on the distribution identifier and distribution mapping relationship, and writes the data to be processed to that storage address. After writing, the destination NIC updates the processed data records: if the retransmission deduplication table is not full, a new record containing the currently searched key value is added to the end of the table, the disk-written flag is set to true, and the most recent write timestamp is updated; if the table is full, the oldest written or least accessed old table entry is discarded, the write pointer of the new record is pointed to the physical storage location of the discarded entry, completing the cyclic overwriting and status update of the pointer, ensuring that the sliding window always covers the most recent valid transmission record.

[0159] In this implementation, the hardware-level deduplication mechanism of the target network card effectively avoids duplicate data being written to the video memory due to network retransmission, reduces the risk of invalid PCIe bandwidth occupation and video memory pollution, and improves the data transmission reliability and overall system throughput efficiency of the aggregated communication.

[0160] In some optional implementations of this embodiment, the execution entity determines whether the data to be processed is received data to be processed in the following way: by combining the source node corresponding to the data to be processed and the data group to which the data to be processed belongs, it determines whether the processed data record includes the data to be processed.

[0161] Continue to refer to Figure 11 The diagram illustrates a processed data record. Specifically, the processed data record is represented as a sliding window cache table stored in the destination network interface card's (NIC) local memory. This record uses a triplet as its key, containing the source node identifier, the set communication task group identifier, and the data sequence number, uniquely identifying a data transmission request. The record's value fields include a disk write flag and a recent write timestamp. By querying this record, the destination NIC determines whether the currently received data to be processed has been successfully received and written to memory within the sliding window depth, thereby identifying network retransmissions or redundant data packets.

[0162] If a data packet is determined to be a received duplicate, the destination network interface card (NIC) discards the data payload of the packet, but still needs to send an acknowledgment (ACK) signal back to the source node to trigger the source NIC to release the transmission context and terminate the retransmission.

[0163] After receiving the data to be processed from the source network interface card (NIC) of the source node, the destination NIC parses the packet header to obtain a triple index consisting of the source node identifier, group identifier, and sequence number. The destination NIC uses this triple index to perform a lookup operation in the retransmission deduplication table. If a table entry is matched and the "written to disk" flag in the entry is true, the data to be processed is determined to be received data, and the destination NIC discards the packet and stops subsequent transmission. If a table entry is not matched or the "written to disk" flag is false, the data to be processed is determined to be different from received data.

[0164] After determining that the data is not already received, the destination network interface card (NIC) queries the distribution mapping relationship based on the distribution identifier to determine the storage address and writes the data to be processed to that storage address. After writing, the destination NIC updates the processed data record: if the retransmission deduplication table is not full, a new record containing the current triplet index is added to the end of the table, the disk-written flag is set to true, and the most recent write timestamp is updated; if the table is full, the oldest written or least accessed old table entry is discarded, the pointer of the new record is pointed to the physical storage location of the discarded entry, and the pointer is cyclically overwritten and the status is updated to ensure that the sliding window always covers the most recent valid transmission sequence.

[0165] In this implementation, a hardware-level deduplication mechanism based on the source node corresponding to the data to be processed and the data group to which the data to be processed belongs is implemented through the target network card, which further improves the accuracy and effectiveness of the deduplication mechanism.

[0166] Step 903: Transmit the data to be processed to the storage area indicated by the storage address through the destination network card, and process the data to be processed through the processing object.

[0167] In this embodiment, the execution entity transmits the data to be processed to the storage area indicated by the storage address through the destination network card, and processes the data to be processed through the processing object.

[0168] As an example, in a scenario based on MoE expert parallel routing distribution, after determining the storage address, the destination network interface card (NIC) uses remote direct memory access (RDA) write operations to directly transfer the data to be processed (i.e., the input token feature vector) to the video memory storage area of ​​the target computing unit indicated by the storage address. This video memory storage area is specifically the input buffer of the corresponding expert model.

[0169] After the data to be processed is written, the destination network card sends an interrupt signal or doorbell notification to the target computing unit. In response to the notification, the target computing unit reads the data to be processed from the video memory storage area and calls the corresponding expert model to perform forward inference calculations on the data to be processed, thereby completing the processing of the data.

[0170] As another example, in a scenario based on distributed training gradient reduction, the target network interface card (NIC) transfers the data to be processed (i.e., local gradient data) to the storage area indicated by the storage address via a remote direct memory access atomic write operation. This storage area is specifically a global buffer in the target computation unit used to temporarily store aggregated gradients.

[0171] After the data to be processed is written to the global buffer, the target computation unit is triggered to execute the gradient reduction task. The target computation unit reads the data to be processed from the global buffer and performs an accumulation operation with the historical gradient data already existing in the buffer to update the global gradient state, thereby realizing the process of processing the data to be processed through the processing object.

[0172] In some optional implementations of this embodiment, the execution entity transmits the data to be processed to the storage area indicated by the storage address in the following manner: in response to multiple processing objects including the data to be processed in the destination node, for each of the multiple processing objects in the target computing unit, the data to be processed is distributed to the storage area indicated by the storage address of the multiple target computing units through the destination network card.

[0173] After parsing the distribution identifier, the destination network interface card (NIC) identifies multiple processing objects within the destination node corresponding to the data to be processed. The destination NIC then sequentially traverses these processing objects, and for each object, uses the distribution mapping relationship to determine the storage address of its target computing unit.

[0174] During the process of determining the storage address, the destination network interface card (NIC) maintains a write pointer pointing to the memory buffer of the current target computing unit. For the first object to be processed, the destination NIC reads the base virtual address and slot offset recorded in the distribution mapping relationship and initializes the write pointer to the sum of the base address and the offset. For subsequent objects to be processed, the destination NIC performs an increment update operation on the write pointer according to the preset data block size or the end position of the previous write operation, so that the write pointer points to the starting position of the storage area of ​​the next target computing unit.

[0175] The destination network interface card (NIC) transmits copies or fragments of the data to be processed to the corresponding storage area via remote direct memory access (RDBMI) write operations, based on the storage address indicated by the updated write pointer. After each data transmission, the destination NIC updates the write pointer again until all target computing units corresponding to the processing objects have received the data to be processed, thereby achieving one-to-many hardware-level data distribution.

[0176] In this implementation, the parallel distribution of single-packet data to multiple computing units is achieved through the destination network card hardware automatic addressing and pointer update mechanism. This avoids bandwidth waste caused by repeated sending at the source end, reduces communication latency, and improves data distribution efficiency and system throughput in aggregated communication scenarios.

[0177] In some optional implementations of this embodiment, the execution entity may also perform the following operation: send feedback information indicating that the data to be processed has been transmitted to the storage area of ​​all target computing units to the source node through the destination network card.

[0178] The target network interface card (NIC) maintains an aggregated state table and corresponding feedback pointers to track the distribution progress of data to be processed to multiple target computing units. Each entry in the aggregated state table contains the expected receive count, the current received count, and a write pointer to the feedback information buffer.

[0179] Once the destination network interface card (NIC) has completed transmitting the data to be processed to the storage area indicated by the storage address of a target computing unit, it triggers an update operation on the corresponding entry in the aggregation state table. Specifically, the destination NIC reads the current received count, increments it by one to indicate that a newly added target computing unit has been received, and writes the updated value back to the aggregation state table. Simultaneously, the destination NIC updates the write pointer based on the current received count, causing the write pointer to point to a specific offset position in the feedback information buffer used to record the completion status, and marks the completion flag at that position.

[0180] The destination network interface card (NIC) continuously monitors whether the current received count matches the expected received count. When they are equal, it indicates that the data to be processed has been successfully transmitted to the storage areas of all target computing units. At this point, the destination NIC assembles feedback information containing the group identifier based on the final position of the write pointer in the feedback information buffer, and sends the feedback information to the source node through the network interface to notify the source node to release temporary resources or perform subsequent scheduling.

[0181] In this implementation, the hardware-level aggregation feedback mechanism of the destination network card converges multiple scattered transmission acknowledgments into a single feedback, which significantly reduces the bandwidth overhead of reverse control traffic, alleviates the pressure of source-end processing interruption, and improves the synchronization efficiency and system scalability of large-scale cluster communication.

[0182] In some optional implementations of this embodiment, the execution entity processes the data to be processed through the processing object in the following manner: First, it generates a task completion entry representing that the data to be processed has been transmitted to the storage area of ​​the target computing unit through the target network card, and adds the task completion entry to the task processing queue of the target computing unit; then, in response to the task completion entry in the task processing queue, it processes the data to be processed through the processing object deployed in the target computing unit.

[0183] A Completion Queue Element (CQE) is a lightweight hardware notification primitive submitted by the destination network interface card (NIC) to the target computing unit (PCU) after confirming that the data to be processed has been completely written to the PCU's storage area. This entry does not contain actual data payload, but only metadata information such as a task identifier, a data storage address pointer, and a completion status flag. It is used to establish an asynchronous task handover mechanism between the destination NIC and the PCU. Once the CQE is added to the PCU's task processing queue, it serves as a hardware-level signal to trigger the PCU to start subsequent computation tasks, achieving decoupling and pipelined parallelism between the communication and computation sides.

[0184] After the destination network interface card (NIC) transmits the data to be processed to the storage area indicated by the storage address of the target computing unit via remote direct memory access (RDBMI), it generates a task completion entry. The destination NIC allocates storage space for the task completion entry from its local memory and writes the task identifier, storage address pointer, and status flag indicating the completion of the transmission corresponding to the data to be processed into the entry, thus forming a complete task completion entry.

[0185] Subsequently, the destination network interface card (NIC) adds the generated task completion entry to the task processing queue of the target computing unit. Specifically, the destination NIC reads the current enqueue pointer of the task processing queue, writes the task completion entry to the queue slot pointed to by the enqueue pointer, and moves the enqueue pointer one slot backward along the queue direction to complete the enqueue operation. If the enqueue pointer moves to the end of the queue, it wraps back to the beginning of the queue to maintain the continuous write capability of the circular queue.

[0186] The target computing unit periodically polls or uses interrupts to check the task completion entries in the task processing queue. When the target computing unit detects a new task completion entry, it reads the storage address pointer and task identifier from the entry, retrieves the data to be processed from the corresponding storage area, and calls the processing object deployed in the target computing unit to perform computation on the data to be processed. After completing the computation, the processing object updates the status flag in the task completion entry to the computation completion status, or generates a new computation completion entry and returns it to the destination network interface card (NIC) for subsequent status synchronization and resource release.

[0187] In this implementation, task completion entries are generated by the target network card and added to the task processing queue. This achieves hardware-level asynchronous triggering from communication completion to computation startup, avoiding the waste of computing power caused by the target computing unit actively polling the storage area, reducing the synchronization overhead between communication and computation, and improving the resource utilization and overall processing throughput efficiency of the target computing unit.

[0188] See also Figure 12 , Figure 12This is a schematic diagram of an application scenario 1200 of the aggregate communication method according to this embodiment. User 1201 sends data to be processed to server cluster 1203 through terminal device 1202. The server cluster deploys a MoE large model, in which the source node deploys a MoE gated network generates an aggregate communication task through the source computing unit in the source node and transmits the aggregate communication task to the source network card of the source node. The aggregate communication task includes the data to be processed and object representation data of multiple expert models of the data to be processed. Through the source network card, the destination node of each of the multiple expert models is determined according to the object representation data and the routing mapping relationship, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination node of the expert model. Through the source network card, the data to be processed and the distribution identifier of the processed object are transmitted to the destination node of the expert model.

[0189] The destination node obtains the data to be processed and the distribution identifier through the destination network interface card (NIC). Based on the distribution identifier and the distribution mapping relationship, it determines the storage address of the target computing unit where the expert model is located in the destination node. The distribution mapping relationship represents the correspondence between the distribution identifier of the expert model and the storage address of the target computing unit where the expert model is located. The data to be processed is transmitted to the storage area indicated by the storage address through the destination NIC, and the data to be processed is processed by the expert model.

[0190] In this embodiment, the sequential triggering and data caching mechanism of aggregated communication is pushed down to the network card hardware pipeline. By issuing a single aggregated communication task, the hardware is driven to autonomously complete the sequential scheduling and dynamic data caching of concurrent multi-tasks, realizing hardware-level autonomous execution of aggregated communication. This not only releases the computing power occupied by host CPU busy polling, but also avoids the computing power erosion caused by the GPU side constructing WQE in the Warp loop in the GIN / GDAKI route, and avoids the latency jitter introduced by the CPU Proxy route due to the long path and multi-stage synchronization across devices. By autonomously implementing cross-node on-demand physical deduplication token multicast through network card hardware, the physical transmission path size is reduced while the inter-node communication has multicast capability, saving cross-node network bandwidth.

[0191] Continue to refer to Figure 13 A schematic diagram of a collective communication system is shown. The collective communication system 1300 includes: Source node 1301 is used to: generate a collection communication task through the source computing unit in the source node, and transmit the collection communication task to the source network interface card (NIC) of the source node, wherein the collection communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed; determine the destination node of each of the multiple processing objects through the source NIC according to the object representation data and the routing mapping relationship, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination node of the processing object; and transmit the data to be processed and the distribution identifier of the processing object to the destination node of the processing object through the source NIC.

[0192] The destination node 1302 is used to: obtain the data to be processed and the distribution identifier through the destination network interface card; determine the storage address of the target computing unit where the processing object is located in the destination node according to the distribution identifier and the distribution mapping relationship, wherein the distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the processing object corresponding to the distribution identifier is located; transmit the data to be processed to the storage area indicated by the storage address through the destination network interface card, and process the data to be processed through the processing object.

[0193] In some optional implementations of this embodiment, the source node 1301 is further configured to: for each destination node, determine the connection identifier of the network connection between the source computing unit and the destination node according to the node connection mapping relationship, wherein the node connection mapping relationship represents the connection identifier of the network connection between the source computing unit and the destination node; combine the connection identifier and the storage address of the data to be processed in the source computing unit to generate and add a work queue entry to the work queue corresponding to the network connection; for the work queue entry in the work queue, use the network connection corresponding to the connection identifier in the work queue entry to transmit the data to be processed at the storage address in the work queue entry and the distribution identifier of the processing object of the data to be processed in the destination node to the destination node.

[0194] In some optional implementations of this embodiment, the source node 1301 is further configured to: in response to an active connection whose connection context is cached by the source network card in a network connection whose work queue is not empty, use the active connection to transmit the data to be processed at the storage address in the work queue entry of the active connection and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the active connection to the destination node of the active connection.

[0195] In some optional implementations of this embodiment, the source node 1301 is further configured to: combine the data to be processed at the storage address in the work queue entry corresponding to the active connection with the distribution identifier of each processing object of the data to be processed in the destination node to generate a sending message; and use the active connection to transmit the sending message to the destination node.

[0196] In some optional implementations of this embodiment, the source node 1301 is further configured to: use an active connection to continuously transmit the sending messages corresponding to each work queue entry of the active connection's work queue to the destination node.

[0197] In some optional implementations of this embodiment, the source node 1301 is further configured to: load and cache the connection context of inactive connections whose work queues are not empty, in response to the number of active connections whose connection contexts are cached in the source network interface card not reaching a preset threshold; and use inactive connections to transmit the data to be processed at the storage address of the work queue entry in the work queue of the inactive connection, and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the inactive connection, to the destination node of the inactive connection.

[0198] In some optional implementations of this embodiment, the collective communication task is stored in the task cache queue of the source node, and the source node 1301 is further configured to: generate a collective communication task through the source computing unit in response to the fact that the difference between the consumer pointer and the producer pointer corresponding to the task cache queue does not reach the length value of the task cache queue, wherein the consumer pointer is a pointer to the processed tasks in the task cache queue, and the producer pointer is a pointer to the unprocessed tasks in the task cache queue; and the source node 1301 is further configured to: add the collective communication task to the task cache queue and update the producer pointer.

[0199] In some optional implementations of this embodiment, the destination node 1302 is further configured to: determine whether the data to be processed is received data to be processed based on the processed data record through the destination network interface card, wherein the processed data record represents the data to be processed that has been received by the destination network interface card; and determine the storage address based on the distribution identifier and the distribution mapping relationship in response to the determination that no data is received.

[0200] In some optional implementations of this embodiment, the destination node 1302 is further used to: determine whether the processed data record includes the data to be processed by combining the source node corresponding to the data to be processed and the data group to which the data to be processed belongs.

[0201] In some optional implementations of this embodiment, the destination node 1302 is further configured to: in response to multiple processing objects including data to be processed in the destination node, distribute the data to be processed to the storage area indicated by the storage address of the multiple target computing units through the destination network card for each of the multiple processing objects.

[0202] In some optional implementations of this embodiment, the destination node 1302 is further configured to: generate a task completion entry representing that the data to be processed has been transmitted to the storage area of ​​the target computing unit through the destination network card, and add the task completion entry to the task processing queue of the target computing unit; and process the data to be processed through the processing object deployed in the target computing unit in response to the task completion entry in the task processing queue.

[0203] In some optional implementations of this embodiment, the destination node 1302 is further configured to: send feedback information indicating that the data to be processed has been transmitted to the storage area of ​​all target computing units to the source node via the destination network card; and the source node 1301 is further configured to: update the consumer pointer in response to receiving feedback information from the destination nodes of all objects to be processed in the aggregate communication task via the source network card.

[0204] In some optional implementations of this embodiment, the processing object is the expert model in the hybrid expert model, and the source node is the computing node where the gating network in the hybrid expert model is located.

[0205] This embodiment provides a collective communication system that pushes the sequential triggering and data caching mechanism of collective communication down to the network card hardware pipeline. By issuing a single collective communication task, the hardware is driven to autonomously complete the sequential scheduling and dynamic data caching of concurrent multi-tasks, realizing hardware-level autonomous execution of collective communication. This not only releases the computing power occupied by host CPU busy polling, but also avoids the computing power erosion caused by the GPU side constructing WQE in the Warp loop in the GIN / GDAKI route, and avoids the latency jitter introduced by the CPU Proxy route due to the long path and multi-stage synchronization across devices. By autonomously implementing cross-node on-demand physical deduplication token multicast through network card hardware, the physical transmission path size is reduced while the inter-node communication has multicast capabilities, saving cross-node network bandwidth.

[0206] Specifically, in this embodiment, based on the aggregated communication task, the Top-K dispatch (distribution) in the aggregated communication scenario, which originally required software looping for many-to-many communication, is abstracted into an aggregated communication data transmission abstraction layer. The GPUSM delivers a single CCD structure containing K object representation data (e.g., Expert ID) to the source network card, ringing the doorbell once. The hardware pipeline inside the network card identifies and parses the CCD, pulls the Token data once via PCIe DMA, temporarily stores it in the network card's internal transmit buffer, and receives an aggregated completion queue item (CQE) after the transmission is complete. In this way, the bandwidth waste caused by the looping pull of the Payload across the PCIe bus K times in the original software Top-K looping unicast scheme is avoided. The number of times the Token data is pulled across the PCIe bus is reduced from K times to 1 time, and the PCIe transmission bandwidth is reduced to 1 / K of that in the original software Top-K looping unicast.

[0207] Continue to refer to Figure 14 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a collection communication device, which is similar to... Figure 6 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0208] like Figure 14 As shown, the aggregated communication device 1400 is deployed at the source node and includes: a task generation unit 1401, configured to generate aggregated communication tasks through the source computing unit in the source node and transmit the aggregated communication tasks to the source network interface card (NIC) of the source node, wherein the aggregated communication tasks include data to be processed and object representation data of multiple processing objects of the data to be processed; a node determination unit 1402, configured to determine the destination nodes of the multiple processing objects according to the object representation data and the routing mapping relationship through the source NIC, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination nodes where the processing objects are located; and a data transmission unit 1403, configured to transmit the data to be processed and the distribution identifiers of the processing objects to the destination nodes where the processing objects are located through the source NIC.

[0209] In some optional implementations of this embodiment, the data transmission unit 1403 is further configured to: for each destination node, determine the connection identifier of the network connection between the source computing unit and the destination node according to the node connection mapping relationship, wherein the node connection mapping relationship represents the connection identifier of the network connection between the source computing unit and the destination node; combine the connection identifier and the storage address of the data to be processed in the source computing unit to generate and add a work queue entry to the work queue corresponding to the network connection; for the work queue entry in the work queue, use the network connection corresponding to the connection identifier in the work queue entry to transmit the data to be processed at the storage address in the work queue entry and the distribution identifier of the processing object of the data to be processed in the destination node to the destination node.

[0210] In some optional implementations of this embodiment, the data transmission unit 1403 is further configured to: in response to an active connection whose connection context is cached by the source network card in a network connection whose work queue is not empty, use the active connection to transmit the data to be processed at the storage address in the work queue entry of the active connection and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the active connection to the destination node of the active connection.

[0211] In some optional implementations of this embodiment, the data transmission unit 1403 is further configured to: combine the data to be processed at the storage address in the work queue entry corresponding to the active connection with the distribution identifier of each processing object of the data to be processed in the destination node to generate a sending message; and use the active connection to transmit the sending message to the destination node.

[0212] In some optional implementations of this embodiment, the data transmission unit 1403 is further configured to: use an active connection to continuously transmit the sending messages corresponding to each work queue entry of the active connection's work queue to the destination node.

[0213] In some optional implementations of this embodiment, the above apparatus further includes: a connection loading unit (not shown in the figure), configured to: load and cache the connection context of inactive connections whose work queues are not empty, in response to the number of active connections whose cached connection contexts in the source network card not reaching a preset threshold; the data transmission unit 1403 is further configured to: use inactive connections to transmit the data to be processed at the storage address in the work queue entry of the inactive connection's work queue, and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the inactive connection, to the destination node of the inactive connection.

[0214] In some optional implementations of this embodiment, the collective communication task is stored in the task cache queue of the source node, and the task generation unit 1401 is further configured to: generate a collective communication task through the source computing unit in response to the fact that the difference between the consumer pointer and the producer pointer corresponding to the task cache queue does not reach the length value of the task cache queue, wherein the consumer pointer is a pointer to the processed tasks in the task cache queue, and the producer pointer is a pointer to the unprocessed tasks in the task cache queue; and the above-mentioned device further includes: a pointer update unit (not shown in the figure), configured to: add the collective communication task to the task cache queue and update the producer pointer.

[0215] In some optional implementations of this embodiment, the pointer update unit is further configured to update the consumer pointer in response to receiving feedback information from the destination node of each of the objects to be processed in the aggregate communication task through the source network card, wherein the feedback information is used to indicate that the data to be processed has been transmitted to the storage area of ​​the target computing unit to which the feedback information belongs.

[0216] In some optional implementations of this embodiment, the processing object is the expert model in the hybrid expert model, and the source node is the computing node where the gating network in the hybrid expert model is located.

[0217] In this embodiment, a collective communication device is provided, which sinks the entire multicast control plane to the network card hardware pipeline. By issuing a single collective communication task, the network card hardware is triggered to realize the multicast process of data to be processed across nodes. This enables inter-node communication to have multicast capabilities while reducing the physical transmission path size and saving cross-node network bandwidth.

[0218] Continue to refer to Figure 15 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a collection communication device, which is similar to... Figure 9 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0219] like Figure 15As shown, the aggregated communication device 1500 is deployed at the destination node and includes: a data acquisition unit 1501, configured to acquire, through the destination network interface card (NIC) of the destination node, the data to be processed sent by the source NIC of the source node and the distribution identifier of the processing object of the data to be processed; an address determination unit 1502, configured to determine the storage address of the target computing unit where the processing object is located in the destination node according to the distribution identifier and the distribution mapping relationship, wherein the distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the processing object corresponding to the distribution identifier is located; and a data processing unit 1503, configured to transmit the data to be processed to the storage area indicated by the storage address through the destination NIC, and process the data to be processed through the processing object.

[0220] In some optional implementations of this embodiment, the address determination unit 1502 is further configured to: determine whether the data to be processed is received data to be processed based on the processed data record through the destination network card, wherein the processed data record represents the data to be processed that has been received by the destination network card; and determine the storage address based on the distribution identifier and the distribution mapping relationship in response to the determination that no.

[0221] In some optional implementations of this embodiment, the address determination unit 1502 is further configured to: determine whether the processed data record includes the data to be processed by combining the source node corresponding to the data to be processed and the data group to which the data to be processed belongs.

[0222] In some optional implementations of this embodiment, the data processing unit 1503 is further configured to: in response to multiple processing objects including data to be processed in the destination node, distribute the data to be processed to the storage area indicated by the storage address of the multiple target computing units through the destination network card for each of the multiple processing objects.

[0223] In some optional implementations of this embodiment, the above apparatus further includes: a feedback unit (not shown in the figure), configured to: send feedback information indicating that the data to be processed has been transmitted to the storage area of ​​all target computing units to the source node via the destination network card.

[0224] In some optional implementations of this embodiment, the data processing unit 1503 is further configured to: generate a task completion entry representing that the data to be processed has been transmitted to the storage area of ​​the target computing unit through the target network card, and add the task completion entry to the task processing queue of the target computing unit; and process the data to be processed through the processing object deployed in the target computing unit in response to the task completion entry in the task processing queue.

[0225] In this embodiment, a collective communication device is provided, which sinks the entire multicast control plane to the network card hardware pipeline. By issuing a single collective communication task, the network card hardware is triggered to realize the multicast process of data to be processed across nodes. This enables inter-node communication to have multicast capabilities while reducing the physical transmission path size and saving cross-node network bandwidth.

[0226] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the collection communication method described in any of the above embodiments when executed.

[0227] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the collection communication method described in any of the above embodiments when executed.

[0228] This disclosure provides a computer program product that, when executed by a processor, can implement the set communication method described in any of the above embodiments.

[0229] Figure 16 A schematic block diagram of an example electronic device 1600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0230] like Figure 16 As shown, device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1602 or a computer program loaded into random access memory (RAM) 1603 from storage unit 1608. The RAM 1603 may also store various programs and data required for the operation of device 1600. The computing unit 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0231] Multiple components in device 1600 are connected to I / O interface 1605, including: input unit 1606, such as keyboard, mouse, etc.; output unit 1607, such as various types of monitors, speakers, etc.; storage unit 1608, such as disk, optical disk, etc.; and communication unit 1609, such as network card, modem, wireless transceiver, etc. Communication unit 1609 allows device 1600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0232] The computing unit 1601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods and processes described above, such as the collective communication method. For example, in some embodiments, the collective communication method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1600 via ROM 1602 and / or communication unit 1609. When the computer program is loaded into RAM 1603 and executed by the computing unit 1601, one or more steps of the collective communication method described above may be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to perform the collective communication method by any other suitable means (e.g., by means of firmware).

[0233] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0234] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable collective communication device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0235] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0236] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0237] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0238] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers incorporating blockchain technology.

[0239] According to the technical solution of this disclosure, a collective communication method, system, and apparatus are provided. A source node generates a collective communication task through a source computing unit and transmits the collective communication task to a source network interface card (NIC). The collective communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed. Through the source NIC, the destination node of each of the multiple processing objects is determined based on the object representation data and a routing mapping relationship, where the routing mapping relationship represents the correspondence between the object representation data and the destination nodes of the processing objects. Through the source NIC, the data to be processed and the distribution identifiers of the processing objects are transmitted to the destination nodes of the processing objects. The destination nodes obtain the data to be processed through the destination NIC. The distribution identifier and the distribution mapping relationship are used to determine the storage address of the target computing unit where the processing object is located in the destination node. The distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the processing object is located. The data to be processed is transmitted to the storage area indicated by the storage address through the destination network card, and the data to be processed is processed through the processing object. This pushes the multicast control plane down to the network card hardware pipeline. By issuing a single collection communication task, the network card hardware is triggered to realize the multicast process of the data to be processed across nodes. This enables the communication between nodes to reduce the physical transmission path size while having multicast capability, thus saving the network bandwidth across nodes.

[0240] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0241] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A set communication method, applied to a source node, comprising: A collection communication task is generated by the source computing unit in the source node, and the collection communication task is transmitted to the source network card of the source node. The collection communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed. Using the source network interface card, the destination node of each of the multiple processing objects is determined according to the object representation data and the routing mapping relationship, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination node of the processing object; The source network interface card (NIC) transmits the data to be processed and the distribution identifier of the processing object to the destination node where the processing object is located.

2. The method according to claim 1, wherein, The step of transmitting the data to be processed and the distribution identifier of the processing object to the destination node where the processing object is located includes: For each destination node, the connection identifier of the network connection between the source computing unit and the destination node is determined according to the node connection mapping relationship, wherein the node connection mapping relationship represents the connection identifier of the network connection between the source computing unit and the destination node; By combining the connection identifier and the storage address of the data to be processed in the source computing unit, a work queue entry is generated and added to the work queue corresponding to the network connection; For each work queue entry in the work queue, the network connection corresponding to the connection identifier in the work queue entry is used to transmit the data to be processed at the storage address in the work queue entry and the distribution identifier of the processing object of the data to be processed in the destination node to the destination node.

3. The method according to claim 2, wherein, For each work queue entry in the work queue, the network connection corresponding to the connection identifier in the work queue entry is used to transmit the data to be processed at the storage address in the work queue entry and the distribution identifier of the processing object of the data to be processed in the destination node to the destination node, including: In response to an active connection whose connection context is cached by the source network card in a network connection whose work queue is not empty, the data to be processed at the storage address in the work queue entry of the active connection, and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the active connection are transmitted to the destination node of the active connection.

4. The method according to claim 3, wherein, The step of using the active connection to transmit the data to be processed at the storage address of the work queue entry in the work queue of the active connection, and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the active connection, to the destination node of the active connection includes: Combine the data to be processed at the storage address in the work queue entry corresponding to the active connection with the distribution identifier of each processing object of the data to be processed in the destination node to generate a sending message; The active connection is used to transmit the message to the destination node.

5. The method according to claim 3, wherein, The step of using the active connection to transmit the sent message to the destination node includes: Using the active connection, the send packets corresponding to each work queue entry of the active connection's work queue are continuously transmitted to the destination node.

6. The method according to claim 3, wherein, Also includes: In response to the fact that the number of active connections with cached connection contexts in the source network interface card does not reach a preset threshold, the connection contexts of inactive connections that are not empty in the work queue are loaded and cached through the source network interface card. Using the inactive connection, the data to be processed at the storage address of the work queue entry in the work queue of the inactive connection, and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the inactive connection are transmitted to the destination node of the inactive connection.

7. The method according to claim 1, wherein, The collective communication task is stored in the task cache queue of the source node, and The generation of the collection communication task through the source computing unit in the source node includes: In response to the difference between the consumer pointer and the producer pointer corresponding to the task cache queue not reaching the length of the task cache queue, the collection communication task is generated by the source computing unit, wherein the consumer pointer is a pointer to the processed tasks in the task cache queue, and the producer pointer is a pointer to the unprocessed tasks in the task cache queue; and The method further includes: Add the collection communication task to the task cache queue and update the producer pointer.

8. The method according to claim 7, wherein, Also includes: In response to receiving feedback information from the destination nodes of all pending objects of the aggregate communication task through the source network interface card, the consumer pointer is updated, wherein the feedback information is used to indicate that the pending data has been transmitted to the storage area of ​​the target computing unit to which the feedback information belongs.

9. The method according to any one of claims 1-8, wherein, The processing object is the expert model in the hybrid expert model, and the source node is the computing node where the gating network in the hybrid expert model is located.

10. A collective communication method, applied to a destination node, comprising: The destination network interface card (NIC) of the destination node is used to obtain the data to be processed sent by the source NIC of the source node and the distribution identifier of the processing object of the data to be processed. The storage address of the target computing unit where the processing object is located in the destination node is determined according to the distribution identifier and the distribution mapping relationship, wherein the distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the processing object corresponding to the distribution identifier is located; The data to be processed is transmitted to the storage area indicated by the storage address through the destination network card, and the data to be processed is processed through the processing object.

11. The method according to claim 10, wherein, Determining the storage address of the target computing unit where the processing object is located in the destination node based on the distribution identifier and distribution mapping relationship includes: Using the target network interface card (NIC), it is determined whether the data to be processed is received data to be processed based on the processed data record, wherein the processed data record represents the data to be processed that has been received by the target NIC. In response to a no-determined condition, the storage address is determined based on the distribution identifier and the distribution mapping relationship.

12. The method according to claim 11, wherein, The step of determining whether the data to be processed is received data to be processed based on the processed data record includes: By combining the source node corresponding to the data to be processed and the data group to which the data to be processed belongs, it is determined whether the data to be processed is included in the processed data record.

13. The method according to claim 10, wherein, The step of transmitting the data to be processed to the storage area indicated by the storage address via the destination network interface card includes: In response to the plurality of processing objects including the data to be processed in the destination node, for each of the plurality of processing objects in the target computing unit, the data to be processed is distributed to the storage area indicated by the storage address of the plurality of target computing units through the destination network interface card.

14. The method of claim 10, wherein, Also includes: The target network interface card sends feedback information indicating that the data to be processed has been transmitted to the storage area of ​​all the target computing units to the source node.

15. The method according to claim 10, wherein, The process of processing the data to be processed through the processing object includes: The target network interface card generates a task completion entry indicating that the data to be processed has been transmitted to the storage area of ​​the target computing unit, and adds the task completion entry to the task processing queue of the target computing unit. In response to task completion entries in the task processing queue, the data to be processed is processed by the processing object deployed in the target computing unit.

16. A collective communication system, comprising: A source node is configured to: generate a collective communication task through a source computing unit within the source node, and transmit the collective communication task to a source network interface card (NIC) of the source node, wherein the collective communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed; determine the destination node of each of the multiple processing objects based on the object representation data and a routing mapping relationship through the source NIC, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination node of the processing object; and transmit the data to be processed and the distribution identifier of the processing object to the destination node of the processing object through the source NIC. The destination node is configured to: acquire the data to be processed and the distribution identifier through the destination network interface card (NIC); determine the storage address of the target computing unit where the processing object is located in the destination node according to the distribution identifier and the distribution mapping relationship, wherein the distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the processing object corresponding to the distribution identifier is located; transmit the data to be processed to the storage area indicated by the storage address through the destination NIC; and process the data to be processed through the processing object.

17. The system according to claim 16, wherein, The source node is further used for: For each destination node, a connection identifier for the network connection between the source computing unit and the destination node is determined according to the node connection mapping relationship, wherein the node connection mapping relationship represents the connection identifier for the network connection between the source computing unit and the destination node; combining the connection identifier and the storage address of the data to be processed in the source computing unit, a work queue entry is generated and added to the work queue corresponding to the network connection; for the work queue entry in the work queue, the network connection corresponding to the connection identifier in the work queue entry is used to transmit the data to be processed at the storage address in the work queue entry and the distribution identifier of the processing object of the data to be processed in the destination node to the destination node.

18. The system according to claim 17, wherein, The source node is further used for: In response to an active connection whose connection context is cached by the source network card in a network connection whose work queue is not empty, the data to be processed at the storage address in the work queue entry of the active connection, and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the active connection are transmitted to the destination node of the active connection.

19. The system according to claim 18, wherein, The source node is further used for: Combine the data to be processed at the storage address in the work queue entry corresponding to the active connection with the distribution identifier of each processing object of the data to be processed in the destination node to generate a sending message; The active connection is used to transmit the message to the destination node.

20. The system according to claim 18, wherein, The source node is further used for: Using the active connection, the send packets corresponding to each work queue entry of the active connection's work queue are continuously transmitted to the destination node.

21. The system according to claim 18, wherein, The source node is also used for: In response to the fact that the number of active connections with cached connection contexts in the source network interface card does not reach a preset threshold, the connection contexts of inactive connections with non-empty work queues are loaded and cached. Using the inactive connection, the data to be processed at the storage address in the work queue entry of the inactive connection and the distribution identifier of the processing object of the data to be processed in the destination node corresponding to the inactive connection are transmitted to the destination node of the inactive connection.

22. The system according to claim 16, wherein, The collective communication task is stored in the task cache queue of the source node, and The source node is further used for: In response to the difference between the consumer pointer and the producer pointer corresponding to the task cache queue not reaching the length of the task cache queue, the collection communication task is generated by the source computing unit, wherein the consumer pointer is a pointer to the processed tasks in the task cache queue, and the producer pointer is a pointer to the unprocessed tasks in the task cache queue; and The source node is also used for: Add the collection communication task to the task cache queue and update the producer pointer.

23. The system according to claim 16, wherein, The destination node is further used for: Using the destination network interface card (NIC), it is determined whether the data to be processed is received data to be processed based on the processed data record, wherein the processed data record represents the data to be processed that has been received by the destination NIC; in response to determining no, the storage address is determined based on the distribution identifier and the distribution mapping relationship.

24. The system according to claim 23, wherein, The destination node is further used for: By combining the source node corresponding to the data to be processed and the data group to which the data to be processed belongs, it is determined whether the data to be processed is included in the processed data record.

25. The system according to claim 16, wherein, The destination node is further used for: In response to the plurality of processing objects including the data to be processed in the destination node, for each of the plurality of processing objects in the target computing unit, the data to be processed is distributed to the storage area indicated by the storage address of the plurality of target computing units through the destination network interface card.

26. The system according to claim 16, wherein, The destination node is further used for: The target network interface card generates a task completion entry indicating that the data to be processed has been transmitted to the storage area of ​​the target computing unit, and adds the task completion entry to the task processing queue of the target computing unit; in response to the task completion entry in the task processing queue, the data to be processed is processed by the processing object deployed in the target computing unit.

27. The system according to claim 22, wherein, The destination node is also used for: The target network interface card (NIC) sends feedback information indicating that the data to be processed has been transmitted to the storage areas of all the target computing units to the source node; and The source node is also used for: In response to receiving feedback information from the destination nodes of all pending objects of the aggregate communication task through the source network interface card, the consumer pointer is updated.

28. The system according to any one of claims 16-27, wherein, The processing object is the expert model in the hybrid expert model, and the source node is the computing node where the gating network in the hybrid expert model is located.

29. A collective communication device, deployed at a source node, comprising: The task generation unit is configured to generate a collection communication task through the source computing unit in the source node and transmit the collection communication task to the source network card of the source node, wherein the collection communication task includes data to be processed and object representation data of multiple processing objects of the data to be processed; The node determination unit is configured to determine the destination node of each of the multiple processing objects based on the object representation data and the routing mapping relationship through the source network interface card, wherein the routing mapping relationship represents the correspondence between the object representation data and the destination node of the processing object; The data transmission unit is configured to transmit the data to be processed and the distribution identifier of the processing object to the destination node where the processing object is located via the source network interface card.

30. A collective communication device, deployed at a destination node, comprising: The data acquisition unit is configured to acquire, through the destination network card of the destination node, the data to be processed sent by the source network card of the source node and the distribution identifier of the processing object of the data to be processed; The address determination unit is configured to determine the storage address of the target computing unit where the processing object in the destination node is located based on the distribution identifier and the distribution mapping relationship, wherein the distribution mapping relationship represents the correspondence between the distribution identifier and the storage address of the target computing unit where the processing object corresponding to the distribution identifier is located; The data processing unit is configured to transmit the data to be processed to the storage area indicated by the storage address via the destination network card, and to process the data to be processed via the processing object.

31. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.

32. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-15.

33. A computer program product comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-15.

Images