A method and system for offloading control plane of cross-node RDMA communication to network adapters in collective communication
Patent Information
- Application Number
- CN202611251775.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-18
- Publication Date
- 2026-11-03
AI Technical Summary
Existing cross-node RDMA communication schemes consume a large amount of CPU and GPU resources during large model training, resulting in reduced utilization of computing resources and issues with communication latency and overhead.
By adding a collective communication agent hardware module to the RDMA network interface card or data processing unit, the progress of cross-node RDMA communication can be directly managed and controlled, replacing the host CPU agent thread and offloading the communication control plane to the network adapter.
It completely eliminates the CPU overhead and GPU SM core usage caused by CPU proxy threads, reduces operating system context switching and software scheduling overhead, and improves communication bandwidth and computing resource utilization, especially reducing end-to-end communication latency in scenarios with massive small messages.
Smart Images
Figure CN123035237A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of high-performance computing and network communication technology, and in particular to a method and system for offloading the cross-node RDMA communication control plane to a network adapter in aggregated communication. Background Technology
[0002] In intelligent computing platforms, efficient ensemble communication within clusters of neural network accelerators such as GPUs (Graphics Processing Units), NPUs (Neural Processing Units), and XPUs (Heterogeneous Processing Units) is crucial for improving the performance of training large-scale artificial intelligence models and performing inference and training with these large models. Deep learning frameworks use ensemble communication libraries to synchronize or aggregate gradients. Mainstream ensemble communication libraries typically provide various communication primitives such as broadcast, scattering, full collection, full reduction, reduction-scattering, reduction, and full switching. The implementation of these ensemble communication primitives mainly consists of two parts: communication between GPUs within a node is achieved through the PCIe bus or vendor-specific chip-level interconnect technologies (such as NVLink); communication between GPUs across nodes is typically achieved through RDMA (Remote Direct Memory Access) networks. RDMA technology allows computers to directly access remote memory without CPU (Central Processing Unit) intervention, significantly reducing network communication latency and CPU overhead. Common RDMA implementations include InfiniBand, RoCE (RDMA over Converged Ethernet), and iWARP.
[0003] However, with the continuous expansion of large-scale models and the increasing size of clusters, the overhead of cross-node ensemble communication has become a significant bottleneck restricting the overall performance of the system. Taking ResNet training as an example, communication time can account for more than 40% in a 128-card cluster. In cross-node communication scenarios, implementing ensemble communication usually requires the use of ensemble communication libraries provided by GPU manufacturers. Currently, the mainstream ensemble communication libraries' cross-node RDMA implementation schemes mainly fall into the following two categories:
[0004] The first type is the CPU-based agent-based implementation (i.e., the IBRC scheme, CPURelayed Communication). This scheme uses an agent thread on the host CPU to manage the control plane logic, including initiating RDMA communication, progress control, work queue construction, and processing queue elements. Specifically, after the GPU generates data, the CPU copies it to host memory, transmits it via InfiniBand, and the receiving CPU then copies the data back to GPU memory. While this scheme is suitable for large data block transmission scenarios, the CPU-based agent consumes significant CPU computing resources. Furthermore, in scenarios with a large number of small message communications, control plane latency becomes a performance bottleneck, and the processing capacity of the CPU agent thread becomes a constraint on high-speed communication.
[0005] The second category is the implementation scheme based on GPU-initiated communication (i.e., the GIN scheme or IBGDA scheme, GPU DirectAsync). This scheme utilizes the GPU's streaming multiprocessor cores to directly control network communication. The GPU generates working descriptors through SM (Streaming Multiprocessor) threads and writes them to the GPU memory buffer, with the network interface card completing the data transmission. This scheme avoids CPU involvement, avoiding the CPU bottleneck when sending a large number of small packets, and can reduce latency by several times in small data packet transmission scenarios. However, this scheme consumes SM core resources that the GPU should be using for computational tasks, reducing the effective computing power utilization of the GPU, and while improving communication performance, it also brings high computational resource costs.
[0006] The fundamental drawback of both of these approaches is that achieving cross-node RDMA network communication requires consuming computing resources within the cloud platform or intelligent computing center—whether CPU resources (IBRC solution) or GPU SM core resources (IBGDA solution). Taking large model training as an example, DeepSeekV3 allocates 20 out of 132 SM cores on the H800 GPU specifically for communication tasks, with approximately 15% of the computing resources consumed by the communication control plane. This portion of the occupied computing resources cannot be used for actual model computation, effectively reducing the utilization rate of CPU and GPU computing power and increasing the overall cost of achieving high-speed communication. Furthermore, whether CPU proxy threads or GPU SM cores participate in communication control, it involves additional overhead such as operating system context switching and software scheduling, further deteriorating communication efficiency.
[0007] Therefore, there is an urgent need in this field for a cross-node aggregated communication technology solution that can minimize the occupation of host computing resources (CPU and GPU SM cores) while maintaining low latency and high bandwidth communication performance. Summary of the Invention
[0008] The purpose of this invention is to provide a method and system for offloading the cross-node RDMA communication control plane to the network adapter in aggregated communication, so as to solve the problems existing in the prior art.
[0009] To achieve the above objectives, the present invention provides the following solution:
[0010] This invention provides a method for offloading the control plane of cross-node RDMA communication to a network adapter in aggregated communication, comprising:
[0011] In network hardware that supports remote direct memory access, a collective communication proxy hardware module is set up. The collective communication proxy hardware module replaces the CPU proxy thread running on the host central processing unit with a hardware execution unit and directly manages and controls the progress of remote direct memory access communication between nodes.
[0012] When performing aggregated communication across compute nodes, the aggregated communication library, for each accelerator, directly issues aggregated communication tasks to a shared queue between the aggregated communication library and the network hardware, and writes the queue producer index to the doorbell of the network hardware. The aggregated communication task includes aggregated operation type, source memory base address, destination memory base address, data offset, and data size information, where the destination memory base address is obtained through the control plane of the aggregated communication library.
[0013] After the collective communication agent hardware module senses a new collective communication task through the doorbell, it reads the collective communication task through the direct memory access shared queue, parses the collective communication task through the hardware state machine, generates one or more remote direct memory access work queue requests, and sends the work queue requests to the remote direct memory access engine for execution.
[0014] The aggregate communication agent hardware module tracks the execution status of the work queue request and reports the completion result of the aggregate communication task to the software layer when the preset completion conditions are met.
[0015] Preferably, the collective communication proxy hardware module reads the collective communication task, including:
[0016] When the aggregate communication agent hardware module reads aggregate communication tasks in response to sensing signals, it reads multiple aggregate communication tasks at once through the direct memory access engine.
[0017] Preferably, generating one or more remote direct memory access work queue requests includes:
[0018] The aggregate communication proxy hardware module segments an aggregate communication task into message fragments according to the message size, and combines each fragment into a work queue request.
[0019] Preferably, sending the work queue request to the remote direct memory access engine for execution includes:
[0020] According to the load balancing strategy of the work queue pairs, the work queue requests are placed on multiple work queue pairs;
[0021] The receiving network card uses direct placement technology to achieve out-of-order reception of data corresponding to the work queue requests.
[0022] Preferably, the aggregated communication proxy hardware module tracks the execution status of the work queue request and, when a preset completion condition is met, reports the completion result of the aggregated communication task to the software layer, including:
[0023] The aggregated communication proxy hardware module polls the RDMA completion queue when processing aggregated communication tasks.
[0024] When all work queue elements corresponding to the completion queue elements of the task have been processed, or when an error handling result is received, the completion result or error handling result of the collection communication task is reported to the software layer.
[0025] Preferably, the direct management and control of the progress of remote direct memory access communication between nodes in the aggregated communication includes: the aggregated communication agent hardware module managing the communication queue, managing the memory buffer status, initiating remote direct memory access data transmission requests, processing transmission completion notifications, and completing communication progress control.
[0026] Preferably, the network hardware is an intelligent remote direct memory access network card or a data processing unit.
[0027] Preferably, after the aggregated communication library writes the aggregated communication task into the aggregated communication task queue in the host memory and updates the producer pointer, it updates the doorbell record in the host memory and notifies the network hardware doorbell; after the aggregated communication agent hardware module responds to the doorbell, it reads the aggregated communication task through direct memory access and adds it to the hardware aggregated communication task queue.
[0028] Preferably, the aggregated communication agent hardware module maintains a task cache to track the communication status changes of each incomplete task; after the remote direct memory access engine completes data transmission, it writes a completion queue element to the completion queue, and the aggregated communication agent hardware module polls the completion queue to detect whether the communication operation is completed; when communication is completed, the aggregated communication agent hardware module notifies the accelerator that the task has been completed by updating the status of the aggregated communication task queue.
[0029] Preferably, when the aggregate communication agent hardware module polls and finds that a completion queue element contains a communication error code, it stops the current communication and updates the error code to the hardware aggregate communication task queue; when it polls and finds that a completion queue element indicates success and is the last completion queue element of the current communication, it sets the return code in the hardware aggregate communication task queue element to success and updates the consumer pointer.
[0030] Preferably, during the initialization phase, the aggregate communication library starts and initializes the aggregate communication proxy hardware module on the network hardware through the network hardware driver interface; establishes the aggregate communication topology and performs a full exchange of remote direct memory access keys for all members in the communicator; creates a remote direct memory access connection and notifies the aggregate communication proxy hardware module to create a hardware communicator, and maintains the aggregate communication task queue associated with the hardware communicator.
[0031] The present invention also provides a network adapter having a collection of communication task execution units, comprising:
[0032] The remote direct memory access engine is used to execute data transfer requests;
[0033] The communication agent hardware module is implemented as a hardware execution unit and is connected to the remote direct memory access engine.
[0034] The hardware execution unit may be an ASIC, FPGA, programmable logic device, or embedded processor firmware.
[0035] The aggregated communication proxy hardware module is configured as follows:
[0036] Receive a collection communication task from the collection communication library, the task including collection operation type, source address, destination address and data size information;
[0037] The collection communication task is parsed using a hardware state machine to generate one or more remote direct memory access work queue requests;
[0038] The work queue request is sent to the remote direct memory access engine for execution, and the execution status is tracked to report the completion result to the software layer.
[0039] The network card hardware driver interface includes the address of the communication task queue and the index of the doorbell delivery queue producer provided by the network card.
[0040] The elements of the communication task queue include operation type, source address, destination address, and data size information.
[0041] Preferably, the aggregated communication proxy hardware module is configured as follows:
[0042] When reading collection communication tasks in response to sensing signals, multiple collection communication tasks are read at once through the direct memory access engine;
[0043] Alternatively, a collection communication task can be fragmented according to the message size, each fragment can be combined into a work queue request, and then placed on multiple work queue pairs according to the load balancing strategy of the work queue pairs.
[0044] The present invention also provides a system for offloading the control plane of cross-node RDMA communication to a network adapter in aggregated communication, comprising:
[0045] At least two computing nodes are provided, each containing a host processor, a neural network accelerator, and the aforementioned network adapter. The host processor runs a collective communication library adapted to the network adapter. During cross-node collective communication, the collective communication library directly distributes the collective communication task to the collective communication proxy hardware module in the network adapter. The control signaling and data flow of the cross-node remote direct memory access communication are processed and forwarded through the collective communication proxy hardware module and the remote direct memory access engine within the network adapter, without requiring the host CPU to participate in the scheduling and execution of the CPU proxy thread.
[0046] Preferably, the neural network accelerator or CPU maintains a shared collective communication task queue with the intelligent network hardware, and the collective communication proxy hardware module reads one or more collective communication tasks from the shared collective communication task queue for processing through direct memory access.
[0047] The present invention also provides a collection communication library adapted to a network adapter, the collection communication library being adapted to the aforementioned driver interface; and without a proxy thread directly constructing RDMA work requests, the collection communication task is directly sent to the aforementioned network adapter.
[0048] The present invention achieves the following beneficial technical effects compared to the prior art:
[0049] This invention provides a method and system for offloading the cross-node RDMA communication control plane to a network adapter in aggregated communication. By offloading the aggregated communication proxy hardware module, which implements the RDMA communication control plane function as a hardware execution unit, to the network hardware for execution, physical separation of computation and communication control is achieved. This allows CPU and GPU resources to be fully dedicated to application computation, completely eliminating the CPU overhead and GPU SM core occupancy caused by CPU proxy threads. Simultaneously, hardware offloading reduces overhead such as operating system context switching and software scheduling. Especially for aggregated communication scenarios with massive amounts of small messages, it can effectively reduce end-to-end communication latency and increase aggregated communication bandwidth. Furthermore, it reduces dependence on external high-specification CPUs or GPUs, contributing to the construction of intelligent computing infrastructure with higher energy efficiency and lower total cost of ownership. Attached Figure Description
[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 This is a diagram of the internal logical architecture of the RNIC device;
[0052] Figure 2 A data structure diagram for a collection of communication tasks;
[0053] Figure 3 Interaction diagram for the newly added hardware collection communication module;
[0054] Figure 4 State transition diagram for the newly added hardware collection communication module;
[0055] Figure 5 Flowchart for using the communication interface of the training application set;
[0056] Figure 6 This is a flowchart illustrating the cross-node communication process of the method described in this invention.
[0057] Figure 7 This is a schematic diagram of a collective communication system architecture. Detailed Implementation
[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0059] The purpose of this invention is to provide a method and system for offloading the cross-node RDMA communication control plane to the network adapter in aggregated communication, so as to solve the problems existing in the prior art.
[0060] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0061] Example 1:
[0062] This embodiment uses the implementation of the AllReduce primitive in a GPU cluster using the RoCEv2 protocol as an example. The implementation of other collection communication primitives such as Broadcast, Scatter, AllGather, ReduceScatter, Reduce, and AlltoAll is similar. Those skilled in the art can apply the teachings of this embodiment to other collection communication primitives without creative effort.
[0063] This invention provides a method and system for offloading the control plane of cross-node RDMA communication in aggregated communication to a network adapter. Its core lies in adding a dedicated aggregated communication proxy hardware module within intelligent network hardware such as an RDMA network interface card or data processing unit. This hardware execution unit completely replaces the traditional CPU proxy thread running on the host CPU, directly managing and controlling the RDMA communication progress between nodes in aggregated communication. This aggregated communication proxy hardware module is responsible for managing communication queues, managing memory buffer states, initiating RDMA data transfer requests, processing transmission completion notifications, and performing control plane tasks such as communication progress control. By offloading this control plane work to the network hardware, the CPU overhead and GPU SM core usage caused by CPU proxy threads can be completely eliminated, freeing up more computing resources for actual business calculations while maintaining high-performance communication.
[0064] Figure 1 This is a diagram of the internal logical architecture of the RNIC device, illustrating the internal logical architecture of the intelligent network hardware supporting the solution of this invention. Figure 1 As shown, the intelligent network hardware, in addition to traditional components such as the RDMA protocol processing engine, DMA engine, PCIe interface, and network interface, also includes a new aggregated communication proxy hardware module. This module connects to the host via the PCIe interface and to the external network via the RDMA protocol processing unit and the network interface. The DMA unit is connected to the PCIe interface, the aggregated communication proxy hardware module, and the RDMA protocol processing unit to achieve efficient data transmission. Traditional RDMA communication bypasses this module; new aggregated communication tasks are processed through it.
[0065] Figure 3 The newly added hardware aggregation communication module interaction diagram illustrates the interaction relationships between the aggregation communication proxy hardware module and the CPU, host memory, and RDMA engine. For example... Figure 3As shown, the aggregated communication proxy hardware module maintains a shared hardware aggregated communication task queue with the CPU. Each time an aggregated communication operation is issued, the CPU adds the aggregated communication task to this queue and updates the producer index of the queue. If the queue is full, the issuance fails. When the aggregated communication proxy hardware module receives a doorbell, it copies one or more aggregated communication tasks to the intelligent network hardware for processing via direct memory access. After copying, it updates the consumer index of the shared queue. When processing tasks, the newly added aggregated communication proxy hardware module converts the task into multiple work queue elements and issues them to the RDMA engine. Work queue elements can be sent to the peer node through one or more queue pairs. The sending end includes the message sequence number when constructing the work queue request. The receiving end's network card uses direct placement processing technology to ensure communication ordering, consistent with the out-of-order handling process of adaptive routing. The aggregated communication proxy module can also adaptively select the work queue to send work queue elements based on the work queue load. Furthermore, the newly added aggregated communication proxy hardware module maintains a task cache to track the communication status changes of each incomplete task. When communication is complete, the status of the proxy task queue is updated to inform the GPU that the task is complete.
[0066] Figure 4 To introduce the state transition diagram for the newly added hardware aggregated communication module, after receiving a doorbell signal, the network interface card (NIC) reads the shared aggregated communication task queue element based on the consumer index via direct memory access and writes the data to the hardware communication task queue. The hardware aggregated communication module reads tasks from the hardware queue in a loop, then divides the data to be sent into one or more fragments. A work queue request is constructed based on the fragments and written to the RDMA engine. The module then polls the RDMA completion queue; if an ACK is received from the peer, processing of the next fragment continues until completion. If the peer's response to the work queue request times out or an error event is received from the completion queue, subsequent fragment processing is terminated. When all fragments have completed communication processing, a completion event or error message is written to the aggregated communication completion queue, and the next task is read.
[0067] Figure 5 The flowchart illustrating the use of the collection communication interface in the training application demonstrates the process of calling the collection communication primitive interface at the application level. Since the external interface and behavior of the collection communication library remain unchanged under this solution, upper-layer AI applications can directly integrate the current methods and systems without modification, enabling rapid business integration.
[0068] Figure 6 This is a flowchart of the cross-node communication method described in this invention. It shows in detail the complete process of offloading the RDMA control plane process to the RDMA network card or data processing unit during the aggregate communication process. The following will describe each stage in detail with reference to this flowchart.
[0069] Figure 7 This is a schematic diagram of a collective communication system architecture, which shows the overall system architecture including the solution of the present invention. It includes multiple computing nodes, each node including a host processor, a neural network accelerator, and intelligent network hardware supporting RDMA. The intelligent network hardware includes a collective communication proxy hardware module implemented by a hardware execution unit.
[0070] refer to Figure 6 The detailed communication process of this invention is described below according to different stages.
[0071] During the aggregated communication initialization phase, the aggregated communication library, through a newly added intelligent network hardware driver interface, starts and initializes the aforementioned aggregated communication proxy hardware module on the RDMA network card or data processing unit during network device initialization. Specifically, the aggregated communication library first establishes the aggregated communication topology and performs a full exchange of all members in the communicator, exchanging the RDMA keys of each member to ensure that each node can recognize each other and establish a secure communication relationship. Subsequently, the aggregated communication library creates an RDMA connection when creating the aggregated communicator, and simultaneously calls the new driver interface to notify the newly added aggregated communication proxy hardware module on the network card to create the hardware communicator and maintain the aggregated communication task queue associated with the hardware communicator. Each hardware aggregated communicator corresponds to multiple aggregated communication task queue groups, and a corresponding queue is maintained for each remote node within the hardware communicator to facilitate independent management of communication tasks for different nodes. In addition, the aggregated communication library also calls the driver interface to register the memory region of the accelerator that needs to perform RDMA communication with the intelligent network hardware, enabling the intelligent network hardware to directly access the accelerator's memory region through DMA.
[0072] Continue to refer to Figure 6 During the set communication operation distribution phase:
[0073] 1. The application stores the data to be communicated in a specified location in the GPU's video memory within the accelerator's computing kernel. Once this is complete, the next batch of computations can be processed, thus achieving overlap between communication and computation.
[0074] 2. The GPU writes the working descriptor corresponding to the data into the agent task queue and updates the producer pointer of the queue. Each element in the queue contains information such as the set operation type, source address, destination address, and data size. If the number of elements in the queue exceeds a pre-set maximum value, communication is blocked or fails, and an error is returned; otherwise, the application continues with subsequent business processing.
[0075] 3. The aggregate communication library divides aggregate communication tasks into two parts for processing: intra-node communication uses the PCIe protocol or chip-level interconnect protocol, and this part of the communication process remains unchanged;
[0076] 4. If cross-node communication exists, the collection communication library updates the collection communication agent queue producer pointer to the doorbell record in the host memory, backs up the doorbell to prevent it from being lost, and then notifies the smart network hardware doorbell.
[0077] 5. After the smart network hardware responds to the doorbell, it reads the aggregated communication task from the host memory shared queue via direct memory access and adds it to the newly added hardware aggregated communication task queue. If the hardware queue is full, the group will wait.
[0078] 6. The network interface card (NIC) hardware collection communication agent hardware task periodically rotates and consumes tasks in the hardware collection communication task queue.
[0079] 7. Specifically, the network interface card (NIC) hardware aggregated communication agent task creates work queue elements based on task elements and stores them in the intelligent network hardware's transmit work queue. The relationship between the aggregated communication task and the work queue elements is one-to-many; the NIC can split application messages into multiple RDMA work queue elements. The NIC can select one RDMA queue pair for message transmission and reception, or it can select multiple queue pairs. When multiple queue pairs are selected, the appropriate queue transmission / reception work queue element can be adaptively selected based on the load of each queue pair. In this case, the receiver needs to implement order preservation functionality. This work queue element indicates the type of requested operation, such as an RDMA write operation, and includes the source address, destination address, data size, and other network information.
[0080] 8. The network interface card (NIC) hardware collection communication agent task notifies the RDMA engine that there is a new work queue request.
[0081] 9. The RDMA engine of the intelligent network hardware reads work queue elements.
[0082] 10. The RDMA engine copies the data that needs to be communicated from the GPU memory via DMA based on information such as the source address and data size in the work queue element.
[0083] 11. The RDMA engine writes work queue elements and data to remote nodes via the RDMA protocol.
[0084] 12. After the remote node's intelligent network hardware receives the work queue element and completes the data operation, it sends a response confirmation message to the local node.
[0085] 13. After receiving the acknowledgment message, the RDMA engine of the local intelligent network hardware writes a completion queue element to the hardware completion queue buffer, indicating that the network operation has been completed. If no acknowledgment message is received from the remote node or a communication error occurs, the RDMA engine will also write a completion queue element to the completion queue, which contains the specific communication error code.
[0086] 14. The network interface card (NIC) hardware set communication agent task polls the completion queue buffer to check if there are any completion queue elements in order to determine whether the network operation of the corresponding node is completed.
[0087] 15. When a communication error code is detected in a completion queue element, the current communication is stopped. Simultaneously, the communication error code is updated in the hardware aggregated communication task queue, and the consumer pointer of the hardware aggregated communication task queue is updated. When an error code in a completion queue element indicates success, and it is not the last completion queue element of this communication, polling continues to wait for subsequent completion queue elements. Upon receiving the last completion queue element of this communication, the return code in the hardware aggregated communication task queue element is set to success, and the consumer pointer of the hardware aggregated communication task queue is updated. Subsequently, the network interface card hardware aggregated communication proxy task performs an atomic write operation to ensure data writing is complete. When all queues corresponding to this aggregated communication task have been processed, this aggregated communication task is considered complete. The processing results from multiple remote nodes are summarized and the task queue elements in the host memory are updated. Simultaneously, the consumer pointer of the host-side task queue is updated, and the communication results in the communicator are refreshed.
[0088] 16. The application calls the collection communication library to asynchronously poll the result of the current communication and proceed to the next step of processing.
[0089] 17. The application calls the collection communication primitive interface again to send the next round of communication according to business needs.
[0090] Since the set communication primitive interface still supports asynchronous interfaces under the current scheme, it will not block the application while communicating, and can perform business logic such as weight calculation in parallel. Therefore, the current scheme can achieve effective superposition of calculation and communication.
[0091] When the collective communication ends, the relevant resources need to be released, including the communicator, the collective communicator queue, and the buffer in the queue in the intelligent network hardware.
[0092] The invention will be further described in detail below with reference to a specific example. This example uses the implementation of the AllReduce operation of the set communication primitive in a GPU cluster using the RoCEv2 protocol as an example. The implementation of other set communication primitives is similar.
[0093] In terms of the implementation environment, the hardware configuration consists of a server equipped with a GPU and an RDMA smart network card that supports the scheme of this invention; the software configuration consists of a collection communication library that supports the scheme of this invention and a network card driver that supports the scheme of this invention.
[0094] During the production of RDMA smart network card hardware, the logic for implementing the collective communication agent task and the hardware execution unit is first burned into the smart network card, enabling the smart network card to have the collective communication agent hardware function.
[0095] During the deployment phase, the existing collection communication library is replaced with a collection communication library that supports the scheme of this invention. The driver is updated to support the driver program of this invention.
[0096] During the initialization phase of the aggregated communication, the aggregated communication library writes the memory address of the shared queue into the network card hardware through the newly added driver interface.
[0097] During the execution phase of the aggregate communication, the user program calls the aggregate communication primitive AllReduce. The modified aggregate communication library no longer creates agent threads on the CPU at runtime. Instead, it performs the following operations: First, for each accelerator across compute nodes, the aggregate communication task parameters for this AllReduce operation are written to an independent aggregate communication shared task queue in the host memory. These task parameters include the operator, source data memory base address, destination data memory base address, source memory handle, destination memory handle, data size, etc. See the binary layout below. Figure 2 The current operation type is AllReduce; secondly, a command is written to the doorbell register of the smart network card to notify it that there is a new communication operation to be processed. After the collective communication agent hardware in the smart network card firmware is triggered by the doorbell, it reads the collective communication task information in the shared queue and begins to coordinate the data exchange and protocol process between the nodes. The specific data flow follows the steps described in the aforementioned technical solution, that is, the collective communication agent hardware module reads the collective communication task through DMA, converts the task into a work queue element and sends it to the RDMA engine, the RDMA engine copies the data from the GPU memory through DMA according to the work queue element and sends it to the remote node, the remote node returns an acknowledgment message after completing the operation, the local end completes the writing of the completion queue element, and the collective communication agent hardware module polls the completion queue and updates the task status.
[0098] During the completion and asynchronous processing phase, after the last AllReduce operation is initiated, the AllReduce library sends the task to the queue and returns immediately, allowing the application to continue processing other business logic, thus achieving overlap between communication and computation. The smart network interface card firmware confirms the completion of the entire AllReduce operation and refreshes the consumer pointer of the AllReduce queue in the host memory. The application then polls the status of the last AllReduce operation during the next AllReduce operation, or through a dedicated thread calling an interface, to confirm whether the communication was complete.
[0099] In terms of upper-layer processing, since the modified collection communication library interface has not changed, the upper-layer business is unaware of the changes in the underlying communication mechanism, and the business can quickly integrate the current methods and systems.
[0100] Furthermore, this invention provides several alternative implementation schemes. The aforementioned schemes are essentially zero SM core occupancy schemes, minimizing the occupation of GPU computing resources by the communication control plane. The RMA-based scheme for aggregated communication libraries is also a zero SM core occupancy scheme, which can alleviate GPU resource occupation to some extent. However, this scheme still requires a CPU agent to perform control plane logic such as work queue element construction, queue element processing, and communication progress control, thus not completely solving the technical problem addressed by this invention. In another alternative scheme, the aggregated communication agent hardware module can also be implemented through firmware logic rather than pure hardware circuits. That is, a dedicated firmware program runs on the embedded processor of the intelligent network hardware to implement the aggregated communication agent function. This scheme can also offload control plane work from the host CPU and GPU to the network hardware for execution, but its processing performance and latency are slightly inferior to the pure hardware circuit implementation scheme. In yet another alternative scheme, the aggregated communication task queue can be maintained in the host's system memory rather than the GPU's HBM. The aggregated communication agent hardware module reads aggregated communication tasks from the host system memory via DMA. This scheme is more applicable when GPU memory resources are scarce. All of the above alternative schemes fall within the protection scope of this invention.
[0101] This invention adds a dedicated aggregated communication proxy hardware module within intelligent network hardware such as RDMA network interface cards or data processing units. This hardware execution unit completely replaces the traditional CPU proxy thread running on the host CPU, directly managing and controlling the RDMA communication progress between nodes, thus achieving physical separation of computation and communication control. This solution allows CPU and GPU resources to be fully dedicated to application computation, completely eliminating the CPU overhead and GPU SM core usage caused by CPU proxy threads. Simultaneously, hardware offloading reduces overhead such as operating system context switching and software scheduling. Especially for aggregated communication scenarios with massive amounts of small messages, it effectively reduces end-to-end communication latency and increases aggregated communication bandwidth, contributing to the construction of intelligent computing infrastructure with higher energy efficiency and lower total cost of ownership.
[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0103] It should be noted that the components mentioned in the above embodiments are all general standard parts or components known to those skilled in the art. Their structures and principles can be learned by those skilled in the art through technical manuals or conventional experimental methods.
[0104] This invention has illustrated its principles and implementation methods using specific examples. The descriptions of these embodiments are merely illustrative of the method and its core ideas; furthermore, those skilled in the art will recognize that modifications may be made to the specific implementation methods and application scope based on the principles of this invention. Therefore, the content of this specification should not be construed as limiting the invention.
Claims
1. A method for offloading the control plane of cross-node RDMA communication to a network adapter in aggregated communication, characterized in that, include: In network hardware that supports remote direct memory access, a collective communication proxy hardware module is set up. The collective communication proxy hardware module replaces the CPU proxy thread running on the host central processing unit with a hardware execution unit and directly manages and controls the progress of remote direct memory access communication between nodes. When performing aggregated communication across compute nodes, the aggregated communication library, for each accelerator, directly issues aggregated communication tasks to a shared queue between the aggregated communication library and the network hardware, and writes the queue producer index to the doorbell of the network hardware. The aggregated communication task includes aggregated operation type, source memory base address, destination memory base address, data offset, and data size information, where the destination memory base address is obtained through the control plane of the aggregated communication library. After the collective communication agent hardware module senses a new collective communication task through the doorbell, it reads the collective communication task through the direct memory access shared queue, parses the collective communication task through the hardware state machine, generates one or more remote direct memory access work queue requests, and sends the work queue requests to the remote direct memory access engine for execution. The aggregate communication agent hardware module tracks the execution status of the work queue request and reports the completion result of the aggregate communication task to the software layer when the preset completion conditions are met.
2. The method according to claim 1, characterized in that, The aggregated communication proxy hardware module reads the aggregated communication task, including: When the aggregate communication agent hardware module reads aggregate communication tasks in response to sensing signals, it reads multiple aggregate communication tasks at once through the direct memory access engine.
3. The method according to claim 1, characterized in that, The generation of one or more remote direct memory access work queue requests includes: The aggregate communication proxy hardware module segments an aggregate communication task into message fragments according to the message size, and combines each fragment into a work queue request.
4. The method according to claim 3, characterized in that, Sending the work queue request to the remote direct memory access engine for execution includes: According to the load balancing strategy of the work queue pairs, the work queue requests are placed on multiple work queue pairs; The receiving network card uses direct placement technology to achieve out-of-order reception of data corresponding to the work queue requests.
5. The method according to claim 1, characterized in that, The aggregated communication proxy hardware module tracks the execution status of the work queue requests and, when preset completion conditions are met, reports the completion result of the aggregated communication task to the software layer, including: The aggregated communication proxy hardware module polls the RDMA completion queue when processing aggregated communication tasks. When all work queue elements corresponding to the completion queue elements of the task have been processed, or when an error handling result is received, the completion result or error handling result of the collection communication task is reported to the software layer.
6. The method according to claim 1, characterized in that, The direct management and control of the progress of remote direct memory access communication between nodes in the aggregated communication includes: the aggregated communication agent hardware module managing the communication queue, managing the memory buffer status, initiating remote direct memory access data transmission requests, processing transmission completion notifications, and completing communication progress control.
7. The method according to claim 1, characterized in that, The network hardware is an intelligent remote direct memory access network card or a data processing unit.
8. The method according to claim 1, characterized in that, The collective communication library writes the collective communication task into the collective communication task queue in the host memory and updates the producer pointer. Then, it updates the doorbell record in the host memory and notifies the network hardware doorbell. After the collective communication agent hardware module responds to the doorbell, it reads the collective communication task through direct memory access and adds it to the hardware collective communication task queue.
9. The method according to claim 1, characterized in that, The collective communication agent hardware module maintains a task cache to track the communication status changes of each incomplete task; after the remote direct memory access engine completes data transmission, it writes a completion queue element to the completion queue, and the collective communication agent hardware module polls the completion queue to detect whether the communication operation is completed; when communication is completed, the collective communication agent hardware module notifies the accelerator that the task has been completed by updating the status of the collective communication task queue.
10. The method according to claim 9, characterized in that, When the aggregate communication agent hardware module polls and finds that a completion queue element contains a communication error code, it stops the current communication and updates the error code to the hardware aggregate communication task queue; when it polls and finds that a completion queue element indicates success and is the last completion queue element of the current communication, it sets the return code in the hardware aggregate communication task queue element to success and updates the consumer pointer.
11. The method according to claim 1, characterized in that, During the initialization phase, the aggregate communication library starts and initializes the aggregate communication proxy hardware module on the network hardware through the network hardware driver interface; establishes the aggregate communication topology and performs a full exchange of remote direct memory access keys for all members in the communicator; creates a remote direct memory access connection and notifies the aggregate communication proxy hardware module to create a hardware communicator, and maintains the aggregate communication task queue associated with the hardware communicator.
12. The method according to claim 1, characterized in that, During the initialization phase, the aggregate communication library starts and initializes the aggregate communication proxy hardware module on the network hardware through the network hardware driver interface; establishes the aggregate communication topology and performs a full exchange of remote direct memory access keys for all members in the communicator; creates a remote direct memory access connection and notifies the aggregate communication proxy hardware module to create a hardware communicator, and maintains the aggregate communication task queue associated with the hardware communicator.
13. The network adapter according to claim 12, characterized in that, The aggregated communication proxy hardware module is configured as follows: When reading collection communication tasks in response to sensing signals, multiple collection communication tasks are read at once through the direct memory access engine; Alternatively, a collection communication task can be fragmented according to the message size, each fragment can be combined into a work queue request, and then placed on multiple work queue pairs according to the load balancing strategy of the work queue pairs.
14. A system for offloading the control plane of cross-node RDMA communication to a network adapter in aggregated communication, characterized in that, include: At least two computing nodes, each computing node comprising a host processor, a neural network accelerator, and a network adapter as described in any one of claims 12 to 13; The host processor runs a collective communication library adapted to the network adapter. During cross-node collective communication, the collective communication library directly distributes the collective communication task to the collective communication proxy hardware module in the network adapter. The control signaling and data flow of the cross-node remote direct memory access communication are processed and forwarded by the collective communication proxy hardware module and the remote direct memory access engine inside the network adapter, without the host central processing unit participating in the scheduling and execution of the CPU proxy thread.
15. The system according to claim 14, characterized in that, The neural network accelerator or CPU maintains a shared collective communication task queue with the intelligent network hardware. The collective communication proxy hardware module reads one or more collective communication tasks from the shared collective communication task queue for processing through direct memory access.
16. A collection communication library adapted to a network adapter conforming to any one of claims 12 to 13, characterized in that, The collective communication library is adapted to the driver interface described in any one of claims 12 to 13; at the same time, instead of a proxy thread directly constructing RDMA work requests, the collective communication task is directly sent to the aforementioned network adapter.