Inter-node communication methods and electronic devices

Abstract

This application discloses an inter-node communication method and electronic device, relating to the field of inter-node communication technology. The method includes: when a first node initializes, it creates a first completion queue group and a first queue pair group to form a first resource pool, and applies for a first memory pool as a unified storage area; when the first node joins a cluster, it sends a node announcement message to other nodes in the cluster; in response to the first node receiving a communication task directed to a second node, it selects a first queue pair from the available queue pairs and allocates a first memory area from the first memory pool; it obtains the second queue pair number and the second remote access key of the second node from a preset connection table; based on the first queue pair and the first memory area, it initiates a remote direct memory access operation to the peer identified by the second queue pair number, using the second remote access key. This solves the technical problem of large inter-node communication latency in related technologies and can improve the efficiency and reliability of inter-node communication.

Description

Technical Field

[0001] This application relates to the field of inter-node communication technology, and in particular to an inter-node communication method and electronic device. Background Technology

[0002] In traditional inter-node communication, each connection establishment requires the dynamic creation of core resources such as queues and queue pairs, and the real-time exchange of metadata such as memory access keys. This results in significant latency for the initial data packet communication. Furthermore, this series of control plane operations frequently requires execution in the operating system kernel mode, consuming substantial CPU (Central Processing Unit) computing resources and introducing significant connection establishment latency. This problem is particularly prominent in scenarios such as high-performance computing and artificial intelligence training, which require frequent short-burst communication among massive numbers of nodes, and has become a key bottleneck restricting the overall performance of distributed systems.

[0003] To address the aforementioned issues, most related technologies focus on optimizing the network protocol stack or network interface card (NIC) hardware, such as improving memory registration efficiency or optimizing driver interrupt handling. However, these solutions have inherent limitations: First, their optimization scope is limited, and they still cannot avoid the core resource creation and metadata exchange processes required for each connection establishment; second, their deployment is complex, usually requiring modification of the operating system kernel or customization of dedicated hardware; most importantly, these solutions are essentially still local accelerations for a single communication process, failing to reconstruct the communication initialization and resource management mechanisms at the system architecture level, and thus unable to achieve "zero handshake" and "zero wait" throughout the entire communication process, making it difficult to fundamentally eliminate the problems of high latency and CPU overhead in the initial packet communication. Summary of the Invention

[0004] This application provides a method and electronic device for inter-node communication, which at least solves the technical problem in the related art that the establishment of inter-node communication requires the dynamic creation of resources and the real-time exchange of metadata, resulting in large data packet communication latency and high processor overhead.

[0005] This application provides an inter-node communication method applied to a first node, the inter-node communication method comprising: In response to the initialization of the first node, a first completion queue group and a first queue pair group are created to form a first resource pool, and a first memory pool is requested as a unified storage area. In response to the first node joining the cluster, a node announcement message is sent to other nodes in the cluster. The node announcement message includes the identifier of the first node, the available queue pairs in the first queue pair group, the address range of the first memory pool, and the remote access key of the first memory pool. In response to the first node receiving a communication task for direct remote memory access to the second node, a first queue pair is selected from the available queue pairs, and a first memory area is allocated from the first memory pool. At the same time, the second queue pair number and the second remote access key of the second node are obtained from the preset connection table. Based on the first queue pair and the first memory area, a direct remote memory access operation is initiated to the peer identified by the second queue pair number, using the second remote access key.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the inter-node communication method described in the following embodiments.

[0007] In response to the initialization of the first node, a first completion queue group and a first queue pair group are created to form a first resource pool, and a first memory pool is requested as a unified storage area. In response to the first node joining the cluster, a node announcement message is sent to other nodes in the cluster. The node announcement message includes the identifier of the first node, the available queue pairs in the first queue pair group, the address range of the first memory pool, and the remote access key of the first memory pool. In response to the first node receiving a communication task for direct remote memory access to the second node, a first queue pair is selected from the available queue pairs, and a first memory area is allocated from the first memory pool. At the same time, the second queue pair number and the second remote access key of the second node are obtained from the preset connection table. Based on the first queue pair and the first memory area, a direct remote memory access operation is initiated to the peer identified by the second queue pair number, using the second remote access key.

[0008] The inter-node communication method provided in this application transforms the traditional on-demand, dynamic resource creation process into a one-time batch pre-creation during the node initialization phase. This forms a first resource pool containing completed queue groups, queue pair groups, and a unified memory pool, thereby fundamentally eliminating the initial latency caused by kernel mode switching, memory page locking, and hardware context configuration required for each communication. Subsequently, when a node joins the cluster, it actively broadcasts a node announcement message containing its own identifier, available queue pairs, and memory access keys. This transforms the traditional real-time metadata exchange into cluster-level pre-synchronization, constructing a distributed pre-set connection table. This allows any node to obtain metadata with zero latency when communication is needed. When a specific communication task is received, the allocation of local resources is simplified to direct acquisition from the pre-set pool, and the acquisition of peer information is simplified to querying the local pre-set connection table. Neither of these requires system calls or network interaction, compressing the originally serial and time-consuming preparation phase into extremely fast concurrent operations. Ultimately, with all necessary resources and information in place, the application can completely skip the traditional kernel connection establishment process and directly construct and submit remote memory direct access work requests, achieving line-speed communication performance for the first packet. In this way, the control plane overhead is completely stripped away and completed in advance, enabling each data communication, especially the first packet communication, to be initiated at a speed close to the hardware limit. This fundamentally solves the technical problems of high overhead and high latency in the first packet communication between nodes, and significantly improves the communication efficiency and system throughput of high-performance cluster networks. Attached Figure Description

[0009] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating an inter-node communication method provided in an embodiment of this application; Figure 2 A flowchart illustrating an inter-node communication method provided in another embodiment of this application; Figure 3 A flowchart illustrating an inter-node communication method provided in another embodiment of this application; Figure 4 This is an internal structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0012] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or network device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or network device. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] RDMA (Remote Direct Memory Access) technology provides a messaging service that allows applications to directly access virtual memory on a remote computer. As a high-performance network communication technology, RDMA offloads network packet processing to the network card without going through the operating system's kernel protocol stack, greatly saving computing resources, increasing network packet throughput, and significantly reducing communication latency.

[0015] RDMA, as a high-bandwidth, low-latency, and low-CPU-utilization communication technology, overcomes many of the shortcomings of traditional TCP / IP networks and is now widely used in data center scenarios such as artificial intelligence, network storage, and high-performance computing.

[0016] In related technologies, completing an RDMA communication requires the participation of both the sending and receiving nodes. Both the sending and receiving nodes involve control and data paths. Specifically, the control path within a node needs to enter kernel mode to prepare various resources required for communication, such as configuring the Completion Queue (CQ) and Queue Pair (QP), and simultaneously instructing the network interface card (NIC) hardware to receive the configuration. The data path within the node is responsible for data transmission and reception, such as RDMAWrite, RDMA Read, RDMA Send, and RDMA Receive. Since the resources required for data transmission and reception have been prepared by the control path, the application can directly call the RDMA Verbs interface to directly hand over remote memory read / write requests to the NIC, avoiding the involvement of the operating system kernel mode. Therefore, this greatly improves communication efficiency and reduces communication latency.

[0017] In scenarios using RDMA technology, there are numerous remote memory access operations, accompanied by frequent cross-node memory interaction requirements. Each node can be considered a physical CPU server or a GPU computing node within an AI server. Each server or computing node has external communication needs, and the communication carrier is a communication network built based on RDMA technology. RDMA communication is a two-way operation, requiring the participation of both CPUs to complete a single communication. As described above, the data path, by bypassing the kernel, can fully leverage the advantages of RDMA technology. However, the communication needs between different nodes still require the intervention of the control path. That is, the software on both the sending and receiving ends need to enter kernel mode through the control path to create the various resources required for communication.

[0018] Therefore, communication between RDMA nodes requires the following two types of operations: 1. Node initialization; 2. Node communication connection establishment (see appendix). Figure 1 (Existing RDMA connection establishment communication diagram). The following describes each point: I. Node Initialization: In traditional RDMA technology, when a communication request is received, the sending and receiving nodes first perform node initialization work, including allocating PD (Protection Domain), requesting data buffer and registering MR (Memory Region), creating CQ (Completion Queue), and creating QP (Queue Pair), to ensure that all kinds of hardware and software resources required for communication are ready.

[0019] Register MR (Memory Region): To register a memory region (MR), the application calls the ibv_reg_mr() interface to initiate a registration request.

[0020] The first step in registering MapReduce is to perform memory page locking. The application locks all physical memory pages corresponding to a size starting from buf by calling the kernel's get_user_pages() mechanism, thus ensuring that the physical addresses remain unchanged during the registration process.

[0021] After memory page locking, a memory translation table (MTT) needs to be built.

[0022] MTT (Mean Time Tolerance) provides a mapping from virtual pages to physical pages. Network cards (NICs) supporting RDMA (Direct Memory Access) technology obtain the physical location of target data by querying the MTT, which is crucial for the NIC to directly access memory. The MTT records physical addresses in units of pages, allowing the NIC to "aggregate" memory pages scattered across different physical locations for efficient DMA operations. The final step in registering MR (Memory Access Register) is generating access keys, including a local key (L-Key) and a remote key (R-Key), used for controlling the NIC's access permissions to local memory regions and remote memory regions, respectively.

[0023] Create a CQ (Completion Queue): Completion Queues (CQs) are a mechanism for hardware to report completion status to software. CQs need to be created before work queues (QPs) are created because QP creation requires a CQ as an input parameter.

[0024] The application calls the ibv_create_cq() interface to create a completion queue. Then, it allocates memory from the operating system as a CQ buffer to store completion queue elements (CQEs), providing a place to store task reports after the hardware completes its tasks. Finally, the application accesses the driver device file, enters the kernel driver, and fills the basic CQ information into memory according to the specified format, so that the hardware can retrieve the CQ's memory address translation table from memory.

[0025] Create a QP (Queue Pair): Queue Pair, abbreviated as QP, means a pair of work queues. QP is a combination of a send queue (SQ) and a receive queue (RQ) used for data transmission.

[0026] SQ: Specifically used to store sending tasks.

[0027] RQ: Specifically used to store receiving tasks.

[0028] The application calls the `ibv_create_qp()` interface to create a QP, specifying the operation type, associated send-complete queue, associated receive-complete queue, and other information. In an RDMA send-return (SQ) process, the sender places the send task in the send queue (SQ). Similarly, the receiver software places the receive task in the receive queue (RQ), informing the hardware where to store the received data in memory. With a QP, the system knows the memory address to write received data and the destination address to write data to when sending data.

[0029] II. Node Communication and Link Establishment: Before actual RDMA communication, both ends of the node establish a communication link (VA, R-Key, QPN) to exchange essential information for RDMA communication. Establishing this link is a crucial step in RDMA communication. The RDMA link establishment operation involves operations in both the sending and receiving directions, as described below.

[0030] After both the sending and receiving ends have completed tasks such as registering the MR, creating the CQ, and creating the QP, they can use sockets in TCP / IP communication to establish a connection and obtain information from the other end. This process is called communication connection establishment. The information exchanged with the other end includes the QP Number, data buffer address, and remote access key (R-Key).

[0031] Unlike physical connections, a connection in RDMA is a communication "pipeline." Once the pipeline is established, data from the sending end will always travel along this pipeline to the other end. That is, after the connection is established, each QP is associated with only the QP of the other remote node. The QP context of a node records the QP information of the remote node, and the two nodes exchange peer information during the connection establishment process.

[0032] If the local end wants to access the remote memory via Write / Read operations, it needs to know the available Virtual Address (VA) and corresponding R-Key information of the remote node. After successful connection establishment, the application can remotely read and write memory data of the remote node through the RDMA interface.

[0033] Therefore, when there is a communication request between nodes, the application needs to call a series of RDMA application programming interfaces (APIs) to complete tasks such as registering the Name Mapper (MR), creating a Query Query (CQ), and creating a Query Pointer (QP). Then, it obtains information about the peer via a socket, including the QPNumber, data buffer address, and remote access key (R-Key). This series of actions forms the basis for subsequent operations such as RDMA Write, RDMA Read, Send, and Receive. These actions belong to the RDMA control path and are executed in kernel mode, consuming a certain amount of connection establishment time. Furthermore, the connection establishment process inevitably incurs CPU computational resource overhead, affecting communication efficiency.

[0034] In scenarios using RDMA technology, there are numerous remote memory access operations, accompanied by frequent cross-node memory interaction requirements. For multi-node communication, each node needs to communicate with another node in the resource pool. Each communication establishment between nodes involves tasks such as registering MR, creating CQ, and creating QP, frequently entering kernel mode and consuming CPU computing resources. In large-scale AI model training, the use of distributed training methods with multi-device parallel computing generates a large number of RDMA inter-node communication requests. These requests are bursty communication requests, and the problem of long connection establishment time is particularly prominent. Increased communication latency will seriously affect training efficiency. In summary, RDMA nodes suffer from excessive first-packet overhead when completing a single communication, specifically manifested as high CPU connection establishment overhead and prolonged first-packet communication time.

[0035] In response to the above technical problems, such as Figure 1 As shown, an embodiment of this application provides an inter-node communication method applied to a first node, which specifically includes the following steps: Step 101: In response to the initialization of the first node, create the first completion queue group and the first queue pair group to form the first resource pool, and request the first memory pool as a unified storage area.

[0036] In response to the initialization of the first node, the first node pre-creates a set of completion queues to form a first completion queue group and pre-creates a set of queue pairs to form a first queue pair group, thereby forming a first resource pool. The first completion queue group is associated with the queue pairs in the first queue pair group. In response to the formation of the first resource pool, the first node calls the registration interface for direct access to remote memory, requests and registers the first memory pool, and performs page locking operations on the physical memory pages corresponding to the first memory pool.

[0037] The first completion queue group refers to a set of completion queues (CQs) that are pre-created by the RDMA driver during node initialization. The CQ is the queue by which the RDMA hardware notifies the software of the operation completion status.

[0038] The first queue pair refers to a group of queue pairs (QPs) that are pre-created in batches during node initialization. Each QP contains a transmit queue (SQ) and a receive queue (RQ), and is the working unit for RDMA data transmission.

[0039] The first resource pool is a collection of resources composed of the pre-created CQ group and QP group, and is managed uniformly at the software level (such as the resource pool manager) to maintain the "idle" and "allocated" status.

[0040] The first memory pool refers to a large contiguous virtual memory region (Memory Region, MR) that is requested and registered all at once during node initialization by calling the ibv_reg_mr() interface. This memory pool has already completed page locking (locking physical memory pages) and generated a unified access key (LKey / RKey) during registration.

[0041] By pre-creating a queue resource pool, the node RDMA driver, in response to initialization instructions, creates multiple completion queues (CQs) in batches to form the first completion queue group, and simultaneously creates multiple queue pairs (QPs) in batches to form the first queue pair group. During the creation process, a logical association is established between CQs and QPs, ensuring that each QP is bound to its corresponding transmit and receive CQ. These pre-created CQs and QPs are placed in a "resource pool" for unified management. The node maintains resource pool information, including a "list of free resources" and a "list of allocated resources". Subsequently, a unified memory pool is pre-registered, meaning the node calls the RDMA memory registration interface (such as ibv_reg_mr()) to request and register a large contiguous virtual memory region as the first memory pool. During this process, the corresponding physical memory pages are locked through the kernel page locking mechanism to ensure address stability, and a unified local access key (LKey) and remote access key (RKey) are generated for the entire memory pool. When an application has RDMA communication requirements, it does not need to apply for additional CQ and QP resources, nor does it need to apply for send and receive buffers again. Therefore, it does not need to perform time-consuming MR registration and memory page locking operations. Instead, it can directly use the resources that have been initialized in the resource pool, reducing the connection establishment time of the first packet communication.

[0042] Thus, the core resource pre-configuration operation performed by the first node during the system startup initialization phase aims to transform the on-demand, dynamic resource creation and registration process in traditional RDMA communication into a one-time batch completion at system startup through a resource pre-configuration mechanism. This eliminates runtime resource creation overhead, avoids repeated memory page locking, achieves rapid resource allocation, and reduces the latency of the first packet communication. The resource preparation phase, which traditionally had to be completed serially before data transmission, is now completed in parallel at system startup, enabling the first packet data to be sent with "zero waiting time," thus laying a solid foundation for the entire high-performance communication solution.

[0043] Step 102: In response to the first node joining the cluster, send a node announcement message to other nodes in the cluster. The node announcement message includes the identifier of the first node, the available queue pairs in the first queue pair group, the address range of the first memory pool, and the remote access key of the first memory pool.

[0044] In response to the first node joining the cluster, the network card driver of the first node actively sends a node announcement message to the multicast address of the cluster; the node announcement message also includes the types of remote memory direct access operations supported by the node and the message timestamp.

[0045] Please refer to this document. Figure 2 The application specifies that nodes can exchange pre-defined connection information. Specifically, after initializing and joining the cluster, the network card driver on each node can proactively send node announcement messages to all nodes in the cluster via multicast, announcing the existence and basic information of the node. These messages can be periodically sent to other nodes in the cluster, for example, every 10 minutes.

[0046] The above node notification message includes the following necessary information: the node ID (identifier of the first node), the operation types supported by the node (such as read / write operations), the available QP number for the node, the virtual address (VA) of the memory pool pre-registered by the node, the remote access key (R-Key) for the node, timestamps, and other information.

[0047] For example, a node announcement message can be shown in Table 1: Table 1

[0048] In one embodiment, in response to the first node receiving a node advertisement message from other nodes in the cluster, the first node parses the message to obtain the identifier of the other node, the list of available queue pairs of the other node, the memory pool address range of the other node, and the remote access key, and records them in a preset connection table maintained locally; wherein, the preset connection table is stored in the network card hardware storage space of the first node.

[0049] The default connection table is a mapping table maintained locally on the first node, used to store communication metadata of other nodes in the cluster. This table uses the "node identifier" as the key and records necessary communication parameters such as the list of available queue pairs, memory pool address range, and remote access key for the corresponding node.

[0050] Network card hardware storage space refers to the dedicated storage area (such as on-chip SRAM or dedicated cache) integrated on the RDMA network card. Its access speed is much higher than that of the host memory, and it can be directly accessed and managed by the network card hardware without the need for CPU intervention.

[0051] When the network interface card (NIC) hardware of the first node receives a node advertisement message broadcast by other nodes in the cluster, the NIC driver layer first parses the message to extract key structured information, including the unique identifier of the sending node, the list of currently available queue pairs for that node, the start and end virtual addresses of its pre-registered memory pool, the remote access key corresponding to the memory pool, and optional extended information such as message timestamps and supported RDMA operation types. After parsing, the system records this communication metadata in a locally maintained preset connection table. This preset connection table acts as a distributed communication contact book, with each record corresponding to a node in the cluster and containing all the parameters necessary to establish RDMA communication with that node. The pre-defined connection table is explicitly stored in the network interface card (NIC) hardware storage space of the first node. This design allows the NIC to directly query peer information from its own storage space when processing subsequent RDMA operations, without having to access the host memory through the PCIe bus, greatly reducing query latency. At the same time, table maintenance can be completed in the background by the NIC hardware or driver, without occupying the host CPU computing resources, achieving true control plane offloading. Even if the host software restarts, as long as the NIC remains powered on, the information in the table can still be retained, thereby accelerating the cluster reconstruction process.

[0052] Thus, this application achieves pre-synchronization of communication metadata, transforming the metadata that must be exchanged in real time through Socket handshake during the traditional RDMA link establishment process into active broadcasting and asynchronous recording during the cluster startup phase, establishing a global pre-connection state; it achieves localized high-speed query, so that when communication with a certain node is needed later, the peer's QP number and RKey can be obtained directly from the preset connection table stored in local hardware with extremely low latency, completely eliminating the link establishment latency of network round trip; and it achieves complete offloading of the control plane, by placing the preset connection table in the network card hardware storage and having it maintained autonomously by the network card, achieving full hardware offloading of link establishment information management, freeing up valuable CPU resources.

[0053] In one embodiment, in response to a scaling operation on the first resource pool or a preset update timer trigger, the first node regenerates and sends a node announcement message; wherein the regenerated node announcement message includes an updated list of available queue pairs in the first queue pair group, and / or an updated address range and remote access key for the first memory pool.

[0054] This application configures the network interface card (NIC) driver (the software interface between the NIC and the operating system for communication and control) to start packet listening after initialization. It records and saves announcement packets sent by other nodes. For node information already recorded, it checks for updates. If the peer node information is not updated, the packet is discarded; otherwise, the updated peer node information is updated and saved locally. When changes occur in the node's resource pool, such as an increase in the number of QPs or an expansion of the memory pool, the application actively notifies the NIC, triggering the proactive sending of node announcement packets.

[0055] Specifically, when the first node performs a size adjustment operation on its first resource pool (such as proactive expansion or reclamation based on a predictive model), or when the preset update timer reaches its trigger period, the system automatically initiates the information synchronization process. Specifically, the first node will regenerate a node announcement message, which dynamically reflects the latest state of the resource pool: its content includes the updated list of currently available queue pairs in the first queue pair group, and / or the updated address range and remote access key of the first memory pool. Through a dual-trigger mode of event-driven and time-driven approaches, it ensures that the pre-connection tables maintained by each node in the cluster can synchronize resource change information in real time and accurately, thereby maintaining the consistency of the global view. This not only allows the cluster to immediately perceive and utilize newly added resources, but also avoids invalid access or communication failures due to outdated information, achieving dynamic adaptive resource management and full lifecycle maintenance. This further improves the information pre-synchronization in this application and ensures the continuous efficiency and stability of RDMA communication under large-scale dynamic load scenarios.

[0056] It is understandable that the sending, receiving, processing, and node information maintenance of the above messages are actively handled by the network card hardware. The node information is recorded in the network card's storage space without the need for CPU involvement, thus fully freeing up the CPU computing unit.

[0057] Step 103: In response to the first node receiving a communication task for remote direct memory access to the second node, the first queue pair is selected from the available queue pairs, and a first memory region is allocated from the first memory pool. At the same time, the second queue pair number and the second remote access key of the second node are obtained from the preset connection table.

[0058] In response to the first node receiving a communication task for remote direct memory access to the second node, the first node, through its resource pool manager, selects a first queue pair from the available queue pairs and allocates a fixed-size memory block from the first memory pool as the first memory region. Upon completion of selecting the first queue pair and allocating the first memory region, the first node returns the queue pair number and the virtual memory address of the first memory region. The first node then queries its locally maintained preset connection table to obtain the second queue pair number and the second remote access key from the second node.

[0059] In this application, when acquiring local resources within a node, the application no longer calls interfaces such as `ibv_create_qp`, but instead requests an available QP and a memory block from the local "resource pool manager". This process involves memory pointer operations, resulting in extremely high efficiency. When querying peer node information, the application provides the target node's identifier, and the system directly queries the locally maintained "preset connection table" to immediately retrieve the peer's QP Number, data cache address, and remote access key (R-Key) information from local memory. If the peer node's preset connection table is not found, a peer information sending notification message is triggered via TCP / IP packets to synchronize the node information. During data transmission, once the QP enters the send-ready state, the application can immediately submit a send queue request and initiate RDMA Write / Read operations. The entire process requires no CPU involvement in kernel-mode system calls and does not require waiting for the exchange of connection establishment data with the peer.

[0060] Specifically, when an application generates a specific communication requirement, the system quickly completes the key operation process of local resource allocation and peer information acquisition. This process compresses the serial and time-consuming resource preparation and negotiation phases in traditional RDMA communication into extremely fast local operations, directly embodying the realization of "zero handshake" communication. The specific process is as follows: When an application on the first node (such as an AI training process) generates an RDMA communication task pointing to the second node, it first triggers rapid local resource allocation: the application calls a simplified resource request interface to submit the request to the resource pool manager running on the first node; the manager immediately queries its maintained "free resource list", that is, selects a free and available queue pair from the available queue pairs in the first queue pair group as the first queue pair, and simultaneously allocates a fixed-size memory block from the first memory pool as the first memory region; after the selection and allocation are completed, the resource pool manager updates the resource status to allocated and returns the number of the first queue pair and the virtual address of the first memory region to the application. The entire process is completed in user space with extremely low latency. Simultaneously or shortly thereafter, the system performs a rapid query for peer information: based on the identifier of the second node provided by the application, the system queries a preset connection table stored in the network card hardware storage space, immediately obtaining the second queue pair number and the second remote access key. Through the above process, the entire resource allocation and information query are completed in user space or directly by the network card hardware, completely bypassing kernel space. This achieves on-demand access to local resources and zero network latency in obtaining peer metadata, enabling the application to synchronously obtain the local QP, local memory address, peer QP number, peer access key, and other elements required for this communication in a very short time.

[0061] Step 104: Based on the first queue pair and the first memory region, initiate a direct remote memory access operation to the peer identified by the second queue pair number, using the second remote access key.

[0062] In response to the readiness of the first queue pair, the first memory region, the second queue pair number, and the second remote access key, the first node constructs a remote memory direct access work request. The work request specifies the local address of the first memory region, the peer target address determined based on the second queue pair number, and the second remote access key. In response to the completion of the work request construction, the first node submits the work request to the send queue of the first queue pair to trigger the network interface card hardware to directly initiate a remote memory direct access write operation or a remote memory direct access read operation.

[0063] Once the system has obtained the first queue pair (local communication unit), first memory region (data buffer), second queue pair number (peer target identifier), and second remote access key (remote access authorization) required for this communication through the aforementioned steps, and confirms that they are all ready, the first node immediately constructs a remote memory direct access work request. This work request is a standardized instruction description block, which explicitly specifies the local virtual address of the first memory region as the source or destination address of the data, the peer target address parsed from the second queue pair number, and the second remote access key used for authorization verification. After the work request is constructed, the first node submits it directly to the send queue of the first queue pair through the user-mode mapped hardware interface. This submission immediately triggers the network card hardware to autonomously execute subsequent processes. The network card DMA engine retrieves the work request from the send queue and, based on the encapsulated parameters, can directly initiate a remote memory direct access write operation (writing data from the first memory region to the peer's specified address) or a remote memory direct access read operation (reading data from the peer's specified address to the first memory region) without further CPU intervention. The entire process completely bypasses the operating system kernel protocol stack, achieving end-to-end pass-through from user-mode request to hardware execution. This enables the sending / receiving of the first data packet to achieve extremely low latency, close to hardware line speed, fundamentally completing the "zero handshake" and "zero wait" of RDMA communication.

[0064] To avoid resource constraints or waste, this application introduces a resource pool prediction mechanism within the node to dynamically adjust the resource pool size. The background program within the node continuously monitors the following information: communication connection frequency, idle CQ / QP resources, and memory pool consumption rate, dynamically adjusting the resource pool accordingly. When it detects that the resource pool is about to run out or the communication load is increasing, it proactively creates new CQ / QP resources in batches and expands the memory pool size; when it detects that communication is idle, it proactively releases the newly created CQ / QP resources and memory pool resources, maintaining the resource pool at its initial size.

[0065] Specifically, the first node continuously monitors the historical usage status and remote memory direct access communication mode data of the first resource pool. The remote memory direct access communication mode data includes at least the frequency of newly created queue pair allocation requests per unit time, the average holding time of memory region blocks, and the ratio of write operations to read operations in remote memory direct access. Based on the prediction model, historical usage status, and communication mode data, the first node predicts the trend of the peak allocation rate of queue pairs and the memory region consumption rate within a preset time window. In response to the predicted peak allocation rate of queue pairs exceeding a preset first threshold for the proportion of idle queue pairs in the first queue pair group, the first node actively creates and initializes new queue pairs and adds them to the first queue pair group. In response to the predicted memory region consumption rate causing the number of idle memory blocks in the first memory pool to fall below a preset second threshold, the first node actively expands the registered scale of the first memory pool.

[0066] The historical usage status of the first resource pool refers to the statistical information on resource allocation and release over a period of time, such as resource utilization curves and allocation success rates.

[0067] Remote Memory Direct Access (RDMA) communication mode data refers to key indicators that reflect load characteristics extracted from actual RDMA communication behavior.

[0068] The frequency of new queue allocation requests per unit of time reflects the activity and pressure of establishing communication connections.

[0069] The average holding time of a memory region block reflects the typical duration for which the data buffer is occupied, indicating memory turnover efficiency.

[0070] The ratio of RDMA write operations to read operations reflects the characteristics of data flow. More write operations mean that data is mainly pushed, while more read operations mean that data is mainly pulled. The two have different resource demand patterns.

[0071] The first node continuously collects its historical usage status and RDMA communication mode data (including queue pair allocation request frequency, average memory block holding time, and read / write operation ratio per unit time) and inputs it into a built-in prediction model for analysis. The prediction model uses algorithms based on time series analysis and / or regression analysis to achieve its prediction function. In one specific embodiment, the model predicts queue pair allocation demand based on recent queue pair allocation request frequency time series, calculates the expected allocation rate within the next preset time window using exponential smoothing or sliding window averaging, and corrects the predicted rate through correlation analysis based on the characteristics of the current communication task, outputting the predicted peak queue pair allocation rate. For memory consumption rate prediction, it analyzes recent memory block allocation and release records, calculates the historical memory consumption rate by combining the average holding time of memory blocks with the number of memory block allocation requests per unit time, and further adjusts the historical rate by analyzing the read / write operation ratio to predict the expected rate within the next preset time window. The system calculates the memory region consumption rate; compares the predicted results with a preset resource capacity threshold and drives decision-making. When the predicted peak rate of queue pair allocation exceeds the preset first threshold that the proportion of idle queue pairs in the first queue pair group can support, the system actively creates and initializes new queue pairs in batches to join the first queue pair group. When the predicted memory region consumption rate will cause the number of idle memory blocks in the first memory pool to fall below a preset second threshold, the system actively expands the registration scale of the first memory pool. Through the above data-driven prediction model and forward-looking resource adjustment mechanism, this application achieves dynamic and accurate matching between resource supply and communication demand. While avoiding communication bottlenecks caused by insufficient resources, it also prevents static over-configuration and waste of resources, thereby improving the resource utilization efficiency of the entire cluster system.

[0072] In one feasible implementation, the first node can continuously analyze the RDMA communication data stream flowing through it, extract and generate a traffic fingerprint of the data stream, which includes at least the data stream size distribution pattern, the communication peer node association pattern, and time periodicity characteristics; based on the traffic fingerprint and historical records, it can predict the specific data stream communication task that is about to occur and the size of the target memory region required; before the predicted communication task actually arrives, the first node actively pre-allocates a memory region matching the predicted size from the first memory pool and marks its status as warm-up ready; it sends a resource preparation request to the second node that made the prediction, triggering the second node to prepare the receive buffer in advance; based on the communication peer node association pattern, it pre-selects a low-latency or high-bandwidth physical network path for the predicted communication task and associates the selected path identifier with the predicted task.

[0073] Here, traffic fingerprint refers to the set of features extracted from the RDMA communication data stream flowing through the first node that characterizes the unique pattern of the data stream. The data stream size distribution pattern can be, for example, a fixed size (e.g., 4MB), an exponential distribution, or a burst of large flows. The communication peer node association pattern refers to whether this node always communicates regularly with a few specific nodes (e.g., a parameter server). The time periodicity feature refers to whether communication occurs at fixed time intervals (e.g., every 100ms) or is synchronized with the training iteration cycle.

[0074] By introducing a traffic fingerprint-driven predictive coordination mechanism, systematic preparatory operations are initiated before the actual arrival of the predicted task. The first node, based on accurately predicted memory requirements, proactively pre-allocates and marks warm-up buffers from its local memory pool; simultaneously, it sends a lightweight resource preparation request to the second node, triggering synchronous preparation of the receiving buffer on the other end, achieving coordinated pre-readiness on both sending and receiving sides. Furthermore, the system pre-selects and binds the optimal physical network path for the predicted task based on the communication mode and real-time topology. This series of coordinated preparatory operations completely advances control plane operations such as resource allocation and path selection to before the actual generation of the data plane task, effectively eliminating the latency introduced by on-site resource allocation and real-time path calculation in traditional communication processes. This allows end-to-end communication latency to approach the forwarding limit of physical hardware, achieving a performance leap from milliseconds to microseconds. Secondly, the task-level accurate prediction based on traffic fingerprints makes resource preparation highly targeted and accurate, avoiding resource idleness or waste that may result from coarse-grained pre-allocation based on statistics. This significantly improves the utilization efficiency of memory and network resources while ensuring low latency.

[0075] In one specific implementation, such as Figure 3 As shown, taking a training cluster with 4 computing nodes (nodes 0 to 3) as an example, the steps of the inter-node communication method described in this application can be as follows: 1. System startup and resource pre-allocation: Upon startup, each node's RDMA driver automatically performs initialization operations. Specifically, the driver pre-creates 512 queue pairs (QPs) and 16 completion queues (CQs) in batches. These pre-created QPs and CQs are centrally managed, forming the node's resource pool. Simultaneously, the driver pre-allocates and registers a contiguous 4GB virtual memory region as the node's memory pool by calling the ibv_reg_mr() interface. This memory pool is logically divided into 4096 1MB memory blocks (i.e., MR blocks), and page locking operations for physical memory pages are performed during registration, generating a unified local access key (LKey) and remote access key (RKey).

[0076] 2. Initial startup of the resource prediction and management module: The resource prediction and management module on each node starts with the system. Based on the resource status at node initialization, this module continuously monitors key operational metrics of the node, including but not limited to: communication connection establishment frequency, the number of idle CQ / QP in the resource pool, and the consumption and release rate of the memory pool. Based on this monitoring data and its historical trends, the module's built-in prediction model will predict resource demand within a preset time window and dynamically adjust the resource pool size accordingly (e.g., proactively expanding capacity before a predicted increase in resource demand and timely reclaiming capacity after a decrease in demand).

[0077] 3. Construct the pre-join table: After node 0 starts up and joins the cluster, its network interface card (NIC) driver proactively sends a node advertisement message to the cluster's multicast address. This message contains node 0's identifier, a list of available QPs in its resource pool (e.g., numbered 0x1000 to 0x11FF), the address range of its memory pool, and the corresponding remote access key (e.g., RKey 0xAABBCCDD). Nodes 1, 2, and 3 receive this broadcast message after starting up and joining the cluster. To complete the initial cluster information synchronization, nodes 1, 2, and 3 can each perform a one-time control interaction with node 0 (e.g., through a simplified handshake protocol) to exchange and confirm each other's QP lists and RKeys. After obtaining the information from nodes 1, 2, and 3, node 0 records it in its locally maintained pre-connection table. Similarly, each other node also obtains and records information about all other nodes in the cluster by receiving broadcast messages and performing necessary initial interactions. Ultimately, each node independently maintains a pre-connection table containing metadata about all cluster node communication. It is important to emphasize that this "chain establishment" interaction only occurs during the cluster initialization phase of building the global pre-connection table. Its purpose is to complete the global information synchronization in one go, rather than being a necessary step before each communication.

[0078] 4. Fast communication during cluster training: Suppose a training process on node 0 needs to send gradient data to node 1. The process first requests communication resources from the local resource pool manager. The resource pool manager allocates a free QP (e.g., QPN 0x1001) and a 1MB block of free memory (e.g., virtual address VA 0x7F0010000000) from its managed resource pool. Subsequently, the process provides the identifier of the target node (node ​​1), and the system immediately obtains an available QP number (e.g., 0x2001) and its memory pool remote access key (e.g., 0x11223344) for node 1 by querying the local pre-connection table. Once ready, the RDMA driver efficiently configures the local QP (0x1001) to a transmit-ready state. The training process then constructs an RDMA write operation request, specifying the source data address (local VA), the destination address (the corresponding VA in the node 1 memory pool, e.g., 0x7F1010000000), and the access key (node ​​1's RKey), and submits the request to the local QP's send queue. Upon receiving this request, the network interface card (NIC) hardware directly initiates the RDMA write operation, writing data from the specified memory block on node 0 to the specified memory location on node 1. This process completely skips the kernel-mode connection establishment process and real-time metadata exchange with the peer, which are necessary in traditional models.

[0079] 5. Resource Maintenance: During stable cluster operation, each node maintains the pre-connection table information according to a preset strategy to keep it up-to-date. For example, every 10 minutes, each node's network card driver periodically regenerates and broadcasts a node announcement message indicating its current resource status. Simultaneously, the resource prediction and management module continues to run. If it detects the end of a training task or a significant decrease in cluster load, it instructs the resource pool manager to gradually release allocated but idle QP and MR blocks, reclaiming resources until the system shuts down or awaits the next task cycle. This dynamic adjustment enables elastic resource management.

[0080] In this application, when the application requires RDMA communication, there is no need to create resources temporarily: pre-initialized CQ, QP, and memory blocks in the resource pool are used directly, eliminating time-consuming operations such as ibv_create_cq, ibv_create_qp, ibv_reg_mr, and memory page locking. Resource readiness and metadata acquisition are completed instantly, allowing the first packet of data to be sent with "zero wait". Furthermore, from resource allocation to data transmission, no CPU involvement in kernel-mode system calls is required, nor is there a need to wait for the exchange of connection establishment data with the peer. An intelligent prediction mechanism is also introduced, enabling the resource pool to dynamically scale with cluster load, improving the overall resource utilization and scalability of the system.

[0081] Embodiments of this application also provide an electronic device, such as... Figure 4 As shown, it includes a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above-described inter-node communication method embodiments.

[0082] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described inter-node communication method embodiments when running.

[0083] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), mobile network device, magnetic disk, or optical disk.

[0084] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0085] The inter-node communication method provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method of inter-node communication, characterized by, Applied to the first node, the method includes: In response to the initialization of the first node, a first completion queue group and a first queue pair group are created to form a first resource pool, and a first memory pool is requested as a unified storage area. In response to the initialization of the first node, the first node pre-creates a set of completion queues to form a first completion queue group and pre-creates a set of queue pairs to form a first queue pair group, thereby forming the first resource pool. The first completion queue group is associated with the queue pairs in the first queue pair group. In response to the formation of the first resource pool, the first node calls the registration interface for direct remote memory access, requests and registers the first memory pool, and performs page locking operations on the physical memory pages corresponding to the first memory pool. In response to the first node joining the cluster, a node announcement message is sent to other nodes in the cluster. The node announcement message includes the identifier of the first node, the available queue pairs in the first queue pair group, the address range of the first memory pool, and the remote access key of the first memory pool. In response to the first node receiving a communication task for direct remote memory access to the second node, a first queue pair is selected from the available queue pairs, and a first memory region is allocated from the first memory pool; simultaneously, the second queue pair number and the second remote access key of the second node are obtained from a preset connection table. Based on the first queue pair and the first memory region, initiate a direct remote memory access operation using the second remote access key to the peer identified by the second queue pair number. In response to the first node receiving a node announcement message from other nodes in the cluster, the first node parses the message to obtain the identifier of the other node, the list of available queue pairs of the other node, the memory pool address range of the other node, and the remote access key, and records them in the preset connection table maintained locally; the preset connection table is stored in the network card hardware storage space.

2. The method of inter-node communication of claim 1, wherein, The response to the first node joining the cluster, sending a node announcement message to other nodes in the cluster includes: In response to the first node joining the cluster, the network card driver of the first node actively sends the node announcement message to the multicast address of the cluster.

3. The method of inter-node communication of claim 1, wherein, The step of responding to the first node receiving a communication task for direct remote memory access to the second node, selecting a first queue pair from the available queue pairs, and partitioning a first memory region from the first memory pool includes: In response to the first node receiving a communication task for direct remote memory access to the second node, the resource pool manager of the first node selects a first queue pair from the available queue pairs and allocates a fixed-size memory block from the first memory pool as a first memory region. In response to the completion of selecting the first queue pair and allocating the first memory region, the queue pair number of the first queue pair and the virtual memory address of the first memory region are returned.

4. The method of inter-node communication of claim 1, wherein, The step of obtaining the second queue pair number and the second remote access key of the second node from the preset connection table includes: The first node queries a locally maintained preset connection table and obtains the second queue pair number and the second remote access key of the second node from the preset connection table.

5. The method of inter-node communication of claim 1, wherein, The step of initiating a direct remote memory access operation using the second remote access key to the peer identified by the second queue pair number, based on the first queue pair and the first memory region, includes: In response to the readiness of the first queue pair, the first memory region, the second queue pair number, and the second remote access key, the first node constructs a remote memory direct access work request, wherein the work request specifies the local address of the first memory region, the peer target address determined based on the second queue pair number, and the second remote access key. In response to the completion of the work request construction, the first node submits the work request to the send queue of the first queue pair to trigger the network card hardware to directly initiate a remote memory direct access write operation or a remote memory direct access read operation.

6. The method of inter-node communication of claim 1, wherein, The method further includes: In response to a scaling operation of the first resource pool or a preset update timer being triggered, the first node regenerates and sends the node announcement message. The regenerated node announcement message includes the updated list of available queue pairs in the first queue pair group, and / or the updated address range and remote access key of the first memory pool.

7. The method of inter-node communication of claim 1, wherein, The method further includes: The first node continuously monitors the historical usage status and remote memory direct access communication mode data of the first resource pool. Among them, the remote memory direct access communication mode data includes at least the frequency of allocation requests for newly created queue pairs per unit time, the average holding time of memory region blocks, and the ratio of write operations to read operations in remote memory direct access. The first node calls the prediction model and uses the historical usage status and communication mode data as input to the prediction model, so as to predict the trend of queue allocation peak rate and memory area consumption rate within a preset time window in the future. In response to the predicted peak rate of queue pair allocation exceeding a preset first threshold for the proportion of idle queue pairs in the first queue pair group, the first node actively creates and initializes a new queue pair and adds the new queue pair to the first queue pair group. In response to the predicted memory region consumption rate causing the number of free memory blocks in the first memory pool to fall below a preset second threshold, the first node actively expands the registered size of the first memory pool.

8. An electronic device, comprising: include: Memory, used to store computer programs; A processor, configured to implement the steps of the inter-node communication method as described in any one of claims 1 to 7 when executing the computer program.

Images