Decentralized high-performance multicast method and system based on Gossip and RDMA

Abstract

The invention discloses a decentralized high-performance multicast method and system based on Gossip and RDMA, and relates to the field of computer network communication. The method has better reliability, realizes zero-loss transmission of data packets through message de-duplication, timeout retransmission, ACK / NACK convergence and a dynamic topology adaptation mechanism, ensures data consistency, and solves the problem of insufficient reliability of traditional UDP multicast; the method has the advantages that the real-time performance is high, the delay is low, the throughput is high, the zero copy and kernel bypass characteristics of the RDMA and the batch processing mechanism of the MPMC are combined, the average delay is lower than 10 microseconds, the throughput in a large data packet scene reaches 374GB / s, and the low-delay requirements of high-frequency transaction, high-performance calculation and the like are met; the method has high expansibility, is based on a decentralized architecture of an improved Gossip protocol, does not need core node maintenance topology, supports dynamic joining or quitting of nodes, can adapt to large-scale cluster expansion, and solves the expansibility bottleneck of centralized multicast.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer network communication, in particular to a decentralized high-performance multicast method and system based on Gossip and RDMA. BACKGROUND

[0002] As the core technology of "one-to-many" data distribution in distributed systems, multicast communication is crucial in high-performance computing, high-frequency trading, and Internet of Things. Traditional multicast solutions are mostly based on UDP protocol, but its connectionless nature leads to insufficient reliability, such as packet loss, out-of-order, and lack of congestion control mechanism, making it difficult to meet the requirements of data consistency in large-scale clusters.

[0003] Centralized multicast architecture relies on core nodes to maintain forwarding topology, which has single point of failure risk and scalability bottleneck. As the scale of distributed systems expands, nodes frequently join / exit dynamically, making it difficult for centralized scheduling to adapt to topology changes, resulting in increased communication delay and decreased resource utilization.

[0004] To solve the performance bottleneck of traditional protocol stack, RDMA technology realizes direct access to host memory through "zero-copy" and "kernel bypass" mechanisms, significantly reducing communication delay and CPU overhead. However, RDMA does not support multicast natively, and existing multicast solutions based on RDMA mostly rely on simulated unicast implementation, which has complex resource management and poor compatibility.

[0005] Gossip protocol, as a decentralized propagation model, achieves global synchronization through random information exchange between nodes, with good fault tolerance and scalability. However, traditional Gossip protocols have problems such as redundant message propagation and bandwidth waste, and the combination of Gossip and RDMA is still in the exploratory stage, lacking efficient coordination mechanisms.

[0006] In existing improved solutions, some research attempts to combine Gossip and RDMA, but has the following limitations:

[0007] 1. The communication initialization process is complex, requiring manual management of queue pairs (QPs) and memory registration, with poor compatibility;

[0008] 2. The message propagation mechanism does not fully utilize the hardware acceleration features of RDMA, with limited performance improvement;

[0009] 3. Lack of adaptation to heterogeneous network environments, making it difficult to run efficiently in mixed links such as InfiniBand and Ethernet.

[0010] Therefore, a new solution is needed to address the above problems. SUMMARY

[0011] The application aims to provide a decentralized high-performance multicast method and system based on Gossip and RDMA to solve the technical problems proposed in the background art.

[0012] To achieve the above-mentioned purpose, the application provides the following technical solutions: a decentralized high-performance multicast method based on Gossip and RDMA, at least comprising the following steps:

[0013] S1: building a network framework, which adopts a modular layered architecture;

[0014] S2: establishing a communication initialization mechanism;

[0015] S3: designing a message propagation strategy and performing reliability control;

[0016] S4: designing a performance optimization strategy.

[0017] Further, the network framework comprises a transmission layer, a propagation layer and a control layer;

[0018] The transmission layer realizes data transmission based on an RDMA Verbs interface or an MPMC queue framework, and supports zero-copy communication;

[0019] The propagation layer constructs a multi-hop propagation path of nodes through an improved Gossip protocol, and realizes decentralized information diffusion;

[0020] The control layer is responsible for neighbor node management, reliability confirmation and exception handling;

[0021] Supposing a node set in a cluster, the multicast target is to efficiently distribute source node data to all nodes, and the network framework constructs a propagation topology through a dynamic neighbor selection strategy, each node maintains a local neighbor list, and periodically exchanges state information to adapt to topology changes.

[0022] Further, the S2 at least comprises the following steps:

[0023] S2.1: resource initialization, after the node is started, address information is exchanged through an RDMA connection manager (CM), a protection domain (PD), a completion queue (CQ) and a queue pair (QP) are created, a local memory region (MR) is registered, a remote access key (rkey) is generated;

[0024] S2.2: connection establishment, the QP state is migrated from RESET to INIT, RTR (ReadyToReceive) and RTS (ReadyToSend) in sequence, a reliable connection (RC) or an unreliable datagram (UD) communication path is established;

[0025] S2.3: Perform MPMC adaptation. For nodes that introduce the MPMC framework, encapsulate the underlying resources through the unified communication abstraction layer, automatically adapt to multiple transmission methods, and simplify the initialization process. The transmission methods include at least RDMA and TCP.

[0026] During the initialization mechanism setup phase, node metadata (such as IP, port, QP number) is broadcast via the Gossip protocol to achieve dynamic discovery of neighboring nodes without the need for a centralized registry.

[0027] Furthermore, in S3, a Gossip propagation strategy is used for message propagation;

[0028] The node periodically selects a target node from the neighbor list and spreads the message using a "push-pull hybrid" mode;

[0029] For new messages, the sending node can either write them directly to the receiver's pre-registered memory via RDMAWrite operation, or push them via the MPMC non-blocking interface.

[0030] Furthermore, the reliability control in S3 includes at least the following operations:

[0031] Deduplication is achieved by appending a version number and timestamp to each message. The receiving end then uses local caching to deduplicat the messages, thus avoiding redundant processing.

[0032] A sliding window mechanism is introduced to track unacknowledged messages and trigger a retransmission if no acknowledgment is received within a timeout period.

[0033] By combining ACK / NACK convergence strategies, feedback storms can be reduced;

[0034] The method of dynamically updating the neighbor list to adapt to topology changes ensures reliability. In the event of node failure or link interruption, the neighbor list is updated through the Gossip protocol to automatically rebuild the communication path and ensure continuous propagation.

[0035] Furthermore, the performance optimization strategy includes at least the following:

[0036] Memory management utilizes pre-registered memory pools and buffer reuse mechanisms to reduce the overhead of frequent memory allocation;

[0037] Batch processing mechanism aggregates small messages for batch sending, reducing the frequency of RDMA work queue (WQ) submissions;

[0038] Path selection: The MPMC framework dynamically selects the optimal transmission path based on link quality (e.g., RDMA is preferred over TCP).

[0039] Load balancing uses multi-threaded concurrent message processing to avoid overloading a single node.

[0040] A decentralized high-performance multicast system based on Gossip and RDMA, used for decentralized high-performance multicast methods based on Gossip and RDMA, includes a network framework module, an initialization module, a message processing module, and a performance optimization module;

[0041] The network framework module adopts a modular layered architecture, and the network framework module includes at least a transmission submodule, a propagation submodule, and a control submodule.

[0042] The transmission submodule integrates the RDMAVerbs interface and the MPMC queue framework, supporting zero-copy communication;

[0043] The propagation submodule carries an improved Gossip protocol to construct multi-hop propagation paths;

[0044] The control submodule performs neighbor management, reliability verification, and anomaly handling.

[0045] The initialization module is used to perform resource initialization, connection establishment and MPMC adaptation operations, and to achieve dynamic neighbor discovery by broadcasting node metadata through the Gossip protocol.

[0046] The message processing module uses a "push-pull hybrid" Gossip propagation mode to push messages, and ensures reliability through deduplication, timeout retransmission, ACK / NACK aggregation, and dynamic topology adaptation.

[0047] The performance optimization module is used to implement functions such as pre-registered memory pool reuse, small message batch processing, dynamic path selection, and multi-threaded concurrent processing.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] 1. This invention has good reliability. Through message deduplication, timeout retransmission, ACK / NACK aggregation, and dynamic topology adaptation mechanisms, it achieves zero-loss data packet transmission, ensures data consistency, and solves the problem of insufficient reliability of traditional UDP multicast.

[0050] 2. This invention features low latency and high throughput. Combining the "zero copy" and "kernel bypass" characteristics of RDMA with the batch processing mechanism of MPMC, the average latency is less than 10μs, and the throughput reaches 374GB / s in large data packet scenarios, meeting the low latency requirements of high-frequency trading, high-performance computing, etc.

[0051] 3. This invention has strong scalability. Based on the decentralized architecture of the improved Gossip protocol, it does not require core nodes to maintain the topology, supports dynamic addition or removal of nodes, can adapt to large-scale cluster expansion, and solves the scalability bottleneck of centralized multicast.

[0052] 4. This invention has high compatibility and ease of use. Because the unified communication abstraction layer of the MPMC framework supports multiple transmission methods such as RDMA and TCP, it automatically adapts to heterogeneous network environments and simplifies the communication initialization process, reducing the difficulty of development and deployment.

[0053] 5. This invention has lower CPU overhead. Through pre-registered memory pool reuse and multi-threaded load balancing strategies, the CPU utilization rate is 15% lower than that of native RDMA, reducing server resource consumption and improving the overall cluster resource utilization rate. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of the method of the present invention;

[0056] Figure 2 This is a schematic diagram illustrating the core mechanism of RDMA operation.

[0057] Figure 3 This is a schematic diagram of the communication path for RDMA operation.

[0058] Figure 4 A diagram showing the communication propagation of the Gossip protocol;

[0059] Figure 5 This is a diagram of the architecture of a multitasking MPMC queue multicast system.

[0060] Figure 6 The timing diagram for the MPMC scheme;

[0061] Figure 7 A diagram illustrating the average packet loss rate of three schemes under the same test environment;

[0062] Figure 8 This diagram illustrates the average bandwidth of the two schemes under the same test environment. Detailed Implementation

[0063] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0064] Example 1:

[0065] See Figures 1-6A decentralized, high-performance multicast method based on Gossip and RDMA includes at least the following steps:

[0066] S1: Build the network framework, which adopts a modular and layered architecture;

[0067] S2: Establish communication initialization mechanism;

[0068] S3: Design message propagation strategies and implement reliability control;

[0069] S4: Design performance optimization strategies.

[0070] The network framework includes the transport layer, the propagation layer, and the control layer;

[0071] The transport layer implements data transmission based on the RDMAVerbs interface or MPMC queue framework, supporting zero-copy communication;

[0072] The propagation layer constructs multi-hop propagation paths for nodes through an improved Gossip protocol, achieving decentralized information dissemination.

[0073] The control layer is responsible for neighbor node management, reliability verification, and anomaly handling;

[0074] Given a cluster of nodes, the goal of multicast is to efficiently distribute data from the source node to all nodes. The network framework constructs a propagation topology through a dynamic neighbor selection strategy. Each node maintains a local list of neighbors and periodically exchanges state information to adapt to topology changes.

[0075] S2 includes at least the following steps:

[0076] S2.1: Perform resource initialization. After the node starts, exchange address information through the RDMA Connection Manager (CM), create protection domains, completion queues and queue pairs, register local memory regions (MR), and generate remote access keys (rkeys). The protection domain is PD, the completion queue is CQ, and the queue pair is QP.

[0077] S2.2: Connection establishment. The QP state transitions sequentially from RESET to INIT, RTR (ReadyToReceive), and RTS (ReadyToSend), establishing a reliable connection or an unreliable datagram communication path. A reliable connection is RC, and an unreliable datagram is UD.

[0078] S2.3: Perform MPMC adaptation. For nodes that introduce the MPMC framework, encapsulate the underlying resources through the unified communication abstraction layer, automatically adapt to multiple transmission methods, and simplify the initialization process. The transmission methods include at least RDMA and TCP.

[0079] During the initialization mechanism setup phase, node metadata (such as IP, port, QP number) is broadcast via the Gossip protocol to achieve dynamic discovery of neighboring nodes without the need for a centralized registry.

[0080] S3 uses a Gossip propagation strategy for message propagation.

[0081] The node periodically selects a target node from the neighbor list and spreads the message using a "push-pull hybrid" mode;

[0082] For new messages, the sending node can either write them directly to the receiver's pre-registered memory via RDMAWrite operation, or push them via the MPMC non-blocking interface.

[0083] Reliability control in S3 includes at least the following operations:

[0084] Deduplication is achieved by appending a version number and timestamp to each message. The receiving end then uses local caching to deduplicat the messages, thus avoiding redundant processing.

[0085] A sliding window mechanism is introduced to track unacknowledged messages and trigger a retransmission if no acknowledgment is received within a timeout period.

[0086] By combining ACK / NACK convergence strategies, feedback storms can be reduced;

[0087] The method of dynamically updating the neighbor list to adapt to topology changes ensures reliability. In the event of node failure or link interruption, the neighbor list is updated through the Gossip protocol to automatically rebuild the communication path and ensure continuous propagation.

[0088] Performance optimization strategies should include at least the following:

[0089] Memory management utilizes pre-registered memory pools and buffer reuse mechanisms to reduce the overhead of frequent memory allocation;

[0090] Batch processing mechanism aggregates small messages for batch sending, reducing the frequency of RDMA work queue (WQ) submissions;

[0091] Path selection: The MPMC framework dynamically selects the optimal transmission path based on link quality (e.g., RDMA is preferred over TCP).

[0092] Load balancing uses multi-threaded concurrent message processing to avoid overloading a single node.

[0093] Example 2:

[0094] A decentralized high-performance multicast system based on Gossip and RDMA, used in the decentralized high-performance multicast method based on Gossip and RDMA proposed in Implementation Example 1, includes a network framework module, an initialization module, a message processing module, and a performance optimization module.

[0095] The network framework module adopts a modular and layered architecture, and the network framework module includes at least a transmission submodule, a propagation submodule, and a control submodule.

[0096] The transmission submodule integrates the RDMAVerbs interface and the MPMC queue framework, supporting zero-copy communication;

[0097] The propagation submodule is equipped with an improved Gossip protocol to build multi-hop propagation paths;

[0098] The control submodule performs neighbor management, reliability verification, and exception handling.

[0099] The initialization module is used to perform resource initialization, connection establishment and MPMC adaptation operations, and to achieve dynamic neighbor discovery by broadcasting node metadata through the Gossip protocol;

[0100] The message processing module uses a "push-pull hybrid" Gossip propagation mode to push messages, and ensures reliability through deduplication, timeout retransmission, ACK / NACK aggregation, and dynamic topology adaptation.

[0101] The performance optimization module is used to implement functions such as pre-registered memory pool reuse, small message batch processing, dynamic path selection, and multi-threaded concurrent processing.

[0102] Based on the above embodiments one and two, the following specific application settings are proposed:

[0103] The system of this invention was implemented in the Gossip environment based on RDMAVerbs and MPMC library. The experimental cluster consisted of two servers, each configured with a 6-core 2GHz CPU, 64GB of memory, and an InfiniBand network card that supports RDMA, interconnected by a 40Gbps switch.

[0104] Model implementation based on the method

[0105] Environment configuration: The operating system is Ubuntu 22.04, with Mellanox OFED driver and MPMC library installed, the programming language is C, and concurrency control is implemented using a thread model;

[0106] Parameter settings:

[0107] Gossip propagation time: 100ms;

[0108] Each node maintains a neighbor list of 8-16 items.

[0109] RDMA memory registration size: 4MB-16MB;

[0110] MPMC queue depth: 1024;

[0111] Process Implementation:

[0112] After the node starts, it registers memory via ibv_reg_mr and creates a QP via ibv_create_qp;

[0113] The Gossip module sends messages through the ucp_tag_send_nbx non-blocking interface, and the receiving end listens through the ucp_tag_recv_nbx.

[0114] The reliability module maintains a sliding window for messages, and the timeout retransmission threshold is set to 500ms.

[0115] See Figures 7-8 The following experimental results were obtained:

[0116] The test compared three solutions: UDP multicast, Gossip + native RDMA, and Gossip + MPMC. The results are as follows:

[0117] Throughput: The Gossip+MPMC solution achieves 374GB / s in a 4096-byte packet size scenario, which is significantly higher than the RDMA solution (360GB / s) and the UDP solution (1.2GB / s).

[0118] Latency: The average latency of both RDMA and MPMC schemes is less than 10μs, while the average latency of the UDP scheme is 150μs;

[0119] Reliability: The RDMA and MPMC schemes have a packet loss rate of 0% in 10,000 data packet transmissions, while the UDP scheme's packet loss rate increases to 8% with load.

[0120] CPU utilization: The MPMC solution is 15% lower than native RDMA, simplifying resource management overhead.

[0121] In summary:

[0122] This invention eliminates single points of failure through a decentralized architecture, combines the high performance of RDMA with the scalability of Gossip, and introduces the MPMC framework to improve compatibility, making it suitable for high-reliability data distribution scenarios in large-scale distributed systems.

[0123] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A decentralized high-performance multicast method based on Gossip and RDMA, characterized in that: At least comprising the following steps: S1: build a network framework, the network framework adopts a modular layered architecture; S2: establish a communication initialization mechanism; S3: design a message propagation strategy and perform reliability control; S4: perform performance optimization strategy design. 2.The Gossip and RDMA based decentralized high performance multicast method of claim 1, wherein: The network framework includes a transmission layer, a propagation layer and a control layer; The transmission layer realizes data transmission based on the RDMAVerbs interface or the MPMC queue framework, and supports zero-copy communication; The propagation layer builds a multi-hop propagation path of nodes through an improved Gossip protocol to realize decentralized information diffusion; The control layer is responsible for neighbor node management, reliability confirmation and exception handling; Let the node set in the cluster, the multicast target is to efficiently distribute the source node data to all nodes, and the network framework builds a propagation topology through a dynamic neighbor selection strategy, each node maintains a local neighbor list, and periodically exchanges state information to adapt to topology changes. 3.The Gossip and RDMA based decentralized high performance multicast method of claim 1, wherein: The S2 at least comprises the following steps: S2.1: resource initialization, after the node starts, address information is exchanged through the RDMA connection manager, a protection domain, a completion queue and a queue pair are created, and a local memory area is registered, a remote access key is generated, the protection domain is PD, the completion queue is CQ, and the queue pair is QP; S2.2: connection establishment, the QP state migrates from RESET to INIT, RTR and RTS in turn, and a reliable connection or an unreliable datagram communication path is established, the reliable connection is RC, and the unreliable datagram is UD; S2.3: MPMC adaptation, for nodes introducing the MPMC framework, the underlying resources are encapsulated through a unified communication abstraction layer, various transmission modes are automatically adapted, the initialization process is simplified, and the transmission modes at least include RDMA and TCP; In the initialization mechanism building stage, node metadata is broadcasted through the Gossip protocol to realize dynamic discovery of neighbor nodes without a centralized registration table.

4. The gossip and RDMA based decentralized high performance group communication method of claim 1, wherein: In the S3, a message is propagated through the Gossip propagation strategy; Nodes periodically select target nodes from the neighbor list at random, and diffuse messages in a "push-pull mixed" mode; For new messages, the sending node can directly write into the pre-registered memory of the receiving end through the RDMAWrite operation, or can push through the MPMC non-blocking interface.

5. The gossip and RDMA based decentralized high performance multicast method according to claim 4, characterized in that: The reliability control in the S3 at least includes the following operations: Message deduplication through message version number and timestamp, each message is attached with a version number and a timestamp, and the receiving end deduplicates through a local cache to avoid repeated processing; Sliding window tracking of unconfirmed messages for timeout retransmission, a sliding window mechanism is introduced to track unconfirmed messages, and timeout unconfirmed messages trigger retransmission; Combine ACK / NACK aggregation strategy to reduce feedback storm; Dynamic updating of the neighbor list to adapt to topology changes to ensure reliability, when a node fails or a link is interrupted, the neighbor list is updated through the Gossip protocol, the communication path is automatically rebuilt, and the propagation continuity is ensured.

6. The gossip and RDMA based decentralized high performance group communication method of claim 1, wherein: The performance optimization strategy at least includes the following contents: Memory management, using a pre-registered memory pool and a buffer reuse mechanism to reduce frequent memory allocation overhead; Batch processing mechanism, aggregate small message batch sending, reduce RDMA work queue submission frequency; Path selection, MPMC framework dynamically selects the optimal transmission path according to link quality; Load balancing, through multi-thread concurrent processing of messages, avoid single node overload.

7. A system for gossip and RDMA based decentralized high performance multicast, for the gossip and RDMA based decentralized high performance multicast method according to any of the preceding claims 1-6, characterized in that: It comprises a network framework module, an initialization module, a message processing module and a performance optimization module; The network framework module adopts a modular layered architecture, and at least comprises a transmission submodule, a propagation submodule and a control submodule; The transmission submodule integrates RDMAVerbs interface and MPMC queue framework, supports zero-copy communication; The propagation submodule carries an improved Gossip protocol and constructs multiple propagation paths; The control submodule performs neighbor management, reliability confirmation and exception handling; The initialization module is used for performing resource initialization, connection establishment and MPMC adaptation operation, and neighbor dynamic discovery is realized through Gossip protocol broadcast node metadata; The message processing module adopts "push-pull hybrid" Gossip propagation mode to push messages, and guarantees reliability through deduplication, timeout retransmission, ACK / NACK aggregation and dynamic topology adaptation; The performance optimization module is used for realizing the functions of pre-registration memory pool reuse, small message batch processing, dynamic path selection and multi-thread concurrent processing.

Images