A method and apparatus for improving the communication efficiency of RDMA networks

Abstract

This invention belongs to the field of computer network communication, specifically a device and method for improving the efficiency of RDMA network communication. Based on the characteristics of RDMA network traffic, it designs and employs a traffic-adaptive polling cycle control algorithm. This algorithm adaptively adjusts the polling frequency allocation in polling mode, while simultaneously constraining the maximum and minimum polling frequencies to ensure that network communication threads do not occupy the CPU for extended periods. This solves the problem in traditional simple hybrid polling strategies where the overly simplistic switching rules between polling and event-driven modes lead to frequent, intermittent low-traffic events causing user threads to constantly switch between sleep and wake-up states.

Description

Technical Field

[0001] This invention belongs to the field of computer network communication and relates to a polling and event-driven hybrid method for RDMA networks, specifically a method and apparatus for improving the communication efficiency of RDMA networks. Background Technology

[0002] RDMA, short for Remote Direct Memory Access, is a technology inherited from DMA that allows direct memory access, bypassing the CPU. Two processes running on different hosts can directly read data from each other's memory address space via an RDMA network. Traditional TCP / IP-based network communication technologies require data transmission through multiple software layers, including the operating system kernel and network protocol stack, resulting in repeated data movement within memory and placing a heavy burden on server memory and CPU. RDMA technology, through kernel-bypass and zero-copy, allows user-mode programs to communicate directly through memory, reducing unnecessary overhead during data transmission. In recent years, with the explosive growth of internet users, internet applications have faced massive user requests, leading to excessive system load and increased volatility. Existing network communication is facing bottlenecks; therefore, optimization methods for RDMA networks have significant practical value.

[0003] RDMA's communication model is message-based for network transmission, using queues to manage messages to be sent and received. RDMA queues are divided into three types: Send Queue (SQ), Receive Queue (RQ), and Completion Queue (CQ). SQ and RQ are collectively called the Work Queues (WQ), forming a queue pair (QP). RDMA network transmission operations are primarily queue-related: for example, adding a message to be sent to the send queue, and placing a send completion message in the completion queue after the message is sent, allowing user programs to query the message sending status; similarly, when a message is received in the receive queue, a receive completion message is placed in the completion queue, allowing user programs to check for new messages to process. User programs can query completion events in the CQ in two ways.

[0004] The first completion detection strategy is called busy polling. The user program calls the ibv_cq_poll function to repeatedly poll the completion queue until there is available information. Once a new CQE is available, it can be notified in time. This polling method can react to completion immediately, but it consumes a lot of CPU resources and will generate a high CPU utilization rate. It is suitable for high-throughput data transmission scenarios.

[0005] The second strategy, known as event-driven, involves the user program specifying a completion event channel (ibv_comp_channel) when creating a CQ (Custom Queued Query). It then calls the ibv_get_cq_event interface to wait until the completion event channel notifies the user of a new CQE (Custom Queued Query). If no new CQE is found, calling ibv_get_cq_event will block. This method significantly reduces the utilization of server CPU resources compared to busy polling.

[0006] Because the workload of modern computing center clusters is closely related to business systems, which in turn are positively correlated with social production activities, message middleware clusters are typically responsible for traffic smoothing and trough filling within the overall business system, as well as isolation between subsystems. Therefore, the network load of computing clusters exhibits a significant tidal effect, operating at high throughput for some time to cope with traffic peaks, while operating during periods of low traffic, with little or no traffic.

[0007] When the cluster experiences peak traffic, if the event-driven mode is selected, the IO system will generate a large number of events. Compared to the polling mode, the busy event publishing and notification mechanism consumes a significant amount of CPU resources, greatly reducing overall transmission efficiency. To address this, some researchers have proposed a batch event polling method. Leveraging the characteristic that CQ can be shared by multiple connection channels, multiple messages are processed in batches within a single event, thereby improving transmission efficiency. Additionally, researchers at Baidu bRPC have proposed using event suppression, reducing the frequency of events by simplifying event types.

[0008] When using polling, the system's response time to peak traffic can be improved, ensuring high throughput. However, when traffic is low, the server CPU will still perform a large number of meaningless idle polls, wasting resources. In this case, event-driven mode can free up CPU resources while still ensuring timely processing of any small amount of incoming traffic.

[0009] Based on the above, a better approach is to combine polling and event-driven modes simultaneously, using a suitable switching strategy. During peak traffic periods, polling mode is used to ensure low latency and high throughput for RDMA data communication. Under low load conditions, switching to event-driven mode frees up CPU resources to ensure other computational tasks can continue normally. This maximizes the effective utilization of CPU resources while meeting RDMA transfer rate requirements.

[0010] NAPI (New API) is a high-performance network data processing technology used in the Linux network subsystem. Its goal is to minimize CPU usage due to network interrupts and improve the performance and responsiveness of network applications. NAPI's implementation combines polling and event-driven patterns, representing a simplified hybrid polling strategy.

[0011] In traditional network interrupt handling mechanisms, whenever a network data packet arrives, the network device issues an interrupt request, and the interrupt handler runs on the CPU to process the packet. This results in frequent interrupt handling, which consumes CPU time and can potentially lead to interrupt storms under high load.

[0012] A simplified hybrid polling strategy can significantly reduce CPU usage due to network interrupts, thereby improving system performance. It employs a polling-based model, switching network devices from interrupt-driven mode to polling mode to process network packets during polling. When network packets arrive, they are stored in a memory buffer and then processed during polling, thus avoiding the overhead of interrupt handling. In recent years, hybrid polling strategies, exemplified by Alibaba's X-RDMA, have been used in RDMA communication scenarios. However, the current simplified hybrid polling strategy has relatively simple state switching conditions between polling and event-driven modes. When these methods fail to poll, even if a set of requests arrives at very short intervals, they will immediately return to event mode, leading to poor performance due to interrupt handling. Summary of the Invention

[0013] The purpose of this invention is to provide a method and apparatus for improving the communication efficiency of RDMA networks, in order to solve the problem that in the traditional simple hybrid polling strategy, the switching rules between polling mode and event-driven mode are too simple, resulting in frequent low-traffic events with intermittent characteristics causing user threads to constantly switch between sleep and wake-up states.

[0014] To achieve the above objectives, the present invention adopts the following technical solution:

[0015] A method for improving the communication efficiency of RDMA networks includes the following steps:

[0016] S1. Real-time monitoring and statistics of RDMA traffic in timer interrupts;

[0017] S2. Based on the RDMA traffic in the timer interrupt monitored and counted by S1, confirm whether the communication thread has been awakened by a communication event. If awakened by a communication event, it enters the polling state. At this time, the traffic adaptive polling cycle control algorithm is used to calculate the number of polling times in the polling mode for time period t:

[0018] If the calculated polling count is empty, return to step S1 to obtain the RDMA traffic in the timer interrupt of time period t+1, calculate the polling count of the control polling mode in time period t+1, and if it is still empty, the communication thread switches to event-driven state.

[0019] Furthermore, the calculation method of the adaptive polling cycle control algorithm includes the following sub-steps:

[0020] S2.1. Based on the actual network conditions, set the minimum number of polling attempts N. init Maximum number of polling times N max The polling reference threshold Δ, the change ratio α, and β, where the minimum number of polling iterations N min Let N be the initial number of polling. init The change ratio α > 1, and the change ratio 0 < β < 1;

[0021] S2.2 Select three consecutive sampling points from the monitoring and statistical results, namely the first sampling point, the second sampling point, and the third sampling point; calculate the slope change percentages Slope1 and Slope2 respectively. Slope1 is the slope change percentage between the first sampling point and the second sampling point, and Slope2 is the slope change percentage between the second sampling point and the third sampling point.

[0022]

[0023]

[0024] Among them, Rate n Time represents the instantaneous network rate at sampling point n. n The time at sampling point n is represented by Slope1 and Slope2, which represent the slopes at sampling points 1 and 2 and sampling points 2 and 3, respectively.

[0025] S2.3 Calculate the ratio of slope change between the percentage changes in slope Slope1 and Slope2 to obtain an approximate curvature:

[0026] If the approximate curvature is greater than a preset threshold, the number of polling iterations is increased by a change ratio α.

[0027]

[0028] If the approximate curvature is less than the preset threshold, the number of polling times is reduced by the change ratio β.

[0029]

[0030] Where N is the number of polling iterations before adjustment, and N′ is the number of polling iterations after adjustment. The final result N is obtained by rounding using floor(). finalAnd ensure that it satisfies the interval constraints:

[0031] N min ≤N final =floor(N′)≤N max

[0032] This allows us to obtain the current polling cycle.

[0033] An apparatus for improving the efficiency of RDMA network communication includes: an RDMA traffic monitoring and statistics module for real-time monitoring and statistics in a timer interrupt, and a hybrid polling module for receiving real-time monitoring and statistics results, wherein the polling module is programmed to perform the steps described above for improving the efficiency of RDMA network communication.

[0034] This invention provides a method and apparatus for improving RDMA network communication efficiency. Based on the characteristics of RDMA network traffic, it designs and employs a traffic-adaptive polling cycle control algorithm. This algorithm adaptively adjusts the polling count allocation in polling mode, while simultaneously constraining the maximum and minimum polling counts to ensure that network communication threads do not occupy the CPU for extended periods. Addressing the problem that traditional simple hybrid polling strategies suffer from overly simplistic switching rules between polling and event-driven modes, leading to frequent triggering of events with intermittent low traffic causing user threads to constantly switch between sleep and wake-up states, a retry mechanism is adopted. When the calculated polling count is empty, the process returns to step S1 to obtain the statistical and monitoring results of RDMA traffic in the timer interrupt of time period t+1, calculates the polling count for time period t+1, and if it is still empty, the communication thread switches to event-driven mode. This ensures that intermittent traffic is also handled by the polling mode, guaranteeing reasonable CPU utilization at the communication layer under various network traffic loads and effectively improving RDMA communication efficiency.

[0035] Compared with existing technologies, the present invention has significant improvements in CPU utilization and communication efficiency. Attached Figure Description

[0036] Figure 1 To improve the hybrid polling flowchart;

[0037] Figure 2 A comparison of communication throughput for different strategies;

[0038] Figure 3 A comparison of CPU utilization for different strategies. Detailed Implementation

[0039] like Figure 1 As shown, this embodiment provides a method for improving the communication efficiency of RDMA networks, which is based on adaptive adjustment of traffic characteristics, and includes the following steps:

[0040] S1. Real-time monitoring and statistics of RDMA traffic in timer interrupts.

[0041] S2. Based on the RDMA flow obtained from the timer interrupt in S1, determine whether the communication thread has been awakened by a communication event. If awakened by a communication event, it enters the polling state. At this time, the flow-adaptive polling cycle control algorithm is used to calculate the number of polling cycles in the t-period control polling mode. The calculation method of the adaptive polling cycle control algorithm includes the following sub-steps:

[0042] S2.1. Based on the actual network conditions, set the minimum number of polling attempts N. init Maximum number of polling times N max The polling reference threshold Δ, the change ratio α, and β, where the minimum number of polling iterations N min Let N be the initial number of polling. init The change ratio and reference threshold need to be adjusted according to the actual situation on site, but there are basic range requirements. The change ratio α > 1 and the change ratio 0 < β < 1, with default values ​​of 1.2 and 0.8 respectively. The default value of the reference threshold is 0.1.

[0043] S2.2 Select three consecutive sampling points from the monitoring and statistical results, namely the first sampling point, the second sampling point, and the third sampling point; calculate the slope change percentages Slope1 and Slope2 respectively. Slope1 is the slope change percentage between the first sampling point and the second sampling point, and Slope2 is the slope change percentage between the second sampling point and the third sampling point.

[0044]

[0045]

[0046] Among them, Rate n Time represents the instantaneous network rate at sampling point n. n The time at sampling point n is represented by Slope1 and Slope2, which represent the slopes at sampling points 1 and 2 and sampling points 2 and 3, respectively.

[0047] S2.3 Calculate the ratio of slope change between the percentage changes in slope Slope1 and Slope2 to obtain an approximate curvature:

[0048] If the approximate curvature is greater than a preset threshold, the number of polling iterations is increased by a change ratio α.

[0049]

[0050] If the approximate curvature is less than the preset threshold, the number of polling times is reduced by the change ratio β.

[0051]

[0052] Where N is the number of polling cycles before adjustment, N ′ This is the adjusted polling count. The final result N is obtained by rounding using floor(). final And ensure that it satisfies the interval constraints:

[0053] N min ≤N final =floor(N′)≤N max

[0054] This allows us to obtain the current polling cycle.

[0055] This embodiment employs a traffic-adaptive polling cycle control algorithm in the polling state, which determines the polling frequency based on the rate of change in traffic. When the traffic rate increases, it indicates an increase in network load. If this exceeds a set threshold, the polling frequency is increased accordingly to ensure that a large amount of RDMA communication data can be processed in a timely manner. When the traffic rate of change is less than the set threshold, the current polling frequency is maintained. When the traffic rate decreases, it indicates a decrease in network load. If the decrease rate exceeds a preset threshold, the polling frequency is reduced to alleviate CPU utilization.

[0056] If the calculated polling count is empty, a retry mechanism is initiated. Specifically, the process returns to step S1 to obtain the RDMA traffic statistics and monitoring results for time period t+1, calculates the polling count for the controlled polling mode in time period t+1, and if it is still empty, the communication thread switches to an event-driven state. This prevents the communication thread from immediately entering a sleep state waiting for events; this feature is effective under short-term intermittent traffic. If no new data packets are received within the retry count, it indicates that the network is idle and the current traffic is low. The communication thread then switches to an event-driven state, enters a sleep state, and waits for the next communication event to wake it up.

[0057] This embodiment also provides an apparatus for improving the efficiency of RDMA network communication, including: a traffic monitoring and statistics module for real-time acquisition of RDMA traffic in timer interrupts, and a hybrid polling module for receiving real-time monitoring and statistics results, wherein the polling module is programmed to perform the above-mentioned steps for improving the efficiency of RDMA network communication.

[0058] In this embodiment, N = N max When, it is defined as high flow; N <N max When N is often empty, it is defined as medium traffic; when N is often empty, it is defined as low traffic. The effectiveness of the method for improving RDMA network communication efficiency in this embodiment is verified by examining its operation under different loads.

[0059] (1) Busy workload, i.e., high traffic.

[0060] During peak traffic periods, the scheduler operates in a busy-polling state, with the polling code consistently running in the loop between lines 5 and 17. Due to the high network traffic, N is always fully utilized. In each round of polling, the algorithm retrieves N from the network. max Each completion notification is placed in a pending work queue and processed in batches, thereby achieving minimal processing latency and maximum throughput. Because the algorithm is currently in a polling state, CPU utilization is 100%.

[0061] (2) Intermittent workload, i.e., medium flow rate;

[0062] During periods of moderate traffic, the scheduler still operates in a busy-polling state. However, due to traffic fluctuations, N does not always reach N. max It's possible that during one of the polling rounds, the work queue might be empty, resulting in a return value of zero and triggering a retry mechanism to handle short-term intermittent traffic. Overall, CPU utilization remains at 100% during medium to high traffic, ensuring high data throughput.

[0063] (3) Sudden workload, i.e., low flow rate

[0064] Under low traffic conditions, the scheduler switches between busy polling and event-driven modes. The scheduler defaults to event-driven mode upon initial startup. Whenever a work completion event is received, the event notification mechanism wakes the thread from its sleep state, and the scheduler switches to polling mode to process requests in the queue. Due to low traffic, the queue may become empty and no new data may be available after a period of waiting. In this case, the scheduler puts the thread into a sleep state, relinquishing CPU resources to other processes on the server.

[0065] (4) No load, i.e., no flow.

[0066] In the absence of traffic, the scheduler operates in an event-driven state, waiting for new work events. Threads remain in a dormant state, consuming no server resources.

[0067] As can be seen, the method for improving RDMA network communication efficiency in this embodiment, while simultaneously absorbing the advantages of polling and event-driven modes, avoids the drawback of frequent state changes in the communication framework during low to medium traffic. By monitoring the traffic status and making timely switches, CPU resources in the server are freed up, thereby maximizing the computing resources of the data center.

[0068] For ease of description, the method for improving RDMA network communication efficiency in this embodiment will be referred to as the improved hybrid polling strategy in the following description. The improved polling strategy of this embodiment is compared with the polling strategy, the event-driven strategy, and the simple hybrid polling strategy. The comparison results are as follows:

[0069] The processing strategies within the network framework also affect communication latency. The network is operated under different strategies, and the average latency under each strategy is tested.

[0070] like Figure 2 As shown, comparing the throughput data under each strategy, it can be found that the polling mode always maintains the highest throughput. The improved hybrid polling strategy is generally similar in performance to the polling strategy. The performance of the simple hybrid polling strategy has decreased. The event-driven strategy has relatively poor performance when transmitting small amounts of data, but its throughput is close to that of other strategies when transmitting large amounts of data.

[0071] The analysis reveals that the polling strategy, by occupying the CPU for extended periods to scan data states, can complete data processing in the shortest possible time. During testing, data transmission and reception were relatively intensive; therefore, the improved hybrid polling strategy primarily operated in polling mode, achieving similar performance metrics to the basic polling strategy. The improved hybrid polling strategy's retry mechanism allows it to operate in polling mode for longer, thus achieving higher performance compared to the simple hybrid polling strategy.

[0072] When testing with small datasets, the throughput of the event-driven strategy is significantly lower than other strategies. This is because its event listening mechanism receives a large number of event notifications in a short period, causing the event processing thread to frequently switch between sleep and wake-up states, resulting in additional performance overhead. When testing large data transfers, the switching frequency is relatively low, and the performance becomes close to that of the other three strategies.

[0073] The main advantage of the hybrid polling strategy is that it can free up CPU resources during low to medium traffic. Therefore, we can use different traffic volumes to test CPU utilization under different strategies. By changing the algorithm of the hybrid polling core scheduler, we can adjust the scheduler to make the communication threads work in polling or event-driven states for extended periods, thereby enabling a horizontal comparison of the performance under different strategies.

[0074] CPU utilization test results are as follows: Figure 3 As shown, under peak traffic conditions, the improved hybrid polling strategy performs similarly to the full polling strategy, and both outperform the event-driven strategy. Under low to medium traffic conditions, the improved hybrid polling strategy can free up more CPU resources, even completely relinquishing CPU resources during idle periods. Compared to the polling strategy, it clearly has a greater advantage.

[0075] In summary, the method and apparatus for improving RDMA network communication efficiency proposed in this embodiment is an improvement on the traditional hybrid polling method. By setting traffic characteristics and a maximum polling count constraint, it achieves dynamic control over the number of polling attempts and ensures that network communication threads do not occupy the CPU for extended periods. By introducing a retry mechanism, it solves the problem in traditional simple hybrid polling strategies where frequent low-traffic events with intermittent characteristics cause user threads to constantly switch between sleep and wake-up states. Yes, intermittent traffic is also processed by the polling mode. Compared with existing technologies, this invention significantly improves both CPU utilization and communication efficiency.

Claims

1. A method for improving the communication efficiency of RDMA networks, characterized in that, Includes the following steps: S1. Real-time monitoring and statistics of RDMA traffic in timer interrupts; S2. Based on the RDMA traffic in the timer interrupt monitored and counted by S1, confirm whether the communication thread has been awakened by a communication event. If awakened by a communication event, it enters the polling state. At this time, the traffic adaptive polling cycle control algorithm is used to calculate the number of polling times in the polling mode for time period t: When the calculated number of polling is empty, return to step S1 to obtain the RDMA traffic in the timer interrupt of time period t+1, calculate the number of polling in the polling mode of time period t+1, and if it is still empty, the communication thread switches to event-driven state. The calculation method of the adaptive polling cycle control algorithm includes the following sub-steps: S2.

1. Set the minimum number of polling attempts based on the actual network conditions. Maximum number of polls Polling the reference threshold for changes Change ratio and The minimum number of polling times Initial polling count Change ratio >1, change ratio 0 < <1; S2.2 Select three consecutive sampling points from the monitoring and statistical results, namely the first sampling point, the second sampling point, and the third sampling point; calculate the percentage change in slope for each. and ,in: This represents the percentage change in slope between the first and second sampling points. This indicates the percentage change in slope between the second and third sampling points; ； ； in, This represents the instantaneous network rate at sampling point n. Indicates the time of sampling point n; S2.3 Calculate the ratio of slope change between the percentage changes in slope Slope1 and Slope2 to obtain an approximate curvature: If the approximate curvature is greater than Then according to the change ratio Increase the number of polling cycles: ； in, This refers to the number of polling cycles before the adjustment. This refers to the adjusted number of polling iterations. If the approximate curvature is less than Then, the number of polling iterations will be reduced by the change ratio β: in This refers to the number of polling cycles before the adjustment. This refers to the adjusted number of polling iterations. pass Rounding down the result gives the final outcome. And ensure that it satisfies the interval constraints: The final result Set the number of polling cycles for the current round, and control the number of polling cycles in the polling mode for time period t.

2. An apparatus for improving the communication efficiency of RDMA networks, characterized in that: include: A monitoring and statistics module for real-time monitoring and statistics of RDMA traffic during timer interrupts. And a hybrid polling module for receiving real-time monitoring and statistical results; The hybrid polling module uses a method for improving RDMA network communication efficiency as described in claim 1 to perform polling processing.

Images