Multi-QP scheduling method of RDMA and network card with RDMA function
By using a hierarchical progressive table structure and a location-reordering-location strategy, combined with a priority encoder, the blocking problem in multi-QP scheduling is solved, achieving fast and balanced network communication scheduling and improving resource utilization efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF ACOUSTICS CHINESE ACAD OF SCI
- Filing Date
- 2024-10-15
- Publication Date
- 2026-04-17
AI Technical Summary
In modern cloud data centers, due to the mutual blocking between a large number of concurrent QPs, existing technologies lack effective multi-QP scheduling methods, resulting in low network communication efficiency.
By employing a hierarchical and progressive table structure and a location-reorder-location strategy, combined with a priority encoder, multiple QP work queue elements can be quickly found and processed, achieving fast and balanced scheduling.
It improves the efficiency and performance of network communication, optimizes resource allocation, reduces unnecessary lookups and latency, and achieves balanced processing of multiple concurrent QPs.
Smart Images
Figure CN121887744A_ABST
Abstract
Description
Technical Field
[0001] This invention is applied in the field of RDMA technology, specifically relating to a multi-QP scheduling method for RDMA and a network card with RDMA functionality when multiple RDMA tasks are executed simultaneously. Background Technology
[0002] RDMA (Remote Direct Memory Access) technology allows data to be transferred between remote machines over a network without CPU involvement. All data transmission is offloaded to the RDMA network interface card (RNIC), and data transfer occurs directly between user-space virtual memory and the RDMA NIC, without involving the system kernel. There is no additional data movement or copying. This reduces the complexity and latency of data transmission, thus providing high-throughput, low-latency, high-speed network transmission.
[0003] A queue pair (QP queue pair) consists of a send queue and a receive queue. Send and receive queues are always created as a pair and identified by a queue pair number. The QP is the interface between the software and the RDMA hardware, responsible for storing and managing service requests for RDMA tasks. The send queue is used for send operations, and the receive queue is used for receive operations. The send queue contains work requests pending execution between local memory and another remote memory location, while the receive queue contains pending request information for receiving corresponding work requests from the remote location.
[0004] The host submits a Work Request (WR), which places the instructions for each Work Queue Element (WQE) into the corresponding work queue. The RDMA network interface card (NIC) executes the WQEs in the order they are placed in the work queue. When the RDMA NIC completes a WQE, it places a Completion Queue Element (CQE) into the completion queue. After receiving a response to the corresponding request, the host retrieves the CQE to confirm completion.
[0005] In RDMA technology, the basic unit of communication is the QP (Quality Per Instance), not a node. For each node, each process can use several QPs, and each local QP can be "associated" with a remote QP. That is, RDMA is actually a transfer between a QP on one node and its associated QP on another node. Each QP is independent, isolated by a specific PDProtection Domain memory area. This means that a QP can be considered a resource exclusively for a single user, and a user can use multiple QPs for different data transfer tasks.
[0006] In modern cloud data centers, large-scale distributed applications are typically built on many machines, requiring frequent network communication using a large number of concurrent connections. Therefore, the corresponding RDMA transmission requirements must be carried by a large number of QPs to support concurrent applications and distributed data transmission.
[0007] A large number of concurrent work requests from QPs can lead to mutual blocking between different QPs, thus requiring a fast and balanced scheduling method for multiple QPs. Summary of the Invention
[0008] The purpose of this invention is to overcome the shortcomings of the prior art and to propose a multi-QP scheduling method for RDMA. This invention also discloses an RDMA network card.
[0009] To achieve the above objectives, this invention proposes a multi-QP scheduling method for RDMA, used for scheduling QP queues from the RDMA network card to the host, including:
[0010] Use a record table to register QPs with pending work queue elements; manage QPs by group based on the record table, and maintain a corresponding status flag for each group, constructing a hierarchical and progressive table structure from QP to its group, and then to the group status flag;
[0011] Based on a progressive table structure, a hierarchical search is performed first, and then a positioning-rearranging-positioning strategy is adopted to traverse and complete the scheduling task of all groups.
[0012] Preferably, the hierarchical table structure includes:
[0013] The QP status table is used to track and manage the status of all QPs.
[0014] QP group status table, used for grouping and managing elements in the QP status table;
[0015] The QP group status flag table is a set of flags used to describe the overall status of each QP group.
[0016] Preferably, the table width of the QP status table is equal to the number of QPs, and each bit corresponds sequentially to a QP. When there are work queue elements to be processed in the QP, the corresponding bit is set with a flag bit; otherwise, the corresponding bit is not set with a flag bit.
[0017] Preferably, a read pointer and a write pointer are set in the send work queue of each QP to indicate the execution status of the work queue elements in the send work queue. The read pointer points to the position of the work queue element that has been sent, and the write pointer points to the position of the work queue element that has been sent by the host. When the two pointers coincide, it means that there are no work queue elements that need to be processed in the send work queue.
[0018] Preferably, the QP grouping status table manages the elements in the QP status table in groups, with each group having the same bit width and managing the same number of QPs.
[0019] Preferably, the bit width of the QP group status flag table is equal to the number of QP groups, and each bit corresponds sequentially to one group. When any QP in a group has a work queue element to be processed, the corresponding bit is set with a flag; otherwise, the corresponding bit is not set with a flag.
[0020] Preferably, the progressive table structure first performs a hierarchical search, then employs a location-rearrangement-location strategy to traverse and complete the scheduling task for all groups, including:
[0021] Step 1) Preliminary positioning: According to the QP group status flag table, scan according to the group sequence number, perform fast positioning, find the position where the bit with the smallest sequence number has a flag bit, and obtain the corresponding group sequence number;
[0022] Step 2) In-depth positioning: Based on the found group number, enter the corresponding QP group status table, perform fast positioning, find the position of the bit with the smallest number that has a flag bit, and locate the corresponding QP;
[0023] Step 3) Processing work queue elements: For the located QP, retrieve the work item at the head of the sending work queue and send it, thereby completing the scheduling of a QP and synchronously updating the read pointer position of the sending work queue.
[0024] Step 4) Reordering operation: After processing a work queue element, the corresponding position and all the previous unflagged parts are cyclically shifted, that is, these bits are shifted to the right to the end of the QP group status table;
[0025] Step 5) Repeated location: After rearrangement, repeat the fast location in the new QP group status table to continue to find and process the next pending work queue element, and continue until all QPs with pending work queue elements in the group have been processed.
[0026] Step 6) Return to the flag table: After processing a group, return to the QP group status flag table and perform a rearrangement operation, that is, cyclically shift the unflagged part of the flag table at the current position to the end;
[0027] Step 7) Group processing loop: Repeat steps 1)-6) until all groups in the QP group status flag table have been processed;
[0028] Step 8) Global scheduling complete: retrieve the new QP state table and continue to execute the above steps in a loop.
[0029] Preferably, the rapid positioning in steps 1) and 2) is achieved using a priority encoder.
[0030] On the other hand, the present invention provides a network interface card with RDMA functionality, implemented based on an FPGA, comprising: a table management module and a scheduling module; wherein,
[0031] The table management module is used to register QPs with pending work queue elements using a record table; it manages QPs in groups based on the record table and maintains a corresponding status flag for each group, thus constructing a hierarchical table structure from QP to its group and then to the group status flag.
[0032] The scheduling module is used to perform hierarchical search based on a progressive table structure, and then use a location-rearrangement-location strategy to traverse and complete the scheduling tasks of all groups.
[0033] Compared with the prior art, the advantages of the present invention are:
[0034] 1. The method of the present invention is based on sending work queue pointers, maintaining an independent QP status table, and adopting a hierarchical and progressive table structure as the scheduling basis, allowing for fast lookup based on bit operations;
[0035] 2. This invention manages the QP status in groups and creates a flag indicating whether each group needs to be scheduled, and performs hierarchical searching, thereby reducing unnecessary searches and delays and speeding up execution efficiency;
[0036] 3. A priority encoder is used for rapid positioning, combined with cyclic shift rearrangement, so as to complete QP scheduling quickly and evenly. Attached Figure Description
[0037] Figure 1 It is a hierarchical, progressive table structure diagram;
[0038] Figure 2 It indicates the intent of sending the work queue and status representation;
[0039] Figure 3 It is an execution status instance of the QP group status table and flag table;
[0040] Figure 4 This is the overall flowchart of the execution method;
[0041] Figure 5 This is a schematic diagram of the overall method of the present invention. Detailed Implementation
[0042] This invention proposes a scheduling strategy for QP (Queue Pair) work queue processing on RDMA (Remote Direct Memory Access) network cards. In complex scenarios facing a large number of concurrent QPs, the method includes a progressive table structure: a QP status table, which maintains the presence of pending WQEs based on the send queue status; a QP group status table, which divides the QP status table into multiple groups of equal size; and a QP group status flag table, which indicates whether each group contains pending WQEs. A fast location method based on a priority encoder is used, performing a hierarchical search first on the QP group status flag table and then on the QP group table, thereby completing the QP scheduling of pending WQEs. The aim is to achieve rapid location and quick response to QP requests, and to achieve balanced processing of multiple concurrent QPs, thereby optimizing resource allocation and improving the overall efficiency and performance of network communication.
[0043] This method completes the scheduling process based on a progressive table structure. Specific methods include:
[0044] Two pointers are used to manage the state of the QP transmit work queue. One pointer indicates the position of the WQE queue that has been assigned to the RDMA network card by the host, and the other pointer indicates the position of the WQE that has been retrieved and transmitted by the RDMA network card. When the two pointers coincide, it means that there are no WQEs that need to be processed in that transmit work queue.
[0045] The QP status table is used to track and manage the status of all QPs, and its bit width is equal to the total number of QPs in the system. In the status table, each bit represents a specific QP. If a QP has unsent Work Items (WQEs), the corresponding bit is set to a flag, indicating that the QP has pending work items; otherwise, the position is unflashed. The content of each position in the table is updated in real time based on the comparison result of the two pointers in the corresponding transmit work queue, thus reflecting the real-time status of each sequence number QP.
[0046] The QP grouping state table extracts the contents of the QP state table, divides them into multiple groups, and each group has the same bit width, managing the same number of QPs.
[0047] Simultaneously, a QP packet status flag table is created, with a bit width equal to the number of QP packets. Each bit in the flag table corresponds to a specific QP packet. If none of the QP packets in the packet have any unsent WQEs, the corresponding bit in the flag table is de-flagged; otherwise, the flag is set.
[0048] Then, a hierarchical search method is used, employing a "position-reorder-position" strategy to operate on the QP grouping state table and the QP grouping state flag table, in order to optimize the concurrent processing of a large number of QPs. The specific steps of this method are as follows:
[0049] 1. Preliminary Location: First, based on the QP group status flag table, scan according to the group sequence number to perform fast location and find the position where the smallest sequence number has a flag bit. This step utilizes the flag table to quickly narrow down the search range.
[0050] 2. In-depth location: Once the above-mentioned packet location is found, the system will enter the corresponding QP packet status table. Within the status table, the system will perform rapid location again, searching for the position where the smallest sequence number of the bit has a flag bit. This step ensures that the highest priority QP can be quickly identified among the found packets.
[0051] 3. Handling WQE: For a located QP, the system retrieves the WQE from the head of the QP's send queue and sends it. At this point, one QP scheduling cycle is complete, and the corresponding work queue is synchronously updated to indicate the position of the retrieved and sent WQE.
[0052] 4. Reordering operation: After processing a WQE, the system performs a cyclic shift on the position and all the previous unflagged bits, that is, shifts these bits to the right to the end of the QP group status table.
[0053] 5. Repeated Location: After rearrangement, the fast location step is repeated in the new QP group status table to continue searching for and processing the next pending WQE. This process will continue until all QPs with pending WQEs in the group have been processed.
[0054] 6. Return to Flag Table: After processing a group, the system will return to the QP group status flag table and perform a similar rearrangement operation on the QP group status flag table, that is, cyclically shift the flags at that position and the unflagged parts before it to the end.
[0055] 7. Group processing loop: Continue executing this "position-reorder-position" loop until all groups in the QP group status flag table have been processed.
[0056] 8. Global scheduling complete: Once all packets in the QP packet status flag table have been processed sequentially, the entire scheduling process ends. Then, a new QP status table is retrieved, and the above steps are repeated in a loop.
[0057] The aforementioned fast positioning method refers to using a priority encoder, which can accept a multi-bit binary number and quickly determine the least significant bit of the set flag bit through bitwise operations. It works by comparing the high and low values of the input bits, prioritizing the flag bit with the lowest value.
[0058] The scheduling implementation method based on this invention has certain expansion space on the basis of basic sequential scheduling. In application, the input QP state table can be modified according to specific needs to achieve a more flexible scheduling method. For example, mapping a certain QP number to other QP IDs can achieve non-sequential scheduling. Or, increasing or decreasing the corresponding bit width in the state table of a certain QP can achieve weighted scheduling, etc.
[0059] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.
[0060] Example 1
[0061] The present invention proposes a multi-QP scheduling method for RDMA in Example 1.
[0062] This method proposes a scheduling strategy for QP (Queue Pair) work queue processing on RDMA (Remote Direct Memory Access) network cards. It utilizes a hierarchical lookup process based on a progressive table structure to complete the scheduling process, achieving rapid location and response to QP requests, and balanced processing of multiple concurrent QPs, thereby optimizing resource allocation. The described progressive table structure is shown in the figure below. Figure 1 As shown.
[0063] The following describes the specific execution logic of this design method based on the table update and processing order, with examples:
[0064] The QP status table is used to track and manage the status of all QPs. The table's bit width corresponds to the number of QPs, and each bit represents the QP with that sequence number. The value of the QP status table is determined by the status of the corresponding QP's send work queue. If a QP has unsent Work Queue Entry (WQE), the corresponding bit will be set as a flag, indicating that the QP has pending work items; otherwise, the bit will remain unflashed.
[0065] The send work queue status is managed by two pointers. The write pointer indicates the position of the WQE queue that the host has assigned to the RDMA network card, and is updated when the host issues a work request. Its initial value is 0, pointing to the head of the queue. If the host issues n WQEs in this QP, this value is updated to n.
[0066] The read pointer indicates the position of the WQE that the RDMA network card has retrieved and sent. Its initial value is also 0, pointing to the head of the queue. When QP scheduling is completed and the WQE for that QP is confirmed to have been sent, this pointer is incremented by one.
[0067] like Figure 2As shown in the schematic diagram of the transmit work queue and status representation, when the two pointers coincide, it means that there is no WQE to be transmitted in the corresponding QP, and the corresponding bit in the status table is deactivated (e.g., QP 6). Otherwise, the corresponding bit in the status table is deactivated (e.g., QP 1024).
[0068] The QP grouping state table divides the extracted QP state table content into multiple groups, with each group having the same bit width, thus managing the same number of QPs within each group.
[0069] The QP packet status flag table is updated synchronously with the QP packet table. The bit width of this flag table is equal to the number of QP packets. Each bit in the flag table corresponds to a specific QP packet. If none of the QPs within a packet have any unsent WQEs (meaning no flag is set in the packet table), the corresponding bit in the flag table is de-flagged using bitwise operations. If any QP within a packet has unsent WQEs (meaning a flag is set in the packet table), the corresponding bit in the packet status table is set. For example... Figure 3 As shown.
[0070] A hierarchical search method is used to operate on the aforementioned grouping table and flag table. This method employs a "position-reorder-position" strategy to optimize concurrent processing of a large number of QPs. The following section uses... Figure 3 For example, the specific steps of this method will be described in detail:
[0071] 1. Preliminary location: First, perform a fast location on the QP group status flag table to find the bit with the smallest sequence number whose flag is set, that is, QP group n has a QP that needs to be scheduled.
[0072] 2. In-depth positioning: The system will enter the state table of the corresponding QP group n. Within the group state table, the system will perform fast positioning again to find the bit with the smallest sequence number whose flag is set to QP 1024.
[0073] 3. Process WQE: Retrieve the WQE from the head of the QP 1024 send queue and send it. At this point, one QP scheduling cycle has been completed, and the corresponding work queue is updated to indicate the position of the WQE that has been retrieved and sent.
[0074] 4. Reordering Operation: After processing a WQE, the system cyclically shifts the QP position and all previously unset flag bits, moving these bits to the right to the end of the group status table. In this example, only QP1024 is moved to the end of the group table.
[0075] 5. Repeated Location: After rearrangement, the fast location step is repeated in the new group status table to continue searching for and processing the next pending WQE. In this example, QP1025 will be located, and after issuing the WQE, the process continues until all QPs (1026, 1027) in this group with pending WQEs have been processed.
[0076] 6. Return to Flag Table: After processing a group, the system returns to the flag table and performs a similar rearrangement operation on the flag table. In this example, the unflagged portions of the QP group n in the flag table are cyclically shifted to the end.
[0077] 7. Perform preliminary positioning again: obtain the position where the next smallest bit is set to the flag, that is, there is a QP that needs to be scheduled in QP group n+1.
[0078] 8. Execute deep positioning and rearrangement operations in a loop: Send out the WQE of QP1029 and QP1031 in QP group n+1.
[0079] 9. Return to Flag Table: Circularly shift the unflagged portions of the QP group n+1 in the flag table, including those before it, to the end. After performing fast positioning, if no more QPs need processing, the process is complete.
[0080] 10. Update Tables: Read the current QP state table to update the grouping table and flag table.
[0081] The aforementioned fast positioning method refers to using a priority encoder, which can complete the positioning within one cycle. A priority encoder is a digital circuit typically used to convert the priority of an input signal into a binary output. In computer science, it is frequently used to determine which of several events should be processed first. Applying a priority encoder can help quickly locate the least significant bit of a table whose flag is set.
[0082] In summary, the overall execution flow of this scheduling method is as follows: Figure 4 As shown in the diagram, the overall execution is illustrated below. Figure 5 As shown.
[0083] Example 2
[0084] This invention is applied to a network interface card with RDMA functionality, implemented on an FPGA, and adopts the method steps of Embodiment 1, including: a table management module and a scheduling module;
[0085] The table management module is used to register QPs with pending work queue elements using a record table; it manages QPs in groups based on the record table and maintains a corresponding status flag for each group, constructing a hierarchical table structure from QP to its group and then to the group status flag;
[0086] The scheduling module is used to traverse all groups and complete the scheduling tasks based on a progressive table structure, first searching hierarchically, and then using a location-reordering-location strategy.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A multi-QP scheduling method for RDMA, used for scheduling and processing QP queues from the RDMA network interface card to the host, comprising: Use a record table to register the QPs that have pending work queue elements; Based on the record table, QPs are managed in groups, and a corresponding status flag is maintained for each group, thus constructing a hierarchical and progressive table structure from QP to its group and then to the group status flag. Based on a progressive table structure, a hierarchical search is performed first, and then a positioning-rearranging-positioning strategy is adopted to traverse and complete the scheduling task of all groups.
2. The multi-QP scheduling method for RDMA according to claim 1, characterized in that, The hierarchical table structure includes: The QP status table is used to track and manage the status of all QPs. QP grouping status table, used for grouping and managing elements in the QP status table; and The QP group status flag table is a set of flags used to describe the overall status of each QP group.
3. The multi-QP scheduling method for RDMA according to claim 2, characterized in that, The table width of the QP status table is equal to the number of QPs. Each bit corresponds to a QP sequentially. When there are work queue elements to be processed in the QP, the corresponding bit is set as a flag bit; otherwise, the corresponding bit is not set as a flag bit.
4. The RDMA multi-QP scheduling method according to claim 3, characterized in that, In each QP's send work queue, a read pointer and a write pointer are set to indicate the execution status of work queue elements in the send work queue. The read pointer points to the position of the work queue element that has been sent, and the write pointer points to the position of the work queue element that has been sent by the host. When the two pointers coincide, it means that there are no work queue elements that need to be processed in the send work queue.
5. The multi-QP scheduling method for RDMA according to claim 2, characterized in that, The QP grouping status table manages the elements in the QP status table in groups. Each group has the same bit width and manages the same number of QPs.
6. The RDMA multi-QP scheduling method according to claim 2, characterized in that, The bit width of the QP group status flag table is equal to the number of QP groups. Each bit corresponds to a group sequentially. When any QP in a group has a work queue element to be processed, the corresponding bit is set with a flag; otherwise, the corresponding bit is not set with a flag.
7. The RDMA multi-QP scheduling method according to claim 2, characterized in that, The hierarchical table structure, which first performs a hierarchical search and then employs a positioning-rearranging-positioning strategy to traverse and complete the scheduling task for all groups, includes: Step 1) Preliminary positioning: According to the QP group status flag table, scan according to the group sequence number, perform fast positioning, find the position where the bit with the smallest sequence number has a flag bit, and obtain the corresponding group sequence number; Step 2) In-depth positioning: Based on the found group number, enter the corresponding QP group status table, perform fast positioning, find the position of the bit with the smallest number that has a flag bit, and locate the corresponding QP; Step 3) Processing work queue elements: For the located QP, retrieve the work item at the head of the sending work queue and send it, thereby completing the scheduling of a QP and synchronously updating the read pointer position of the sending work queue. Step 4) Reordering operation: After processing a work queue element, the corresponding position and all the previous unflagged parts are cyclically shifted, that is, these bits are shifted to the right to the end of the QP group status table; Step 5) Repeated location: After rearrangement, repeat the fast location in the new QP group status table to continue to find and process the next pending work queue element, and continue until all QPs with pending work queue elements in the group have been processed. Step 6) Return to the flag table: After processing a group, return to the QP group status flag table and perform a rearrangement operation, that is, cyclically shift the unflagged part of the flag table at the current position to the end; Step 7) Group processing loop: Repeat steps 1)-6) until all groups in the QP group status flag table have been processed; Step 8) Global scheduling complete: retrieve the new QP state table and continue to execute the above steps in a loop.
8. The RDMA multi-QP scheduling method according to claim 7, characterized in that, The rapid positioning in steps 1) and 2) is achieved using a priority encoder.
9. A network card with RDMA functionality, characterized in that, Implemented using FPGA, it includes: a table management module and a scheduling module; among which, The table management module is used to register QPs with pending work queue elements using a record table; it manages QPs in groups based on the record table and maintains a corresponding status flag for each group, thus constructing a hierarchical table structure from QP to its group and then to the group status flag. The scheduling module is used to perform hierarchical search based on a progressive table structure, and then use a location-rearrangement-location strategy to traverse and complete the scheduling tasks of all groups.