RDMA queue space lazy distribution method and system
By dynamically allocating memory pages and the LinkWQE mechanism, the problems of large memory usage and allocation failure in the RDMA system are solved, and efficient memory usage and flexible system expansion are achieved. It is suitable for high-concurrency and resource-constrained environments.
Patent Information
- Application Number
- CN202510765802.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-10
- Publication Date
- 2025-09-19
AI Technical Summary
In the existing technology, the RDMA system allocates all work queue memory pages at once during the QP initialization phase, resulting in large memory usage, low utilization efficiency, lack of dynamic scalability, and prone to memory allocation failures in high-concurrency scenarios, affecting system stability and resource scheduling.
A lazy allocation method for RDMA queue space with dynamic memory allocation is adopted. By allocating memory pages on demand and using LinkWQE to implement memory page recycling and cross-page jumps, memory usage is dynamically expanded and an extensible page table structure is maintained.
Significantly improves memory space utilization, reduces initial memory usage, enhances system elasticity, reduces physical memory fragmentation, and avoids frequent allocation failures. It is suitable for cloud virtualization platforms and edge devices where memory is scarce.
Smart Images
Figure CN120670161A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of communication technology, and in particular relates to a lazy allocation method and system for RDMA queue space. Background Art
[0002] In the prior art, the QP (Queue Pair) initialization phase allocates all memory pages required by all work queues (WQ) at once. The specific process is as follows:
[0003] When an RDMA application creates a QP using the ibv_create_qp interface, the user must enter the maximum number of concurrent WRs (WorkRequests) (max_send_wr and max_recv_wr) and the maximum number of SGEs each WR can contain (max_send_sge and max_recv_sge). The driver calculates the total number of WQEs (Work Queue Entries) and the total memory required based on these parameters. It then aligns the required memory upwards according to the system page size and requests all memory pages from the operating system at once during the QP creation phase to pre-allocate them for storing all WQEs in the WQ.
[0004] The above-mentioned method of allocating memory pages at one time has the following technical defects:
[0005] 1. Large initial memory usage and low efficiency:
[0006] Preallocating all memory pages at maximum capacity at once, regardless of whether they are actually used during runtime, will occupy physical memory for a long time, resulting in a large amount of memory waste. This static allocation method leads to extremely low memory space utilization, especially in scenarios with a large number of QPs but each QP using only a small number of WRs.
[0007] 2. Lack of dynamic scaling capabilities and inflexibility:
[0008] Since memory is allocated once when the QP is created, it cannot be dynamically expanded or shrunk based on the actual load. As a result, the system cannot adapt efficiently when the load changes, affecting overall performance and resource scheduling capabilities.
[0009] 3. Allocation is prone to failure in high-concurrency scenarios:
[0010] Allocating large blocks of contiguous memory pages at one time is sensitive to physical memory fragmentation and is prone to allocation failure after the system has been running for a period of time. This is especially prominent when memory is tight or page alignment requirements are high, reducing system stability.
[0011] In summary, the traditional one-time allocation method lacks flexibility and has low resource utilization, which limits the application efficiency and scalability of RDMA systems in actual high-concurrency and resource-constrained scenarios. Summary of the Invention
[0012] The purpose of the present invention is to provide a lazy allocation method and system for RDMA queue space, which adopts a dynamic allocation method to effectively improve the utilization rate of memory space and is suitable for use in scenarios where memory space is relatively scarce, such as cloud virtualization platforms and edge devices.
[0013] The present invention provides a lazy allocation method for RDMA queue space, comprising the following steps:
[0014] Step 1: The RDMA application creates a queue pair through the ibv_create_qp interface and passes in parameters during creation, including the maximum number of concurrent work requests (max_send_wr and max_recv_wr), and the maximum number of scatter-gather elements (max_send_sge and max_recv_sge) in each work request.
[0015] Step 2: The driver calculates the total number of memory pages required for the work queue based on the parameters and the current system memory page size, and records it as page_capacity;
[0016] Step 3: The driver initializes a memory page information table based on page_capacity to maintain the memory page distribution information used by the work queue;
[0017] Step 4: During the queue initialization phase, the driver only requests and allocates one page of memory as the initial memory page of the work queue to meet the most basic usage requirements.
[0018] Step 5: When an RDMA application submits a work request through the ibv_post_send or ibv_post_recv interface, the driver first calculates the number of work queue elements required to complete the work request.
[0019] Step 6: The driver attempts to allocate enough space from the currently used memory page to store the calculated work queue elements. If the current memory page does not have enough remaining space, it proceeds to step 7.
[0020] Step 7: The driver queries the memory page information table and attempts to obtain the next page information of the current memory page. Based on the query results, it handles the following two situations:
[0021] Case 1: No next page
[0022] Subcase 1: If the current memory page number has reached page_capacity, the driver will use the first page in the information table as the next page to achieve recycling;
[0023] Subcase 2: If the current memory page number has not reached page_capacity, the driver will request the operating system to allocate a new memory page and add the page to the end of the memory page information table;
[0024] Case 2: Next page exists
[0025] The driver directly extracts the page information from the memory page information table and uses it as the next page;
[0026] Step 8: The driver adds a LinkWQE at the end of the current page, which contains the physical address of the next page.
[0027] Step 9: The driver adds the calculated work queue element at the beginning of the new memory page for execution by the RDMA hardware.
[0028] Step 10: When the network card hardware processes LinkWQE, it jumps to the next page according to the physical address therein and continues to execute subsequent work queue elements.
[0029] The present invention also provides an RDMA queue space lazy allocation system, comprising a lazy allocation module, and the lazy allocation module executes the RDMA queue space lazy allocation method.
[0030] The present invention also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the RDMA queue space lazy allocation method is implemented.
[0031] The present invention also provides an electronic device, comprising:
[0032] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the RDMA queue space lazy allocation method by executing the computer instructions.
[0033] The above solution, through the RDMA queue space lazy allocation method and system, has the following technical effects:
[0034] 1) By introducing an on-demand allocation and memory page round-robin mechanism, only one page of memory is allocated when a QP is created. The number of required pages is dynamically expanded based on actual WR usage until the preset upper limit is reached. Subsequently, a page table + LinkWQE method is used to jump between pages and round-robin reuse, significantly optimizing the QP work queue memory management method.
[0035] 2) Based on the maximum WR parameter entered by the user and the system page size, the capacity limit is dynamically calculated, a scalable page table structure is maintained, and LinkWQE is introduced to implement cross-page jumps. This ensures functional integrity while achieving technical benefits such as reducing initial memory usage, improving system resilience, reducing physical memory fragmentation, and avoiding frequent allocation failures. This is particularly suitable for large-scale concurrent or resource-constrained RDMA environments.
[0036] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a flowchart of a lazy allocation method for RDMA queue space in one embodiment of the present invention;
[0038] Figure 2 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION
[0039] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.
[0040] Explanation of terms:
[0041] RDMA (Remote Direct Memory Access): Remote direct memory access, a high-performance communication technology that bypasses the CPU to implement memory reading and writing.
[0042] ibv_create_qp: Interface function used to create a queue pair (QP).
[0043] QP (Queue Pair): Queue pair is the basic communication endpoint in RDMA communication, consisting of a send queue (SQ) and a receive queue (RQ).
[0044] WR (Work Request): A work request describes a specific RDMA operation and is submitted to the queue by the user.
[0045] max_send_wr: The maximum number of concurrent WRs allowed in the send queue.
[0046] max_recv_wr: The maximum number of concurrent WRs allowed in the receive queue.
[0047] SGE (Scatter-Gather Element): Scatter-gather element, describes the data buffer address and length.
[0048] max_send_sge: The maximum number of SGEs included in each sent WR.
[0049] max_recv_sge: The maximum number of SGEs contained in each received WR.
[0050] page_capacity: indicates the number of memory pages required by the WQ, which is used to calculate the memory requirements when allocating WQEs.
[0051] WQ (Work Queue): Work queue, used to store WRs to be executed, including the send queue (SQ) and the receive queue (RQ).
[0052] WQE (Work Queue Element): Work queue element, the actual execution unit into which each WR in the queue is converted.
[0053] ibv_post_send: Interface function used to submit and send WR.
[0054] ibv_post_recv: Interface function used to submit and receive WR.
[0055] LinkWQE (Linked Work Queue Element): LinkWQE is a special type of work queue element (WQE) used to establish a jump relationship between multiple memory pages of a work queue. LinkWQE itself does not perform specific RDMA operations, but contains the physical address of the next memory page. When the network card hardware processes a LinkWQE, it will jump to the specified page based on the address it carries to continue processing subsequent WQEs, thereby realizing cross-page work queue execution. The introduction of LinkWQE ensures the continuity and scalability of the work queue.
[0056] Ginseng Figure 1 As shown, this embodiment provides a lazy allocation method for RDMA queue space, including the following steps:
[0057] Step 1: The RDMA application creates a queue pair through the ibv_create_qp interface and passes in parameters during creation, including the maximum number of concurrent work requests (max_send_wr and max_recv_wr), and the maximum number of scatter-gather elements (max_send_sge and max_recv_sge) in each work request.
[0058] Step 2: The driver calculates the total number of memory pages required for the work queue based on the parameters and the current system memory page size, and records it as page_capacity;
[0059] Step 3: The driver initializes a memory page information table based on page_capacity to maintain the memory page distribution information used by the work queue;
[0060] Step 4: During the queue initialization phase, the driver only requests and allocates one page of memory as the initial memory page of the work queue to meet the most basic usage requirements.
[0061] Step 5: When an RDMA application submits a work request through the ibv_post_send or ibv_post_recv interface, the driver first calculates the number of work queue elements required to complete the work request.
[0062] Step 6: The driver attempts to allocate enough space from the currently used memory page to store the calculated work queue elements. If the current memory page does not have enough remaining space, it proceeds to step 7.
[0063] Step 7: The driver queries the memory page information table and attempts to obtain the next page information of the current memory page. Based on the query results, it handles the following two situations:
[0064] Case 1: No next page
[0065] Subcase 1: If the current memory page number has reached page_capacity, the driver will use the first page in the information table as the next page to achieve recycling;
[0066] Subcase 2: If the current memory page number has not reached page_capacity, the driver will request the operating system to allocate a new memory page and add the page to the end of the memory page information table;
[0067] Case 2: Next page exists
[0068] The driver directly extracts the page information from the memory page information table and uses it as the next page;
[0069] Step 8: The driver adds a LinkWQE at the end of the current page, which contains the physical address of the next page.
[0070] Step 9: The driver adds the calculated work queue element at the beginning of the new memory page for execution by the RDMA hardware.
[0071] Step 10: When the network card hardware processes LinkWQE, it jumps to the next page according to the physical address therein and continues to execute subsequent work queue elements.
[0072] With this lazy RDMA queue space allocation method, when an application creates a queue using the ibv_create_qp interface, it must pass in the maximum number of concurrent writes (max_send_wr and max_recv_wr) and the maximum number of session enqueues (SGEs) per write (max_send_sge and max_recv_sge). Based on this information and the current system memory page size, the driver calculates the number of memory pages required for the WQ (expressed as page_capacity) and maintains a table of memory page information. When initializing the QP, the driver allocates only one page of memory to meet the minimum requirements. When a user issues a write using the ibv_post_send / ibv_post_recv interfaces, the driver first calculates the number of WQEs required to complete the write and then allocates memory of that size from the WQ's current memory page. If the current page does not have enough free space, the driver queries the memory page information table for the next page. The query results are handled in two ways. The driver then adds a LinkWQE to the current page, which contains the physical address of the next page. The driver also adds the WQE required to complete the WR function at the beginning of the next page. When the NIC hardware processes the LinkWQE, it jumps to the physical address of the next page indicated by it to continue processing.
[0073] The RDMA queue space lazy allocation method has the following technical effects:
[0074] 1) By introducing an on-demand allocation and memory page round-robin mechanism, only one page of memory is allocated when a QP is created. The number of required pages is dynamically expanded based on actual WR usage until the preset upper limit is reached. Subsequently, a page table + LinkWQE method is used to jump between pages and round-robin reuse, significantly optimizing the QP work queue memory management method.
[0075] 2) Based on the maximum WR parameter entered by the user and the system page size, the capacity limit is dynamically calculated, a scalable page table structure is maintained, and LinkWQE is introduced to implement cross-page jumps. This ensures functional integrity while achieving technical benefits such as reducing initial memory usage, improving system resilience, reducing physical memory fragmentation, and avoiding frequent allocation failures. This is particularly suitable for large-scale concurrent or resource-constrained RDMA environments.
[0076] The present invention also provides an RDMA queue space lazy allocation system, comprising a lazy allocation module, and the lazy allocation module executes the RDMA queue space lazy allocation method.
[0077] The present invention also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the RDMA queue space lazy allocation method is implemented.
[0078] Ginseng Figure 2 As shown, the present invention also provides an electronic device, including:
[0079] The memory 201 and the processor 202 are communicatively connected to each other. The memory 201 stores computer instructions. The processor 202 executes the computer instructions to perform the RDMA queue space lazy allocation method.
[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A lazy allocation method for RDMA queue space, characterized in that: The steps include: Step 1: The RDMA application creates a queue pair through the ibv_create_qp interface and passes in parameters during creation, including the maximum number of concurrent work requests (max_send_wr and max_recv_wr), and the maximum number of scatter-gather elements (max_send_sge and max_recv_sge) in each work request. Step 2: The driver calculates the total number of memory pages required for the work queue based on the parameters and the current system memory page size, and records it as page_capacity; Step 3: The driver initializes a memory page information table based on page_capacity to maintain the memory page distribution information used by the work queue; Step 4: During the queue initialization phase, the driver only requests and allocates one page of memory as the initial memory page of the work queue to meet the most basic usage requirements. Step 5: When an RDMA application submits a work request through the ibv_post_send or ibv_post_recv interface, the driver first calculates the number of work queue elements required to complete the work request. Step 6: The driver attempts to allocate enough space from the currently used memory page to store the calculated work queue elements. If the current memory page does not have enough remaining space, it proceeds to step 7. Step 7: The driver queries the memory page information table and attempts to obtain the next page information of the current memory page. Based on the query results, it handles the following two situations: Case 1: No next page Subcase 1: If the current memory page number has reached page_capacity, the driver will use the first page in the information table as the next page to achieve recycling; Subcase 2: If the current memory page number has not reached page_capacity, the driver will request the operating system to allocate a new memory page and add the page to the end of the memory page information table; Case 2: Next page exists The driver directly extracts the page information from the memory page information table and uses it as the next page; Step 8: The driver adds a LinkWQE at the end of the current page, which contains the physical address of the next page. Step 9: The driver adds the calculated work queue element at the beginning of the new memory page for execution by the RDMA hardware. Step 10: When the network card hardware processes LinkWQE, it jumps to the next page according to the physical address therein and continues to execute subsequent work queue elements.
2. A lazy allocation system for RDMA queue space, characterized in that: The system comprises a lazy allocation module, wherein the lazy allocation module executes the RDMA queue space lazy allocation method according to claim 1.
3. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the RDMA queue space lazy allocation method according to claim 1 is implemented.
4. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the RDMA queue space lazy allocation method according to claim 1 by executing the computer instructions.