A method, apparatus, device, medium, and product for optimizing RDMA shared receive queues.
Patent Information
- Application Number
- CN202511385761.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-25
- Publication Date
- 2026-09-01
- Estimated Expiration
- 2045-09-25
AI Technical Summary
[0003]然而,相关技术中的SRQ管理方案在应对多队列对(Queue Pair,QP)并发场景时,面临资源分配效率与灵活性的根本性矛盾
[0019] The RDMA shared receive queue optimization method, apparatus, device, medium, and product disclosed above enable dynamic resource reuse through container-level resource monitoring and an out-of-order reclamation mechanism independent of other container states. Once all SRQEs in any container are released, they can be immediately redistributed, completely avoiding the waste of tail-end idle resources caused by head-end resource occupation in traditional linear queues.
Smart Images

Figure CN121301248B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of Remote Direct Memory Access (RDMA) technology, and more particularly to an RDMA shared receive queue optimization method, apparatus, device, medium, and product. Background Technology
[0002] With the widespread application of RDMA technology in high-speed data center networks, Shared Receive Queue (SRQ), as an efficient network resource pooling solution, has become a core means to improve network throughput and resource utilization.
[0003] However, SRQ management schemes in related technologies face a fundamental contradiction between resource allocation efficiency and flexibility when dealing with concurrent scenarios involving multiple queue pairs (QPs). Specifically, while the linear pre-allocation mode can guarantee low latency, it suffers from severe head blocking, preventing idle resources from being reused in a timely manner; and while the linked list management mode solves the resource reuse problem, it introduces additional memory read latency, thus compromising the low-latency characteristics of RDMA.
[0004] Therefore, how to achieve efficient and low-latency allocation and reclamation of SRQ resources while being compatible with multiple concurrent QPs has become a key technical challenge for improving the performance of RDMA networks. Summary of the Invention
[0005] In view of this, this application provides an RDMA shared receive queue optimization method, apparatus, device, medium, and product to at least solve the problem of how to achieve efficient and low-latency allocation and reclamation of SRQ resources while being compatible with multiple concurrent QPs.
[0006] This application provides an optimization method for RDMA shared receive queue, the method comprising:
[0007] The receiving software divides the shared receiving queue into multiple containers and links these containers together using a logical linked list; each container contains multiple shared receiving queue entries.
[0008] When the receiving network card detects the arrival of RDMA data for a queue pair, if the queue pair is not bound to a container, it allocates a container for the queue pair from the head of the logical linked list; the receiving network card prioritizes allocating shared receive queue entries to the arriving RDMA data from the containers bound to the queue pair in sequence.
[0009] The receiving software monitors the release status of shared receive queue entries within multiple containers at the container level to determine whether a container is in a completely idle state.
[0010] When the receiving software detects that the target container is in a completely idle state, it relinks the target container into the logical linked list for allocation, independent of the states of other containers among the multiple containers.
[0011] This disclosure also provides an RDMA shared receive queue optimization device, which includes: a container partitioning module, a resource allocation module, a status monitoring module, and a container reclamation module, wherein:
[0012] The container partitioning module is used to divide the shared receive queue into multiple containers and link the multiple containers together through a logical linked list; each container contains multiple shared receive queue entries.
[0013] The resource allocation module is used to allocate a container for the queue pair from the head of the logical linked list when the arrival of RDMA data for the queue pair is detected, if the queue pair is not bound to a container; and to allocate shared receive queue entries to the arriving RDMA data in order from the containers bound to the queue pair.
[0014] The status monitoring module is used to monitor the release status of shared receive queue entries in multiple containers at the container level, and to determine whether the container is in a completely idle state.
[0015] The container recycling module is used to relink the target container back into the logical linked list for allocation, independent of the state of other containers, when the target container is detected to be in a completely idle state.
[0016] This disclosure also provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the above-described RDMA shared receive queue optimization method.
[0017] This disclosure also provides a computer-readable storage medium storing computer instructions for causing a computer to implement the aforementioned RDMA shared receive queue optimization method.
[0018] This disclosure also provides a computer program product, including computer instructions for causing a computer to execute the aforementioned RDMA shared receive queue optimization method.
[0019] The RDMA shared receive queue optimization method, apparatus, device, medium, and product disclosed above enable dynamic resource reuse through container-level resource monitoring and an out-of-order reclamation mechanism independent of other container states. Once all SRQEs in any container are released, they can be immediately redistributed, completely avoiding the waste of tail-end idle resources caused by head-end resource occupation in traditional linear queues.
[0020] The receiver-side network interface card (NIC) allocates SRQEs at the container level, allowing multiple packets within the same QP to be consecutively allocated within a container without frequent lookups of the linked list pointers, significantly reducing processing latency between packets within the same QP. Simultaneously, the contiguous nature of SRQEs within containers provides a foundation for hardware prefetch optimization, further reducing memory access overhead.
[0021] In addition, the logical linked list structure can maintain flexible relationships between containers and support concurrent access by multiple QPs; and the sequential allocation mode within the container can guarantee the locality and determinism of operations, thereby improving resource utilization while ensuring the low latency and high throughput characteristics of RDMA operations. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a flowchart illustrating a SEND operation provided in this publication;
[0024] Figure 2 This is a flowchart illustrating an RDMA shared receive queue optimization method provided in an embodiment of this disclosure;
[0025] Figure 3 This is a schematic diagram of multiple container links in an RDMA shared receive queue optimization method provided in this embodiment of the disclosure;
[0026] Figure 4 This is a schematic diagram of the SRQ container linked list structure of an RDMA shared receive queue optimization method provided in this embodiment of the disclosure;
[0027] Figure 5 This is a schematic diagram of the initial sequence chain structure of the SRQ container in an RDMA shared receive queue optimization method provided in this embodiment;
[0028] Figure 6 This is a schematic diagram of the packet receiving sequence of an RDMA shared receive queue optimization method provided in an embodiment of this disclosure;
[0029] Figure 7 This is a schematic diagram of the SRQ container out-of-order chain structure of an RDMA shared receive queue optimization method provided in this embodiment;
[0030] Figure 8This is a schematic diagram of the structure of an RDMA shared receive queue optimization device provided in an embodiment of this disclosure;
[0031] Figure 9 This is a schematic diagram of another RDMA shared receive queue optimization device provided in an embodiment of this disclosure. Detailed Implementation
[0032] RDMA technology, as a core support for high-performance network transmission, essentially solves the problem of high server-side data processing latency in traditional network transmission by bypassing the operating system kernel and enabling direct cross-device memory interaction, thereby meeting the requirements for zero-copy, low-latency, and high-performance data interaction. The SEND operation, as a typical input / output (IO) channel operation in RDMA technology, is a key carrier for realizing cross-device data transmission. Its core process relies on the collaboration of components such as QP, Send Queue (SQ), Receive Queue (RQ), and Completion Queue (CQ). Please refer to [reference needed]. Figure 1 , Figure 1 This is a flowchart illustrating a SEND operation provided in an embodiment of this disclosure, including the following specific steps:
[0033] ① The receiving user sends a Receive Queue Entry (RQE) to the RQ in the QP;
[0034] ② The sending user sends a request queue entry (SQE) to the SQ in the QP;
[0035] ③ Send the network card to read SQE and obtain the SEND Message;
[0036] ④ Send the data to the network card to retrieve memory data;
[0037] ⑤ The network card sends packets to the other end. When the data length is greater than the Path Maximum Transmission Unit (PMTU), it will be split into multiple packets for transmission.
[0038] ⑥ The receiving network card receives packets and reads the RQE to obtain the receive buffer information; the same RQE is used for the same message;
[0039] ⑦ The receiving network card writes the data into memory;
[0040] ⑧ The receiving network card replies with an acknowledgment message to the other end;
[0041] ⑨ After the receiving network card has received and processed the entire message, it reports the reception completion information (CompletionQueue Entry, CQE) to the CQ.
[0042] ⑩ The receiving user obtains the reception completion information from the CQ and reclaims the RQE space;
[0043] 11. The receiving side user will redeliver the recovered RQE to the hardware for use;
[0044] 12. The sending side network card reports the transmission completion information (CQE) to CQ;
[0045] 13. The sending side user obtains the sending completion information from CQ and reclaims the SQE space.
[0046] In RDMA data reception scenarios, conventional designs configure each QP with an independent RQ. Data must be received sequentially, RQEs must be consumed sequentially, and CQEs must be reported sequentially. However, this independent RQ mode has the limitation of low resource utilization. To address this, SRQ technology was proposed. Its core is to allow multiple QPs to share the same SRQ. The concurrent SEND service of multiple QPs occupies the shared receive queue entry (SRQE) according to the packet reception order: The receiving user first sends an SRQE to the SRQ. When the receiving network card receives a packet pointing to a certain QP (such as QP3), the QP retrieves the first SRQE from the SRQ, completes the data writing, and reports the CQE to the associated CQ (such as CQ3). The user then reclaims the SRQE space. When a packet pointing to another QP (such as QP2) is received subsequently, the QP retrieves the next SRQE and repeats the process. The reclaimed SRQE is finally redelivered to the hardware for reuse.
[0047] However, the conventional usage mode of SRQ still suffers from head-blocking issues. Specifically, if the receiver receives a SendFirst packet from a QP (e.g., QP3) first, the SRQE (e.g., SRQE0) it occupies must wait for the SendLast packet of that message to be processed before it can be reported to the CQE and released. If a SendOnly packet from another QP (e.g., QP2) is received subsequently, the SRQE (e.g., SRQE1) it occupies can be directly reported to the CQE and released early. However, when the software uses a linear (pointer incrementing) method to deliver SRQEs, because the earlier SRQE0 is not released, even if the later SRQE1 has been reclaimed early, it cannot be redelivered to the hardware, resulting in idle resources.
[0048] To address the head-blocking problem, related technologies have proposed using SRQ linked lists for optimization. This involves adding a header before the SRQE data segment and maintaining the linked list relationship between SRQEs using the `next_idx` property in the header. Early-released SRQEs can be added to the tail of the list, and the hardware retrieves an SRQE from the head for use each time. However, this optimization still has performance bottlenecks: the link relationship between SRQEs is random due to differences in release timing. The hardware must first read the header of the current SRQE from the host to obtain the index of the next SRQE, resulting in additional host read latency between packets using the same SRQ. This severely restricts RDMA transmission performance and makes it difficult to meet the low-latency requirements of high-performance network scenarios.
[0049] In summary, the relevant technologies have the following problems: First, the conventional linear delivery method of SRQE is prone to head blocking, which makes it impossible to reuse the prematurely released SRQE in a timely manner, thus wasting resources; Second, although the improved scheme based on SRQ linked lists solves the head blocking problem, the random SRQE linking relationship introduces host read latency, which seriously affects transmission performance and cannot meet the needs of high-performance RDMA data interaction.
[0050] To address the aforementioned issues, various embodiments of this disclosure provide an RDMA shared receive queue optimization method. The method includes: receiving-side software dividing the shared receive queue into multiple containers and linking these containers together via a logical linked list; each container contains multiple shared receive queue entries; when the receiving-side network interface card (NIC) detects the arrival of RDMA data for a queue pair, if the queue pair is not bound to a container, it allocates a container for the queue pair from the head of the logical linked list; the receiving-side NIC prioritizes allocating shared receive queue entries to the arriving RDMA data from the containers bound to the queue pair in sequence; the receiving-side software monitors the release status of shared receive queue entries within multiple containers at the container level, determining whether the containers are in a completely idle state; when the receiving-side software detects that a target container is in a completely idle state, it re-links the target container into the logical linked list for allocation, independent of the status of other containers within the multiple containers.
[0051] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0052] Please refer to Figure 2 , Figure 2This is a flowchart illustrating an RDMA shared receive queue optimization method provided in this embodiment. The method may include the following steps:
[0053] In step S201, the receiving software divides the shared receiving queue into multiple containers and links the multiple containers together through a logical linked list.
[0054] In this embodiment, the receiving side has two core units in the RDMA data reception process: the receiving side software and the receiving side network card. The receiving side software is the core control unit responsible for SRQ resource logic management, status monitoring and dynamic recycling and allocation; correspondingly, the receiving side network card is the core hardware unit responsible for hardware reception of RDMA messages, data processing and resource allocation execution.
[0055] Furthermore, the receiving software performs structured processing on the shared receiving queue (SRQ), constructing the SRQ into an organization with containers as the basic functional units, and connecting each container in series through a logical linked list.
[0056] Each container is configured with multiple shared receive queue entries (SRQE) for carrying data receiving tasks.
[0057] Here, SRQ is a receive queue shared by multiple QPs. It achieves buffer reuse for concurrent receive tasks across multiple QPs through unified management of receive buffer resources. SRQE is the basic functional unit within SRQ, and each SRQE is associated with a host memory buffer used for receiving data. A container is the basic functional unit within SRQ used to hold SRQEs, allowing for the aggregation and management of multiple SRQEs.
[0058] In one possible implementation, the step of the receiving software dividing each SRQE in the SRQ into different containers may include: the receiving software determining a preset division criterion based on business requirements, wherein the preset division criterion includes, but is not limited to: the number of SRQEs in each container, the address range of SRQEs in memory and the associated logic; traversing all SRQEs in the SRQ, and according to the preset division criterion, aggregating several SRQE logics that meet the conditions into one container.
[0059] The containers are linked together using a logical linked list.
[0060] Here, logical linked lists can achieve scheduling links between containers through preset link relationships.
[0061] As an example, please refer to Figure 3 , Figure 3 This is a schematic diagram of multiple container links in an RDMA shared receive queue optimization method provided in this embodiment.
[0062] like Figure 3 As shown, assume that SRQ is divided into 3 containers, namely Container0, Container1 and Container2. Each container is configured with 3 SRQE. Container0 points to Container1 through a linked list, and Container1 points to Container2 through a linked list.
[0063] Furthermore, in one possible implementation, the number of SRQEs within the container can be flexibly configured according to actual needs.
[0064] In step S202, when the receiving network card detects the arrival of RDMA data for a queue pair, if the queue pair is not bound to a container, it allocates a container for the queue pair from the head of the logical linked list; the receiving network card prioritizes allocating shared receive queue entries to the arriving RDMA data from the containers bound to the queue pair in sequence.
[0065] In this embodiment, when RDMA data arrives at the receiving network card, the receiving network card needs to allocate a corresponding SRQE for the RDMA data. Through the predefined memory buffer parameters of the SRQE, the RDMA data is accurately written into the specified memory area of the receiving host.
[0066] Here, during the allocation process of SRQE, a target container is retrieved from the logical linked list of SRQs, and within the target container, at least one SRQE is selected and allocated to RDMA data according to preset rules. Specifically, RDMA data can refer to SEND data.
[0067] Understandably, during SRQE allocation, the receiving network card uses complete containers rather than individual SRQEs as the smallest scheduling unit to avoid inefficient scheduling caused by frequent hardware addressing. At the same time, each QP maintains breakpoint information (such as the index of allocated SRQEs within the container) for each allocated container. This breakpoint information is stored in the QP context to ensure that when subsequent RDMA data arrives for the same QP, sequential allocation can continue directly from the unallocated SRQE position within the container without needing to relocate the starting allocation point within the container.
[0068] As an example, continue with Figure 3Taking the three containers shown as an example, assuming QP1 and QP2 arrive at the receiving network card, when the first RDMA data of QP1 arrives, the receiving network card selects the first available SRQE 0 in Container 0 in sequence and writes the first RDMA data into the memory buffer pre-associated with SRQE 0; when the second RDMA data of QP1 arrives, the receiving network card selects the second available SRQE 1 in Container 0 in sequence and writes the second RDMA data into the memory buffer pre-associated with SRQE 1; when the third RDMA data of QP1 arrives, the receiving network card selects the third available SRQE 2 in Container 0 in sequence and writes the third RDMA data into the memory buffer pre-associated with SRQE 2. At this time, all SRQEs in Container 0 are allocated to QP1.
[0069] When the first RDMA data of QP2 arrives, since QP2 is not bound to a container, the receiving network card does not need to wait for Container0 to be consumed. It directly obtains the next container Container1 from the head of the logical list and binds it to QP2. In Container1, the first available SRQE3 is selected in sequence, and the first RDMA data of QP2 is written to the memory buffer pre-associated with SRQE3.
[0070] Understandably, RDMA data belonging to the same QP is preferentially allocated continuously within a single container via SRQE to avoid frequent cross-container addressing by the network interface card. Only when all SRQEs of the current container are allocated will the system switch to the next container and update the breakpoint information of the QP context. Furthermore, multiple containers can be allocated to different QPs simultaneously (e.g., QP1 occupies Container 0, QP2 occupies Container 1), and each QP independently uses its bound container without waiting for other QPs to release their bound containers, thus adapting to scenarios where data from multiple QPs arrives at different times.
[0071] In step S203, the receiving software monitors the release status of shared receiving queue entries in multiple containers at the container level to determine whether the container is in a completely idle state.
[0072] In this embodiment, after the receiving network card writes the RDMA data into the memory buffer pre-associated with the SRQE, it reports the completion queue entry (CQE) to the completion queue (CQ) bound to the QP to which the SRQE belongs. The CQE is used to indicate that the SRQE has completed data processing. The receiving software reads the CQE from the CQ and obtains the message that the SRQE has completed data processing before releasing the SRQE.
[0073] Here, after reading the CQE, the receiving software locates the SRQE based on the SRQE identifier in the CQE, updates the SRQE's status to idle, and unbinds the SRQE from the original RDMA data.
[0074] Furthermore, the receiving software can continuously poll the CQ bound to each QP and obtain the CQE reported by the receiving network card in real time to obtain the completion status of each SRQE. Based on the completion status of each SRQE, the software can monitor the release status of each SRQE and determine whether the container is in a completely idle state.
[0075] As an example, continue with Figure 3 Taking the three containers shown as an example, if SRQE 0 in Container 0 has completed data transmission, while SRQE 1 and SRQE 2 have not completed data transmission, the receiving software updates the status of SRQE 0 to idle, and at this time it is determined that Container 0 is not in a completely idle state; if SRQE 0 to SRQE 2 in Container 0 have all completed data transmission, that is, SRQE 0 to SRQE 2 have all been released, then Container 0 is determined to be in a completely idle state.
[0076] Step S204: When the receiving software detects that the target container is in a completely idle state, it relinks the target container into the logical linked list for allocation, independent of the states of other containers among the multiple containers.
[0077] In this embodiment, once the receiving software detects that a container satisfies the condition that all internal SRQEs have been released (i.e., the container is in a completely idle state), and is not constrained by the state of other containers in the SRQ, it directly reconnects the container to the logical linked list of the SRQ, so that the container continues to be allocated by the receiving network card.
[0078] Here, even if other containers are still occupied, as long as the target container itself is in a completely idle state, the re-linking process can be executed independently. The target container no longer needs to wait for release in sequence, thereby realizing out-of-order reuse of containers and avoiding resource idleness caused by other containers not being released.
[0079] As an example, continue with Figure 3Taking the three containers shown as an example, assuming that SRQE 0 and SRQE 1 in Container 0 have been allocated to RDMA data for QP1, and SRQE 2 is idle, Container 0 is in a partially idle state, but not completely idle; SRQE 3 to SRQE 5 in Container 1 have all been allocated to RDMA data for QP2, so Container 1 is in a fully occupied state; SRQE 6 to SRQE 7 in Container 2 have all completed data processing. After the receiving software obtains the CQE by polling the CQ, it confirms that all SRQEs in Container 2 have been released, so Container 2 is in a completely idle state.
[0080] At this point, although Container0 is still partially occupied and Container1 is fully occupied and not released, Container2 itself has met the condition of being completely idle. Therefore, the receiving software does not need to wait for Container0 or Container1 to be released and can directly relink Container2 into the tail of the SRQ logical list.
[0081] The RDMA shared receive queue optimization method, apparatus, device, medium, and product disclosed in the above embodiments enable dynamic resource reuse through container-level resource monitoring and an out-of-order reclamation mechanism independent of other container states. Once all SRQEs within any container are released, they can be immediately reallocated, completely avoiding the waste of tail-end idle resources caused by head-end resource occupation in traditional linear queues. The receiving-side network card allocates resources at the container level, allowing multiple packets of the same QP to be continuously allocated SRQEs within a container without frequent lookups of the linked list pointer, significantly reducing processing latency between packets of the same QP. Simultaneously, the continuous nature of SRQEs within a container provides a basis for hardware prefetch optimization, further reducing memory access overhead. The logical linked list structure maintains flexible associations between containers, supporting concurrent access by multiple QPs; and the sequential allocation mode within containers ensures the locality and determinism of operations, thereby improving resource utilization while ensuring low latency and high throughput characteristics of RDMA operations.
[0082] In one possible implementation of step S201 above, multiple shared receive queue entries within the container have contiguous physical addresses in memory.
[0083] In this embodiment, the receiving software divides the host memory into contiguous physical address blocks according to the preset number of SRQEs in each container, ensuring that the address blocks can be directly accessed by the network card via RDMA. Furthermore, the physical addresses corresponding to each SRQE in the physical address block are consecutive, meaning the end address of the previous SRQE is equal to the start address of the next SRQE.
[0084] As an example, suppose Figure 3 In Container 0, SRQE 0 is located at address A, and each SRQE is sized S. Therefore, SRQE 1 in Container 0 is located at address A+S, and SRQE 2 is located at address A+2S. The memory size occupied by each SRQE is preset and fixed.
[0085] In one possible implementation, when the physical addresses of SRQEs within a container are contiguous, the receiving network card can proactively read the entire contiguous physical space corresponding to the container through cache prefetching technology, preloading the entire container and each SRQE within the container into the cache within the network card. This eliminates the need to frequently initiate host memory read operations for individual SRQEs, thereby reducing the number of host accesses and lowering hardware latency.
[0086] Through the RDMA shared receive queue optimization method, apparatus, device, medium, and product of the above embodiments of this disclosure, the physical addresses of SRQEs within the container are contiguous. This allows hardware to read an entire physical space using cache prefetching technology, prefetching the entire container onto the chip without frequently initiating host read operations for individual SRQEs, thus reducing host interaction overhead. When the network card uses SRQEs within the same container, it can quickly locate the next SRQE without relying on linked list index lookups, reducing scheduling latency between QP packets and improving data reception efficiency.
[0087] In one possible implementation of step S201 above, the linking relationship of the logical linked list is maintained by an index pointer stored in the header information of the first shared receive queue entry in each container.
[0088] In this embodiment, when the receiving software initializes a container, it writes an index pointer (referred to as next_idx) pointing to the next container into the header of the first SRQE within that container. Next_idx can be used to indicate the physical address of the first SRQE within the next container.
[0089] In addition, the header information of the first SRQE within the container can also include the address of the first SRQE itself (referred to as cur_idx). Here, cur_idx can be used to locate the current container during container reclamation.
[0090] As an example, please refer to Figure 4 , Figure 4 This is a schematic diagram of the SRQ container linked list structure of an RDMA shared receive queue optimization method provided in this embodiment.
[0091] like Figure 4As shown, this SRQ can include at least four containers: container 0, container 1, container 2, container 3, and at least one other container that may be concatenated subsequently. Each container can include four SRQEs: container 0 includes SRQE 0 to SRQE 3, container 1 includes SRQE 4 to SRQE 7, container 2 includes SRQE 8 to SRQE 11, and container 3 includes SRQE 12 to SRQE 15. The header of the first SRQE in each container includes cur_idx, next_idx, and a data segment.
[0092] The next_idx of the first SRQE (SRQE 0) in container 0 points to the first SRQE (SRQE 4) in container 1. The next_idx of SRQE 4 points to the first SRQE (SRQE 12) in container 3. The next_idx of SRQE 12 points to the first SRQE (SRQE 8) in container 2. The next_idx of SRQE 8 is 16, which can point to a container whose cur_idx of the first SRQE is 16.
[0093] Here, cur_idx = 16 can also be used as a reserved extended container representation. When adding a new container later, you only need to set the cur_idx of the first SRQE of the new container to 16, and it can be accessed through the existing linked list without reconstructing the entire linked list.
[0094] The RDMA shared receive queue optimization method, apparatus, device, medium, and product disclosed in the above embodiments of this invention utilize the next_idx stored in the header of the first SRQE within a container. This index pointer allows direct location of the first SRQE in the next container, reducing the number of host memory accesses. Simultaneously, it supports out-of-order chaining between containers (e.g., container1→container3→container2), avoiding scheduling blockage caused by different QPs waiting for the release of preceding containers. This significantly reduces receive latency between the same QP and different QPs, meeting the high-performance transmission requirements of RDMA. The cur_idx provides precise location information for container reclamation. Software can quickly identify idle containers through cur_idx and re-link them to the SRQ tail without traversing the entire linked list, thus avoiding head blocking. Furthermore, only the first SRQE within a container maintains cur_idx and next_idx; non-first SRQEs do not require redundant fields, reducing memory resource waste and further improving the overall resource utilization of the SRQ.
[0095] In one possible implementation of step S202 above, when the receiving network card detects the arrival of RDMA data for a queue pair, if the queue pair is not bound to a container, it allocates a container for the queue pair from the head of the logical linked list, including:
[0096] If the queue pair is not bound to a container, then obtain a current container from the head of the logical linked list and bind the current container to the queue pair;
[0097] The receiving network interface card (NIC) preferentially allocates shared receive queue entries from the containers bound to the queue pairs, sequentially, to the arriving RDMA data, including:
[0098] Prioritize allocating shared receive queue entries to the RDMA data arriving in the queue pair from the current container bound to the queue pair in sequence;
[0099] When the current container is allocated, the index pointer in the header information of the first shared receive queue entry of the current container is read, and the information of the next container indicated by the index pointer is recorded in the shared receive queue context of the receiving network card.
[0100] Once all shared receive queue entries within the current container have been allocated, the next container is obtained based on the next container information pre-stored in the shared receive queue context and re-bound to the queue pair for use in allocating subsequent RDMA data to the queue pair.
[0101] In this embodiment, the receiving network card locates the first available container (i.e., the current container) in the logical linked list by querying the current linked list head pointer recorded in the Shared ReceiveQueue Context (SRQC). An available container refers to a container in the SRQ's logical linked list that is not bound to other queue pairs (QPs) and can be allocated to the target QP. The definition of an available container is not simply an unoccupied container; this determination must be combined with the binding status of the QPs to avoid different QPs competing for the same container.
[0102] The SRQC is used to store the core control information of the SRQ. Fields stored in the SRQC include, but are not limited to: the current chain head pointer, next container information, container allocation status, and QP binding identifiers for each container. The QP binding identifier is used to identify whether a container has been bound to a QP, ensuring that the receiving network interface card (NIC) can quickly identify the availability status of a container.
[0103] For example, if the chain head recorded in SRQC is Container0, and Container0 is not bound to any QP, when the RDMA data of QP1 is detected to arrive, the receiving network card will identify Container0 as the current container and bind Container0 to QP1, preparing to allocate SRQE in Container0 for the RDMA data of QP1; subsequently, if the RDMA data of QP2 is detected to arrive, since QP2 is not bound to a container, the receiving network card will query the next available chain head container (such as Container 1) from SRQC, bind Container1 to QP2, and allocate SRQE in Container 1 for QP2, thus achieving the isolation effect of different QPs consuming SRQE in different containers.
[0104] The receiving network interface card (NIC) allocates SRQEs to RDMA data arriving from the target QP sequentially according to the physical address order of the SRQEs within the bound container: the first RDMA data is allocated to the first SRQE of the bound container. For example, after QP1 is bound to Container 0, its first RDMA data is allocated to SRQE 0 within Container 0; subsequent RDMA data arriving from the same QP are allocated sequentially to the next SRQE within the bound container, until all SRQEs within the bound container have been allocated. That is, the same QP only consumes SRQEs within its bound container, avoiding cross-QP container resource consumption.
[0105] When the first SRQE of the current container (which is already bound to the target QP) is allocated, the receiving network card reads the next_idx in the header of the first SRQE of the container and writes the next container information corresponding to the next_idx into a special field in the SRQC to complete the preloading. The next container information may include, but is not limited to, the container start address and the address of the first SRQE. This preloading operation is associated with the target QP to ensure that the next container preloaded is allocated only to this QP.
[0106] Once all shared receive queue entries within the current container have been allocated, the receiving network card directly reads the pre-stored next container information from the SRQC, obtains the next container, and rebinds it to the current QP (instead of allocating it to other QPs) for use by the subsequent RDMA data allocation of that QP. For example, after all SRQEs in Container 0 bound to QP1 have been allocated, the receiving network card rebinds the pre-stored next container (such as Container 3) in the SRQC to QP1. The subsequent RDMA data of QP1 is directly allocated from SRQEs in Container 3 in sequence, realizing seamless switching of the same QP between containers.
[0107] The RDMA shared receive queue optimization method, apparatus, device, medium, and product disclosed in the above embodiments of this invention preload the next container information into the SRQC. The receiving-side network card completes the storage of subsequent container information during the current container allocation process. Simultaneously, combined with the contiguous physical address characteristics of SRQEs within containers, hardware cache prefetching technology can be fully utilized during sequential allocation, reducing the number of host memory accesses per SRQE allocation. This significantly reduces the average latency of RDMA data reception and improves RDMA transmission efficiency. Furthermore, the container-level allocation mode, combined with the pre-stored next container information in the SRQC, enables a seamless switching to the next container once the current container is used up. Thus, even in the face of high-burst RDMA data, the network card can quickly complete container switching through the preloaded context information, ensuring the continuity and stability of data reception.
[0108] In one possible implementation of the above embodiments, the method further includes:
[0109] After processing the data for each shared receive queue entry, the receiving network card generates a corresponding completion queue entry and reports the completion queue entry to the corresponding completion queue. The completion queue entry includes identification information to identify the shared receive queue entry.
[0110] The receiving software monitors the release status of shared receive queue entries within multiple containers at the container level, determining whether a container is in a completely idle state, including:
[0111] The receiving software completes the queue by polling it, and then receives and processes the queue entries.
[0112] The receiving software determines the corresponding shared receiving queue entry and the container to which the shared receiving queue entry belongs based on the identification information in the completed queue entry;
[0113] The receiving software updates the release status record of the container to which the shared receive queue entry belongs, marking that the shared receive queue entry has been released;
[0114] The receiving software queries the container's release status record. When it detects that the release status of each shared receiving queue entry in the container is marked as released, it determines that the container is in a completely idle state.
[0115] In this embodiment, after completing data processing for each shared receive queue entry, the receiving network card generates a corresponding completion queue entry and reports the completion queue entry to the corresponding completion queue, including:
[0116] After the receiving network card completes the data processing of a single SRQE, it immediately triggers the generation of the corresponding CQE to ensure that one SRQE corresponds to one CQE and avoid state omissions. Identification information for identifying the SRQE is written into the generated CQE, where the identification information can uniquely locate the target SRQE and its container.
[0117] The receiving network card reports the CQE to the CQ pre-bound to the QP to which the SRQE belongs.
[0118] Here, each QP can uniquely correspond to a dedicated CQ to avoid confusion of state information between different QPs.
[0119] Furthermore, the receiving software polls the CQs bound to each QP at a preset period and reads the CQEs to be processed in the queue. If a new CQE is detected, the subsequent processing procedure is triggered immediately to avoid the untimely release of SRQEs due to delayed polling.
[0120] The receiving software extracts the identification information of the SRQE from the CQE and determines the specific container corresponding to the SRQE.
[0121] In addition, the receiving software can maintain a corresponding release status record for each container, which may include the number of released SRQEs, the total number of SRQEs in the container, etc.
[0122] After locating the container, the receiving software can increment the count of released SRQEs in the release status record by 1 and mark the current SRQE as released to ensure real-time synchronization of the status record.
[0123] The receiving software can query the release status record of the target container and compare the number of released SRQEs with the total number of SRQEs in the container. If they match, the target container is determined to be in a completely idle state; if they do not match, the target container is determined to be in a partially idle or fully occupied state.
[0124] The RDMA shared receive queue optimization method, apparatus, device, medium, and product disclosed in the above embodiments utilize a strict binding mechanism where one SRQE corresponds to one CQE. The receiving-side network card generates a unique CQE for each processed SRQE, and the CQE contains unique identifier information that can uniquely locate the SRQE and its associated container, ensuring that the software can accurately associate the release status of each SRQE. By maintaining an independent release status record for each container and strictly comparing the released count with the total number to determine the completely idle state, it ensures that only containers where all SRQEs have been released are re-linked into the list. This avoids resource conflicts caused by misjudging some idle containers as available, thereby ensuring complete synchronization between the software and hardware's understanding of container status, significantly reducing the risk of scheduling errors, and improving system stability.
[0125] In one possible implementation of step S204 above, relinking the target container into the logical linked list for allocation includes:
[0126] The target container is relinked to the end of the logical linked list for allocation.
[0127] In this embodiment, after the receiving software determines that the target container is in a completely idle state, it reads information such as the cur_idx of the first SRQE in the target container and the total number of SRQEs in the container. By querying the tail pointer recorded in the SRQC, it directly locates the end container of the current logical linked list.
[0128] The receiving software reads the header of the first SRQE in the current end container and updates its next_idx field to the cur_idx of the first SRQE in the target container to establish the link relationship between the end container and the target container.
[0129] Furthermore, the next_idx of the first SRQE in the target container is set to the preset chain tail identifier to indicate that the target container is the last node of the current linked list and there are no other containers after it.
[0130] Update the tail pointer recorded in SRQC to the cur_idx of the first SRQE in the target container, so that SRQC synchronously records the latest tail position of the linked list; modify the status flag of the target container to ensure that the receiving network card can identify through SRQC that the container is in the logical linked list and can be allocated to newly arrived RDMA data during subsequent scheduling.
[0131] The RDMA shared receive queue optimization method, apparatus, device, medium, and product disclosed in the above embodiments re-link completely idle target containers to the end of the logical linked list, allowing temporarily idle containers to re-enter the scheduling link. This avoids resource waste caused by containers not being reclaimed, significantly improving SRQE utilization, especially in high-concurrency scenarios where it can more efficiently support multi-QP data reception requirements. It ensures that the logical linked list always maintains a complete chain structure, allowing the receiving-side network card to smoothly traverse from the head to the new tail of the chain during scheduling, without interrupting or reconstructing the linked list due to container reclamation, thereby increasing the continuity of linked list scheduling.
[0132] In one possible implementation, a specific embodiment is provided in which, in the initial stage, the receiving-side software sequentially links the containers into a logical linked list. Please refer to [reference needed]. Figure 5 , Figure 5 This is a schematic diagram of the initial sequence chain structure of the SRQ container in an RDMA shared receive queue optimization method provided in this embodiment.
[0133] like Figure 5As shown, the SRQ includes four containers, namely container 0 to container 3. Each container can contain four SRQEs. In the header of the first SRQE in each container, the next_idx field points to the physical address of the first SRQE in the subsequent container. For example, the next_idx of SRQE0 in container 0 is equal to the address of the first SRQE (SRQE4) in container 1. container 0 points to container 1; container 1 points to container 2, and so on.
[0134] Secondly, during the normal packet reception phase of the hardware, taking the packet reception process of the QP2 and QP3 queue pairs as an example, please refer to [link / reference needed]. Figure 6 , Figure 6 This is a schematic diagram of the packet receiving sequence of an RDMA shared receive queue optimization method provided in an embodiment of this disclosure.
[0135] like Figure 6 As shown, the package receiving stage includes the following specific steps:
[0136] Step 1: Processing the first message (Send1) of QP2.
[0137] Specifically, when the receiving network card receives the Send1 message from QP2, it retrieves container0 from the head of the SRQ logical linked list (indicated by the head pointer of SRQC) as the current container and binds container0 to QP2; it allocates the first SRQE (SRQE0) in container0 for receiving data; it reads the next_idx of SRQE0 (pointing to SRQE4 of container1), records this information in SRQC, and completes the preloading of the next container; at this time, the head pointer of SRQC is updated to SRQE4 of container1, and the original head container 0 is marked as occupied because it is already bound to QP2.
[0138] Step 2: Processing subsequent messages (Send2, Send3) of QP2.
[0139] Specifically, when receiving Send2 and Send3 messages from QP2, since QP2 is already bound to container0 and has not been exhausted, the next SRQE (SRQE1, SRQE2) in container0 is directly allocated in sequence; there is no need to wait for the completion information of SRQE0 to return, nor is there a need to read the host memory to obtain the index of the next entry, reducing the access latency between the same QP message; at the same time, the receiving network card maintains the SRQE allocation breakpoint in container0 in real time through the context of QP2 (such as SRQE2 already allocated), ensuring that subsequent messages can be allocated continuously.
[0140] Step 3: Processing the first message (Send1) of QP3.
[0141] Specifically, when the receiving network card receives the Send1 message from QP3, it detects that QP3 is not bound to a container and does not need to wait for the SRQE0 corresponding to Send1 from QP2 to return completion information (different QPs bind containers independently, and resources are isolated); it obtains container 1 from the SRQ chain head (currently container 1) as the current container and binds container 1 to QP3; it allocates its first SRQE (SRQE4) for receiving data; it reads the next_idx of SRQE4 (pointing to SRQE8 of container2) and updates it in SRQC (this preloaded information is associated with QP3) for subsequent allocation by QP3.
[0142] Step 4: Processing of subsequent QP3 messages (Send2).
[0143] Specifically, when the Send2 message of QP3 is received, since QP3 is already bound to container1 and has not been consumed, the next SRQE (SRQE5) in container1 is directly allocated in sequence; there is no need to wait for the completion information of the preceding SRQE (SRQE4), thus reducing the scheduling delay between different QPs.
[0144] Step 5: Switching after the container is exhausted.
[0145] Specifically, when QP2 consumes all SRQEs (SRQE0 to SRQE3) in container0, the receiving network card reads the next container information associated with QP2 (SRQE12 of container 3) from the SRQC and rebinds container 3 to QP2; when QP3 consumes all SRQEs (SRQE4 to SRQE7) in container1, the network card directly reads the pre-stored next container information (such as SRQE8 of container 2) from the SRQC and rebinds container 2 to QP3; there is no need to re-query the linked list head for QP2 and QP3, the container switching is completed directly based on the associated pre-loaded information, realizing seamless switching of the same QP between containers.
[0146] Step 6: CQE reports.
[0147] Specifically, after each SRQE completes data processing, the receiving network card generates a corresponding CQE. Based on the QP to which the SRQE belongs (such as SRQE0 of QP2 and SRQE4 of QP3), the CQE is reported to the CQ specifically bound to that QP. The CQE contains the identification information of the SRQE, which is used by the software to accurately locate the binding relationship between the container and the QP.
[0148] Then, in the software container recycling phase, please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of the SRQ container out-of-order chain structure of an RDMA shared receive queue optimization method provided in this embodiment. Figure 7 As shown, the container recycling stage includes the following specific steps:
[0149] The receiving software polls the CQ to obtain the CQE, and locates the corresponding SRQE and its container based on the identification information in the CQE (e.g., locating container3 through the SRQE address); the software updates the container release status record, incrementing the release count by 1 for each SRQE released.
[0150] When the software detects that the released count of a container is equal to the total number of its SRQEs (e.g., all 4 SRQEs of container3 have been released), it determines that the container is in a completely idle state.
[0151] The software reads the cur_idx of the first SRQE in container3; locates the end container of the current linked list (such as container1) through the tail pointer of SRQC, updates the next_idx of its first SRQE to the cur_idx of container3; sets the next_idx of the first SRQE in container3 to the tail identifier (NULL), and updates the tail pointer of SRQC to the cur_idx of container3; marks container3 as available, making it available for hardware reallocation.
[0152] After container2 completes the release of all SRQE, the software repeats the step of out-of-order chaining of containers to the end of the chain, chaining them to the current end of the chain (after container3), thus realizing out-of-order reuse of containers.
[0153] In one embodiment, an RDMA shared receive queue optimization apparatus 800 is provided, which corresponds one-to-one with the RDMA shared receive queue optimization method in the above embodiments. For example... Figure 8 As shown, the RDMA shared receive queue optimization device 800 includes a container partitioning module 801, a resource allocation module 802, a status monitoring module 803, and a container reclamation module 804. The detailed descriptions of each functional module are as follows:
[0154] The container partitioning module 801 is used to divide the shared receiving queue into multiple containers and connect the multiple containers through a logical linked list; wherein each container includes multiple shared receiving queue entries.
[0155] The resource allocation module 802 is used to allocate a container for the queue pair from the head of the logical linked list when the receiving network card detects the arrival of RDMA data of the queue pair. If the queue pair is not bound to a container, the receiving network card will prioritize allocating shared receive queue entries to the arriving RDMA data from the containers bound to the queue pair in order.
[0156] The status monitoring module 803 is used to monitor the release status of shared receive queue entries in multiple containers at the container level, and to determine whether the container is in a completely idle state.
[0157] The container recycling module 804 is used to relink the target container into the logical linked list for allocation, independent of the state of other containers among multiple containers, when the target container is detected to be in a completely idle state.
[0158] In one embodiment, multiple shared receive queue entries within the container are physically contiguous in memory.
[0159] In one embodiment, the linking relationships of the logical linked lists are maintained by an index pointer stored in the header information of the first shared receive queue entry in each container.
[0160] In one embodiment, the resource allocation module 802 is used to obtain a current container from the head of the logical linked list and bind the current container to the queue pair if the queue pair is not bound to a container.
[0161] Prioritize allocating shared receive queue entries to the RDMA data arriving in the queue pair from the current container bound to the queue pair in sequence;
[0162] When the current container is allocated, the index pointer in the header information of the first shared receive queue entry of the current container is read, and the information of the next container indicated by the index pointer is recorded in the shared receive queue context of the receiving network card.
[0163] Once all shared receive queue entries within the current container have been allocated, the next container is obtained based on the next container information pre-stored in the shared receive queue context and re-bound to the queue pair for use in allocating subsequent RDMA data to the queue pair.
[0164] In one embodiment, the RDMA shared receive queue optimization apparatus 800 further includes: a queue reporting module 805, wherein:
[0165] The completion queue reporting module 805 is used to generate corresponding completion queue entries after the receiving-side network card completes data processing for each shared receiving queue entry, and to report the completion queue entries to the corresponding completion queues; wherein, the completion queue entries include identification information for identifying shared receiving queue entries;
[0166] The status monitoring module 803 is specifically used to receive and process queue entries by polling the queue.
[0167] Based on the identification information in the completed queue entry, determine the corresponding shared receive queue entry and the container to which the shared receive queue entry belongs;
[0168] Update the release status record of the container to which the shared receive queue entry belongs, marking that the shared receive queue entry has been released;
[0169] Query the container's release status record. When the release status of all shared receive queue entries within the container is marked as released, the container is determined to be in a completely idle state.
[0170] In one embodiment, the container recycling module 804 is specifically used to relink the target container to the end of the logical linked list for allocation.
[0171] It should be noted that the RDMA shared receive queue optimization device provided in the above embodiments is only illustrated by the division of the above program modules when implementing the corresponding RDMA shared receive queue optimization method. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the above system can be divided into different program modules to complete all or part of the processing described above. In addition, the system provided in the above embodiments and the corresponding Figure 2 The embodiments of the methods shown belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.
[0172] This disclosure also provides an electronic device having the above-described features. Figure 8 The RDMA shared receive queue optimization device is shown.
[0173] Please see Figure 9 , Figure 9 This is a schematic diagram of another RDMA shared receive queue optimization device provided in this disclosure embodiment, as shown below. Figure 9 As shown, the electronic device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take a processor 10 as an example.
[0174] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.
[0175] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.
[0176] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0177] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.
[0178] The electronic device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.
[0179] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touch screen.
[0180] The electronic device also includes a communication interface for communicating with other devices or communication networks.
[0181] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.
[0182] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0183] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A method for optimizing a shared receive queue in RDMA, characterized in that, The method includes: The receiving software divides the shared receiving queue into multiple containers and links these containers together using a logical linked list; each container contains multiple shared receiving queue entries. When the receiving network interface card (NIC) detects the arrival of RDMA data for a queue pair, if the queue pair is not bound to a container, it allocates a container for the queue pair from the head of the logical linked list. The receiving NIC prioritizes allocating shared receive queue entries to the arriving RDMA data from the currently bound containers of the queue pair in sequence. When allocating the current container, it reads the index pointer in the header information of the first shared receive queue entry of the current container and records the information of the next container indicated by the index pointer in the shared receive queue context of the receiving NIC. After all shared receive queue entries in the current container have been allocated, it obtains the next container according to the next container information pre-stored in the shared receive queue context and rebinds it to the queue pair for use in allocating subsequent RDMA data to the queue pair. The receiving software monitors the release status of shared receiving queue entries within multiple containers at the container level, and determines whether the container is in a completely idle state. When the receiving software detects that the target container is in the completely idle state, it relinks the target container to the end of the logical linked list for allocation, independent of the states of other containers among the multiple containers.
2. The method according to claim 1, characterized in that, The multiple shared receive queue entries within the container are physically contiguous in memory.
3. The method according to claim 1, characterized in that, The linking relationships of the logical linked lists are maintained by the index pointers stored in the header information of the first shared receive queue entry in each container.
4. The method according to claim 1, characterized in that, The method further includes: After completing data processing for each shared receive queue entry, the receiving network card generates a corresponding completion queue entry and reports the completion queue entry to the corresponding completion queue; wherein, the completion queue entry includes identification information for identifying the shared receive queue entry; The receiving software monitors the release status of shared receive queue entries within multiple containers at the container level, determining whether a container is in a completely idle state, including: The receiving software receives and processes entries in the completion queue by polling the completion queue. The receiving software determines the corresponding shared receiving queue entry and the container to which the shared receiving queue entry belongs based on the identification information in the completed queue entry. The receiving software updates the release status record of the container to which the shared receiving queue entry belongs, marking that the shared receiving queue entry has been released; The receiving software queries the release status record of the container. When it detects that the release status of each shared receiving queue entry in the container is marked as released, it determines that the container is in a completely idle state.
5. An RDMA shared receive queue optimization device, characterized in that, The device includes: a container partitioning module, a resource allocation module, a status monitoring module, and a container recycling module, wherein: The container partitioning module is used to divide the shared receiving queue into multiple containers and connect the multiple containers through a logical linked list; wherein each container includes multiple shared receiving queue entries. The resource allocation module is used to allocate a container for the queue pair from the head of the logical linked list when RDMA data of the queue pair is detected to arrive, if the queue pair is not bound to a container; prioritize allocating shared receive queue entries to the arriving RDMA data from the currently bound containers of the queue pair in sequence; when the allocation of the current container begins, read the index pointer in the header information of the first shared receive queue entry of the current container, and record the information of the next container indicated by the index pointer in the shared receive queue context of the receiving network card; after all shared receive queue entries in the current container have been allocated, obtain the next container according to the next container information pre-stored in the shared receive queue context and rebind it to the queue pair for use in allocating subsequent RDMA data of the queue pair; The status monitoring module is used to monitor the release status of shared receive queue entries in multiple containers at the container level, and to determine whether the container is in a completely idle state. The container recycling module is used to relink the target container to the end of the logical linked list for allocation, independent of the states of other containers among the multiple containers, when the target container is detected to be in the completely idle state.
6. The apparatus according to claim 5, characterized in that, Within each container formed by the container partitioning module, multiple shared receive queue entries have contiguous physical addresses in memory.
7. The apparatus according to claim 5, characterized in that, The device further includes: The completion queue reporting module is used to generate corresponding completion queue entries after the receiving-side network card completes data processing for each shared receiving queue entry, and to report the completion queue entries to the corresponding completion queue; wherein, the completion queue entries include identification information for identifying the shared receiving queue entries; The status monitoring module is specifically used for: The completion queue is polled to receive and process entries in the completion queue. Based on the identification information in the completed queue entry, determine the corresponding shared receiving queue entry and the container to which the shared receiving queue entry belongs; Update the release status record of the container to which the shared receive queue entry belongs, and mark the shared receive queue entry as released; The container is queried for release status records. When the release status of each shared receive queue entry in the container is marked as released, the container is determined to be in a completely idle state.
8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the RDMA shared receive queue optimization method as described in any one of claims 1 to 4 when executing the computer program.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the RDMA shared receive queue optimization method as described in any one of claims 1 to 4.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the RDMA shared receive queue optimization method as described in any one of claims 1 to 4.
Citation Information
Patent Citations
RDMA-based container network, communication method and computer readable medium
CN112256407A
Processing method and device for mutual exclusion of tasks in heterogeneous network and related products
CN113986498A