A method, device and medium for constructing an RDMA completion queue element based on software

Abstract

This invention provides a method, device, and medium for constructing RDMA completion queue elements based on software, relating to the field of RDMA network card technology. The method includes: allocating a first buffer and a second buffer for each completion queue; reading hardware completion queue elements from the first buffer and returning them to the upper-layer application until there are no unread hardware completion queue elements in the first buffer or the returned count reaches the requested number of completion queue elements; constructing a corresponding number of software completion queue elements with a status of "work request refresh error" for each pending queue; storing the constructed software completion queue elements in the second buffer and returning them to the upper-layer application, updating the remaining number of nodes to be constructed; and returning the final returned count as the return value of the polling interface to the upper-layer application. This invention can restore the hardware's expected refresh error completion queue element reporting behavior without modifying the upper-layer application and hardware.

Description

Technical Field

[0001] This invention relates to the field of RDMA network interface card technology, and in particular to a method, device and medium for constructing RDMA queue elements based on software. Background Technology

[0002] In Remote Direct Memory Access (RDMA) technology, upper-layer applications interact with the RDMA network card hardware through queue pairs (QPs). After completing a work request, the hardware generates a Completion Queue Element (CQE) in the Completion Queue (CQ) to notify the application. When a QP needs to switch to an error state due to connection loss or an anomaly, according to standard behavior, the hardware should generate a CQE with a status of "Work Request Refresh Error" (i.e., flush err CQE) for each unconsumed Work Queue Element (WQE) to notify the application to reclaim related resources. However, some RDMA hardware no longer reports this refresh error CQE after functional changes. This causes upper-layer applications (such as SPDK, GPFS, etc.) to be unable to correctly determine the completion status of QPs due to insufficient completion notifications, thus failing to destroy QPs or release associated memory in a timely manner, resulting in resource leaks or even system anomalies. Existing drivers lack a software simulation mechanism for missing refresh error CQEs and are incompatible with the aforementioned hardware changes. There is an urgent need for a method that can restore the hardware's proper refresh error completion queue element reporting behavior without modifying the upper-layer application and hardware. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: According to a first aspect of this application, a method for completing queue elements based on software-constructed RDMA is provided, the method comprising the following steps: S100, allocate a first buffer and a second buffer to each completion queue; wherein, the first buffer is used to store the completion queue elements written by RDMA hardware, and the second buffer is used to store the completion queue elements constructed by the driver; the driver maintains a linked list of queue pairs to be processed, each node in the linked list of queue pairs to be processed corresponds to a queue pair that has entered an error state and still needs to report refresh error completion queue elements; each node records the identifier of the queue pair and the number of refresh error completion queue elements to be constructed in the queue pair. In response to the upper-layer application calling the polling interface and passing in the number of elements in the completion queue of the request, the following steps are executed: S200: Read hardware completion queue elements from the first buffer and return them to the upper layer application until there are no unread hardware completion queue elements in the first buffer or the returned count reaches the requested number of completion queue elements; proceed to S300. S300, if the count has not reached the number of completed queue elements requested, then obtain the queue pair node to be processed from the queue pair list to be processed, construct a corresponding number of software completion queue elements with the status of work request refresh error for each queue pair to be processed, store the constructed software completion queue elements into the second buffer, read the software completion queue elements from the second buffer and return them to the upper layer application, and update the remaining number of nodes to be constructed. S400, the final returned count is returned to the upper layer application as the return value of the polling interface.

[0004] According to another aspect of this application, a non-transitory computer-readable storage medium is also provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described method for completing queue elements based on software-constructed RDMA.

[0005] According to another aspect of this application, an electronic device is also provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0006] The present invention has at least the following beneficial effects: The present invention discloses a method for constructing RDMA completion queue elements based on software. This method allocates independent first and second buffers to each completion queue, storing hardware-generated and driver-constructed completion queue elements respectively. It also maintains a linked list of pending queue pairs to record queue pairs that have entered an error state and still require reporting and refreshing error completion queue elements, along with their remaining number to be constructed. This allows the driver, when calling the polling interface in the upper-layer application, to first return hardware completion queue elements from the first buffer. When hardware completion queue elements are insufficient, it automatically retrieves pending queue pair nodes from the linked list, dynamically constructs the corresponding number of software completion queue elements with the status of "work request refresh error," and returns them to the upper-layer application. This solution accurately simulates the behavior of refreshing error completion queue elements that the hardware should generate without modifying the hardware or upper-layer application code. It solves the application-layer resource reclamation blocking problem caused by hardware function changes. Furthermore, the dual-buffer design avoids mutual interference between software-constructed and hardware-generated completion queue elements, ensuring semantic compatibility and data integrity of the upper-layer application's polling interface, and significantly improving the hardware adaptability and system stability of the RDMA driver. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A flowchart illustrating a method for constructing queue elements based on software-based RDMA, as provided in an embodiment of the present invention. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0010] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0011] The following will refer to Figure 1 The flowchart shown illustrates a method for constructing RDMA queue elements based on software, which introduces such a method.

[0012] The method for constructing RDMA based on software to complete queue elements includes the following steps: S100, allocate a first buffer and a second buffer to each completion queue; wherein, the first buffer is used to store the completion queue elements written by the RDMA hardware, and the second buffer is used to store the completion queue elements constructed by the driver; the driver maintains a linked list of queue pairs to be processed, each node in the linked list of queue pairs to be processed corresponds to a queue pair that has entered an error state and still needs to report refresh error completion queue elements; each node records the identifier of the queue pair and the number of refresh error completion queue elements to be constructed in the queue pair.

[0013] In this embodiment, the first buffer is a shared memory region between the hardware and the driver, typically implemented through DMA mapping. When the driver creates the Completion Queue (CQ), it calls a kernel memory allocation function (such as `dma_alloc_coherent` or `mmap` during user-space `ibv_create_cq`) to allocate a contiguous block of physical memory and registers it with the RDMA network card hardware, enabling the hardware to directly write to Completion Queue Elements (CQEs). This buffer uses a circular queue structure and contains the following metadata: base_addr: The starting virtual address of the buffer.

[0014] size: The maximum number of CQEs that can be accommodated (usually a power of 2, such as 1024).

[0015] head: The driver read pointer, indicating the next CQE position to be read.

[0016] tail: Hardware write pointer, updated by hardware, indicating the next writable position.

[0017] Each time the hardware completes a work request, it writes a CQE to the position pointed to by tail, and then the tail pointer advances by one position. The driver determines whether a new CQE has arrived by reading head and tail.

[0018] For example: Assuming the CQ depth is 1024, a memory block of size 1024 × sizeof(struct ibv_wc) (approximately 32KB) is allocated during creation, with the address aligned to the page boundary. The driver initializes the head to 0, and the hardware tail is initialized to 0. After the hardware writes to the CQE, the tail becomes 1. The driver reads that head(0) and tail(1) are not equal, thus reading a CQE.

[0019] The second buffer is a purely software-maintained memory area, not shared with hardware. The driver also allocates a memory block (its size may be the same as or smaller than the first buffer) to temporarily store CQEs (Combine Queues) constructed by the driver software. This buffer also uses a circular queue, but reads and writes are controlled by the driver itself. sw_buf_addr: The starting address of the second buffer.

[0020] sw_buf_size: The maximum number of software CQEs that can be accommodated.

[0021] sw_head: Software read pointer, indicating the next software CQE to be returned to the upper-layer application.

[0022] sw_tail: Software write pointer, indicating the next free position when the driver constructs a new CQE.

[0023] When the driver needs to construct a refresh error (CQE), it writes the constructed CQE to the sw_tail location and then increments sw_tail. In subsequent polling calls, the driver reads the returned value from the sw_head location and sends it to the upper-layer application.

[0024] In this embodiment, the dual-buffer design physically separates the hardware-generated CQE from the software-constructed CQE, avoiding the pollution of the hardware circular queue by the software-constructed CQE and preventing asynchronous hardware writes from interfering with software data; at the same time, it allows the driver to return the hardware CQE first and then the software CQE in a single polling call, maintaining semantic consistency with standard RDMA behavior.

[0025] The driver maintains a separate linked list of pending queue pairs, err_state_qp_list, for each completion queue (because CQE needs to be associated with a specific CQ). Each node in the linked list corresponds to a queue pair that has entered an error state and still needs to report and refresh the error completion queue elements. Each node records the identifier of the queue pair and the number of remaining pending refresh error completion queue elements to be constructed in the queue pair. The number of remaining pending elements is equal to the total number of work queue elements in the queue pair that have not yet been consumed by the hardware.

[0026] Each node corresponds to a queue pair (QP) that has entered the IBV_QPS_ERR state (error state) and still has unconsumed Work Queue Elements (WQEs) that need to report flush errors. The initial value of pending_flush_cnt is equal to the total number of WQEs that have not yet been taken by the hardware in the send queue (SQ) and receive queue (RQ) of that QP. During subsequent polling, the driver decrements this value by one for each flush error CQE constructed and returned, and removes the node from the linked list when it reaches zero.

[0027] Linked Lists and CQ Associations: Since a QP may be associated with multiple CQs (e.g., a QP's SQ is associated with CQ1, and its RQ is associated with CQ2), the driver needs to add nodes to the pending linked lists of all associated CQs. An err_state_qp_list head node is maintained in each CQ structure.

[0028] By recording the number of pending QPs for each error in a linked list, the driver can accurately determine how many refresh error CQEs need to be added, avoiding over- or under-construction; it also supports fair handling of multiple QPs and issuing errors at the same time.

[0029] In response to the upper-layer application calling the polling interface and passing in the number of elements in the completion queue of the request, the following steps are executed: In this embodiment, when an upper-layer application (such as SPDK, GPFS, etc.) calls the polling interface (e.g., ibv_poll_cq), it passes in the maximum number of CQEs to be obtained, num_entries, and drives the execution of steps S200~S400.

[0030] It should be noted that each node in the pending list belongs to only one completion queue, and the "remaining number of nodes to be constructed" recorded in the node is the number of refresh errors (CQEs) that the completion queue needs to report. When a queue pair is associated with multiple completion queues, the driver will create an independent node in the linked list of each associated completion queue and set the corresponding number of nodes to be constructed according to the actual queue type (send or receive) associated with that completion queue, thereby ensuring that the number of refresh errors (CQEs) obtained by each completion queue is completely consistent with the hardware standard behavior.

[0031] S200: Read hardware completion queue elements from the first buffer and return them to the upper layer application until there are no unread hardware completion queue elements in the first buffer or the returned count reaches the requested number of completion queue elements; proceed to S300.

[0032] The driver first checks the read pointer `head` and the hardware write pointer `tail` of the first buffer. If `head == tail`, it means there is no new hardware CQE. Otherwise, the driver sequentially retrieves CQEs from the `head` position, converts them into the `struct ibv_wc` format required by the upper-layer application (including fields such as status, opcode, and `qp_num`), fills them into the `wc` array passed in by the application, increments the returned count `polled`, and increments the `head` pointer (modulo the buffer depth). This process is repeated until `head == tail` or `polled == num_entries`.

[0033] For example: Suppose the application calls ibv_poll_cq(cq,16,wc), at which point there are 3 hardware CQEs in the first buffer. The driver sequentially retrieves these 3 CQEs and fills them into wc[0], wc[1], and wc[2], polled becomes 3, and head advances by 3. If head catches up with tail afterward, the hardware CQEs are empty, and S200 ends.

[0034] This step faithfully reflects the actual CQE generated by the hardware, ensuring that upper-layer applications can handle normal completion and some error events in a timely manner, without losing hardware information due to software intervention. At the same time, by limiting the number of responses to not exceed the user's request, it adheres to the behavioral conventions of standard polling interfaces.

[0035] S300, if the count returned still does not reach the number of completed queue elements requested, then obtain the queue pair node to be processed from the queue pair list to be processed, construct a corresponding number of software completion queue elements with the status of work request refresh error for each queue pair to be processed, store the constructed software completion queue elements into the second buffer, read from the second buffer and return to the upper layer application, and update the remaining number of nodes to be constructed.

[0036] Furthermore, step S300 includes the following steps: Step S300 includes the following steps: S310, traverse each node in the pending queue linked list in sequence, construct a software completion queue element with a status of work request refresh error each time according to the remaining number of elements to be constructed recorded in the node, and store the software completion queue element into the second buffer.

[0037] After completing S200 (reading the hardware CQE), if the returned count is still less than the user-requested num_entries, the driver proceeds to S310. The driver first obtains the head node of the current CQ's pending queue pair linked list err_state_qp_list, and then begins traversal.

[0038] For each node, the driver performs the following operations: Get node information: Read qp_id (queue pair identifier) ​​and pending_flush_cnt (number of remaining flush errors CQEs to be constructed) from the node.

[0039] Constructing a single software CQE: Allocate (or reuse on the stack) a struct ibv_wc structure in memory and populate it with the following fields: status=IBV_WC_WR_FLUSH_ERR (Job request refresh error); qp_num = node->qp_id; opcode=IBV_WC_RECV (usually the WQE in the receive queue has not been consumed, but can also be dynamically set according to the QP type); byte_len=0; wr_id=0 (or obtain from the unfinished WQE in QP; a mapping table can be maintained if accurate simulation is required). vendor_err=0; wc_flags=0.

[0040] Store in the second buffer: The driver obtains the write pointer sw_tail of the current CQ's second buffer (sw_buf), copies the constructed ibv_wc structure to sw_buf->addr+sw_tail / sizeof(wc), and then increments sw_tail (modulo the buffer depth). Simultaneously, handling the wraparound of the circular buffer is required.

[0041] For example: Assuming the second buffer depth is 64 and the current sw_tail=10, after the driver constructs the first flush error (CQE), it copies the error to the 10th slot, and sw_tail becomes 11. If sw_tail reaches 64, it wraps back to 0.

[0042] By storing software CQEs in a dedicated buffer, data consistency is ensured by avoiding mixed storage with hardware CQEs. Simultaneously, separate storage allows the driver to pre-construct a batch of CQEs (such as background threads) before polling calls, improving response efficiency. Furthermore, the circular buffer design supports high-frequency construction and retrieval without requiring dynamic memory allocation.

[0043] S320: For each software-completed queue element stored, decrement the remaining number of elements to be constructed for that node by one, and increment the returned count by one.

[0044] After each successful storage of a software CQE into the second buffer, the driver performs the following updates: node->pending_flush_cnt--: Decrement the number of remaining nodes to be constructed in the current node by one.

[0045] polled++: Increments the polled count by one. Note that this polled variable has already accumulated the number of hardware CQE returns in S200, and continues to accumulate in S300.

[0046] The driver needs to guarantee the atomicity of these two decrement and increment operations (no additional lock is needed in a single-threaded polling context, but attention should be paid to the interrupt context, as the polling interface is usually executed in the process context).

[0047] For example: Continuing the previous example, node A initially has pending_flush_cnt=10 and currently polled=3. After storing the first software CQE, pending_flush_cnt becomes 9 and polled becomes 4. And so on.

[0048] This step precisely maintains the remaining number of nodes to be constructed and the overall count of returned nodes for each node, ensuring that the construction process does not exceed the user's request limit or miss any nodes (each unconsumed WQE corresponds to a flush CQE). When polled reaches num_entries, construction can be stopped in time to avoid resource waste.

[0049] S330, repeat S310 and S320 until the remaining number of nodes to be constructed is zero, or the count has reached the number of completed queue elements requested.

[0050] The driver repeatedly executes S310 and S320 in a single loop, processing the next software CQE for the current node each time, until one of the following conditions is met: Condition 1: The pending_flush_cnt of the current node becomes 0. At this point, all pending flush errors (CQEs) for this QP have been constructed.

[0051] Condition 2: Polled reaches num_entries. At this point, the number of CQEs requested by the user has been satisfied, and construction stops.

[0052] When condition 2 is triggered prematurely, the driver immediately exits the loop, no longer processing the remaining number of nodes in the current node (which is retained in the node), and does not continue traversing subsequent nodes. When condition 1 is triggered, the driver enters S340 to delete the node, and then continues processing the next node in the linked list (if polled has not yet reached num_entries).

[0053] For example: Node A has 10 pending constructions, currently polled=3, and user request 16. The driver iteratively constructs the 1st to 10th software CQEs, incrementing polled by 1 for each construction. When the 10th is constructed, polled=13, and node A's pending_flush_cnt becomes 0, triggering condition 1. The driver deletes node A; at this point, polled=13<16, and continues processing node B. Node B has 5 pending constructions. After constructing the 1st to 3rd software CQEs, polled reaches 16, triggering condition 2, and construction stops. Node B retains 2 remaining.

[0054] This loop logic ensures the timely response of the polling interface—the driver returns as much as possible as the number of user requests, preventing blocking due to excessively long software construction processes. Simultaneously, it supports batch construction across multiple polling calls, making it suitable for high-concurrency scenarios (where multiple QPs fail simultaneously, and a single poll cannot return all flush CQEs).

[0055] S340, if the remaining number of nodes to be constructed becomes zero, then the node is removed from the linked list and released.

[0056] When the pending_flush_cnt of the current node becomes 0 in step S330, the driver executes the node deletion operation: Remove the node from the err_state_qp_list linked list (e.g., list_del(&node->list)).

[0057] Release the memory occupied by the node (e.g., kfree(node) or free(node)).

[0058] If the linked list is empty, then no further software CQE can be constructed for the S300.

[0059] It should be noted that deletion operations must be performed while holding an appropriate lock (if multiple threads access the same CQ, a spinlock may be needed to protect the linked list operations). Typically, the RDMA polling interface is called in a single thread, and the driver can be designed as a lock-free linked list (using RCU or a single producer-single consumer pattern).

[0060] This step ensures timely release of memory for nodes that have completed tasks, preventing memory leaks. Once all flush CQEs for erroneous QPs have been constructed, the linked list is cleared, preventing unnecessary construction attempts by the driver and reducing CPU overhead. Furthermore, after a node is deleted, the QP no longer appears in the subsequent polling list, adhering to the semantics of "each unconsumed WQE generates only one flush CQE".

[0061] Furthermore, the generation triggering conditions for nodes in the linked list of the queue to be processed include a first triggering condition and a second triggering condition: The first triggering condition is: when the driver detects that the upper layer application has modified the state of a certain queue pair to an error state by modifying the queue pair state interface, it obtains the sum of the number of work queue elements in the sending queue and receiving queue of the queue pair that have not yet been consumed by the hardware, and uses it as the number of refresh error completion queue elements to be constructed. It then creates a node containing the queue pair identifier and the number of elements, and adds it to the list of pending queue pairs of all completion queues associated with the queue pair.

[0062] When an upper-layer application (such as SPDK calling rdma_disconnect or directly calling ibv_modify_qp) switches the state of a queue pair (QP) to IBV_QPS_ERR (error state), the driver executes this triggering logic in the kernel or user-space function that modifies the state.

[0063] The specific implementation steps are as follows: Step 1: Detect status changes The driver checks if attr->qp_state is IBV_QPS_ERR and if the current QP state is not ERR. If so, it proceeds to the processing flow.

[0064] Step 2: Calculate the number of unconsumed work queue elements (WQE) The driver needs to count the number of WQEs that have not yet been retrieved by hardware in the send queue (SQ) and receive queue (RQ) of the QP. Each QP's context structure (e.g., struct xsc_qp) maintains producer pointers (producerindex) and consumer pointers (consumerindex) for the SQ and RQ respectively. Producer pointer (sq.prod): The location where the driver has been sent to the last WQE in the hardware (i.e., the number of times the driver has written to the doorbell).

[0065] Consumer pointer (sq.cons): The position of the last WQE that the hardware has taken and processed.

[0066] The number of unconsumed WQEs in SQ = sq.prod - sq.cons (the modulo operation of the circular queue needs to be considered, but since the difference will not exceed the queue depth, it can be directly subtracted). Similarly, the number of unconsumed WQEs in RQ = rq.prod - rq.cons.

[0067] The total number of pending items is calculated as follows: pending = (sq.prod - sq.cons) + (rq.prod - rq.cons).

[0068] Step 3: Create linked list nodes The driver calls a memory allocation function (such as kmalloc in kernel mode, malloc in user mode) to create a structerr_qp_node node and fill in: qp_id=qp->qp_num; pending_flush_cnt=pending.

[0069] Step 4: Add to the linked list of all associated completion queues A QP can be associated with a send completion queue (sq_cq) and a receive completion queue (rq_cq), and they may also share the same CQ. The driver needs to obtain all CQs associated with the QP (usually two) and perform the following operations for each CQ: Get the head of the pending list in CQ (cq->err_state_qp_list); Insert a node at the end of the linked list (using list_add_tail).

[0070] If pending==0, it means that the QP has no unconsumed WQEs, so there is no need to construct any flush CQEs, the driver should not create nodes, and should return directly.

[0071] For example: Suppose there is a QP with qp_id=100, SQ depth of 128, current sq.prod=50, sq.cons=30 (20 WQEs are not consumed in the SQ); RQ depth is 64, rq.prod=40, rq.cons=35 (5 WQEs are not consumed in the RQ). The application calls ibv_modify_qp to change QP100 to ERR state. The driver calculates pending=20+5=25, creates a node {qp_id=100, pending_flush_cnt=25}, and adds it to the pending list of the CQ associated with this QP (assuming the SQ and RQ use the same CQ, CQ number 2).

[0072] Subsequently, when the application polls CQ2, the driver will construct 25 flush err CQEs from that node according to the S300 steps.

[0073] For applications like SPDK that actively disconnect and modify the QP state, this trigger condition can promptly capture state changes and prepare the flush CQE information to be constructed before the application calls the polling interface, preventing the application from freezing due to not receiving flush CQEs. By calculating the actual number of unconsumed WQEs in SQ and RQ, it ensures that the number of constructed flush CQEs strictly matches the number that the hardware should generate, conforming to the InfiniBand specification. It is only triggered when the QP state actually changes to ERR, without introducing additional path overhead.

[0074] The second triggering condition is as follows: When the driver reads a completion queue element with a status of general error and not a work request refresh error during the process of reading hardware completion queue elements from the first buffer, it determines the queue pair to which the completion queue element belongs, obtains the sum of the number of work queue elements that have not yet been consumed by the hardware in the sending queue and receiving queue of the queue pair, creates a node and adds it to the pending queue pair linked list of each completion queue associated with the queue pair.

[0075] This condition is triggered when the driver executes S200 (reading hardware CQE from the first buffer) and reads a CQE with a status of IBV_WC_GENERAL_ERR (general error) and not IBV_WC_WR_FLUSH_ERR.

[0076] This situation commonly occurs when a fatal hardware error occurs (such as a corrupted QP or out-of-bounds memory access), but the hardware will still generate a CQE (non-flush type) for this error. However, there may still be many unconsumed WQEs in this QP, and the hardware will no longer generate a flush CQE for them, requiring the driver software to fill in the gaps.

[0077] The specific implementation steps are as follows: Step 1: Identify the QP corresponding to the error CQE In the poll_hardware_cq function, after the driver reads a CQE from the first buffer, it parses its qp_num field to obtain the QP identifier of the error.

[0078] Step 2: Check if the QP is already in the pending list. The driver iterates through the current CQ's err_state_qp_list. If a node with the same qp_id already exists, it means that it has already been triggered (e.g., it has already been added through the first trigger condition). In this case, there is no need to add it again. The driver can simply return the error CQE to the upper layer normally.

[0079] Step 3: Calculate the number of unconsumed WQEs (same as step 2 in the first trigger condition). Step 4: Create the node and add it to the linked list (same as steps 3 and 4 in the first triggering condition). Step 5: Return the error CQE normally. The driver still needs to return the currently read hardware error CQE to the upper-layer application (i.e., fill it into the wc array) because the CQE carries error information that the upper-layer application may need it for logging or statistics.

[0080] For example: Suppose a GPFS application uses a certain QP for data transmission, and a hardware malfunction occurs (such as a remote node disconnection). The hardware enters an error state in the QP's internal state machine, but the application does not actively call modify_qp to ERR. When polling the CQ, the driver reads a CQE with a state of IBV_WC_GENERAL_ERR from the first buffer, with qp_num=200. The driver checks the linked list and finds that QP200 is not in it. It is calculated that there are still 8 unconsumed WQEs in the SQ of QP200 and 3 unconsumed WQEs in the RQ, with pending=11. The driver creates a node {qp_id=200, pending_flush_cnt=11} and adds it to the pending linked list of the CQ. Then, the hardware error CQE (with a state of GENERAL_ERR) is filled into the wc[0] provided by the application and returned.

[0081] When GPFS calls ibv_poll_cq again, the driver will first return any remaining hardware CQEs (if any), then retrieve the QP200 node from the linked list and begin constructing 11 flush err CQEs to return to the application. Upon receiving these flush err CQEs, the application can correctly release queue resources.

[0082] In this embodiment, many applications (such as GPFS and certain versions of Lustre) do not actively modify the QP state after detecting an error, but instead rely on polling for the error CQE and handling it automatically. This triggering condition ensures that even if the application does not actively change the state, the driver can still identify the QP that needs to be flushed with a CQE. By checking the linked list for deduplication, the same QP is prevented from being added multiple times, saving memory and CPU. The original error CQE generated by the hardware is still reported to the application, without affecting the application's original error handling path. Without changing the application's polling semantics, the missing flush CQE is filled in, allowing the resource reclamation logic to be triggered.

[0083] Furthermore, obtaining the sum of the number of work queue elements in the send queue and receive queue of the queue pair that have not yet been consumed by the hardware includes: For the send queue, read the producer pointer and consumer pointer in the send queue context of the queue pair. The producer pointer indicates the last work queue element that the driver has sent to the hardware, and the consumer pointer indicates the last work queue element that the hardware has taken away. The difference between the producer pointer and the consumer pointer is taken as the number of work queue elements that have not yet been consumed in the send queue, NUM1. For the receive queue, read the producer pointer and consumer pointer in the receive queue context of the queue pair, and use the difference between the producer pointer and the consumer pointer as the number of work queue elements that have not yet been consumed in the receive queue, NUM2. Add NUM1 and NUM2 to get the total number of refresh error completion queue elements that need to be constructed for this queue pair.

[0084] In this embodiment, in the RDMA driver, the send queue (SQ) and receive queue (RQ) of each queue pair (QP) are managed by two pointers: the producer pointer indicates the last work queue element (WQE) that the driver has sent to the hardware, and the consumer pointer indicates the last WQE that the hardware has taken and processed. The difference between the two is the number of WQEs that have not yet been consumed by the hardware. By calculating the number of unconsumed SQs and RQs (NUM1 and NUM2) respectively and then adding them together, the total number of flush err CQEs that the QP needs to construct can be obtained. This value determines the number of flush CQEs that the driver needs to simulate and generate in software subsequently.

[0085] Furthermore, the method also includes: When a completion queue is created, the driver records whether the completion queue has registered a completion handler function and whether the upper-layer application is waiting for completion queue elements through an event mechanism.

[0086] In the RDMA driver, when the upper-layer application creates a completion queue (CQ), it calls the creation interface (such as kernel-mode ibv_create_cq or user-mode ibv_create_cq), passing in a completion handler function pointer `comp_handler` and an event context parameter. The driver needs to additionally record the following information in the internal structure of the CQ creation (e.g., `struct xsc_cq`): has_comp_handler: A boolean value indicating whether the upper-layer application has provided a valid completion handler function.

[0087] `uses_event_mode`: A boolean value indicating whether the upper-layer application waits for the CQE via an event mechanism (such as `ibv_req_notify_cq`). This flag is usually set by the driver when the application calls `ibv_req_notify_cq`, or it can be inferred from the parameters when the CQ is created (in some interfaces, if `comp_handler` is not null and the application will subsequently wait for events, it is considered to be in event mode).

[0088] For user-space drivers, similarly, the user-registered comp_handler callback pointer (user-space address) and a flag indicating whether the application has enabled events via ibv_req_notify_cq are stored in struct xsc_cq.

[0089] By recording whether a completion handler function is registered in the completion queue and whether event notifications are enabled, the driver can accurately determine the upper-layer application's operating mode (polling mode or event notification mode). This allows it to adopt different wake-up strategies based on the operating mode when a software-constructed refresh error (CQE) needs to be reported later (e.g., no active wake-up is needed in polling mode, while a callback needs to be triggered in event mode). This avoids the upper-layer application never receiving CQE notifications due to mode misjudgment. Furthermore, this recording has extremely low overhead, only requiring setting once during CQ creation and notification request, and does not affect hot path performance.

[0090] After the first or second triggering condition occurs and the node is added to the pending queue of the completion queue, if the completion queue is in event notification mode, the driver actively calls the completion processing function of the completion queue to wake up the upper-layer application blocked on the completion queue event, prompting the upper-layer application to call the polling interface, thereby triggering step S200, so that the upper-layer application obtains the refresh error completion queue element constructed by the software.

[0091] This step occurs after the first or second trigger condition is executed, i.e., after the driver has added the node of an err_state_qp_list to the pending queue of a certain CQ. At this time, the driver needs to check whether the CQ is in event notification mode (i.e., event_notify_enabled == true and comp_handler is not empty). If so, the driver actively calls the completion handling function of the CQ to wake up any upper-layer applications that may be blocked on the CQ event.

[0092] For example, GPFS uses a user-space driver and an event notification model. When the driver reads a general error CQE (second trigger condition) during polling and adds QP200 to the linked list, the driver calls ibv_query_qp, passing in a private mask. Upon receiving this, the kernel driver executes comp_handler, waking up the GPFS process, which then calls ibv_poll_cq. At this point, the driver executes S200 and subsequent S300 within poll_cq, constructing and returning a flush err CQE to GPFS, enabling it to correctly release resources.

[0093] By actively invoking the completion handling function, this mechanism solves the problem of upper-layer applications potentially being blocked for extended periods and unable to promptly receive information about the readiness of software-constructed flush CQEs in event notification mode. It eliminates the need for applications to manually call time-consuming operations such as `ib_drain_qp`. For kernel-mode applications (such as NVMF and NFS), direct callbacks are efficient and real-time. For user-mode applications (such as GPFS), cross-layer notifications via private attribute masks enable collaboration between user-mode and kernel drivers, waking up the application without modifying its code. This mechanism covers the event models of all mainstream RDMA applications, ensuring that regardless of the event waiting method used by the application, it can immediately obtain a flush CQE when needed, avoiding resource leaks and process suspension.

[0094] Furthermore, the driver includes user-mode drivers and kernel-mode drivers; When the driver is a user-space driver and the completion queue is in event notification mode, the user-space driver cannot directly call the completion handling function of the upper-layer application; the user-space driver calls the query queue pair interface and passes a private attribute mask to the query queue pair interface, which indicates that the completion queue completion event needs to be triggered.

[0095] After receiving the query request, the kernel-mode driver finds the corresponding completion queue based on the private attribute mask and executes the completion processing function of the completion queue, thereby indirectly waking up the upper-layer application in user mode.

[0096] In this embodiment, in an RDMA system, when a user-mode driver is used and the completion queue (CQ) is in event notification mode, the upper-layer application obtains completion notification by registering a completion handler function (comp_handler) and waiting for CQ events. However, although the user-mode driver can access the hardware, it cannot directly call the completion handler function registered by the upper-layer application (because this function resides in user space, and while the user-mode driver also resides in user space, the key difference is that event notification mode typically relies on the kernel's eventfd or signal mechanism to wake up blocked application processes, and the user-mode driver does not have permission to directly trigger such cross-process wake-up operations). Therefore, when the user-mode driver needs to actively report a software-constructed refresh error CQE (e.g., after adding the error QP node to the pending list), it must indirectly wake up the application using a kernel-mode driver.

[0097] The specific implementation is as follows: The user-space driver calls the standard query queue interface (ibv_query_qp), passing in a custom private attribute mask. This mask is specifically used to send a request to the kernel-space driver that a CQ completion event needs to be triggered. After receiving the query request, the kernel-space driver parses the private mask, finds the corresponding completion queue object based on the information carried in the mask, and then directly executes the completion handler function (comp_handler) of that completion queue. This completion handler function wakes up the upper-layer application thread that is blocked waiting for the CQ event (e.g., by writing data to eventfd or sending a signal). After the application is awakened, it will call the polling interface (ibv_poll_cq) again. At this time, the driver can construct and return the software-generated refresh error completion queue element from the pending list, thus completing the entire remediation process.

[0098] This cross-layer collaboration mechanism allows user-mode drivers to proactively trigger application re-polling in event notification mode without modifying application code or requiring hardware support, ensuring that the software-constructed CQE can be obtained by upper-layer applications in a timely manner.

[0099] Furthermore, in step S300, when traversing the linked list of pending queue pairs and constructing software completion queue elements, if the remaining number of elements to be constructed for the current node is greater than the difference between the number of completed queue elements requested and the returned count, then only the number of software completion queue elements equal to the difference is constructed and filled. The remaining number of elements to be constructed for the node is then subtracted from the difference and retained in the linked list. Traversal is then stopped immediately, and the node and subsequent nodes in the linked list are processed again when the next polling interface call is made.

[0100] In step S300, the driver needs to construct a corresponding number of refresh error completion queue elements for each error queue pair from the pending queue pair linked list. However, the number of requests (num_entries) passed by the upper-layer application during each polling interface call is limited. Assuming the driver has already returned polled hardware CQEs from the first buffer, this poll can return at most num_entries-polled software-constructed CQEs.

[0101] When the driver traverses the linked list and processes a node, if the remaining number of nodes to be constructed (pending_flush_cnt) is greater than the number of slots that can be returned (num_entries - polled), it means that this polling cannot return all the flush errors CQE required for this node at once. In this case, the driver takes the following measures: The software CQE is constructed and filled only with the number of differences: that is, only num_entries-polled refresh error CQEs are constructed, stored in a second buffer, and returned to the upper-layer application. This fills all slots for the current user request, avoiding wasted polling opportunities.

[0102] Update the remaining number of nodes to be constructed: Subtract the number of constructed nodes (i.e., subtract num_entries-polled) from the node's pending_flush_cnt to obtain the remaining number of nodes that have not yet been constructed, and keep it in the node.

[0103] Stop traversing immediately: stop processing the current node in the linked list (because there are no remaining slots), and stop traversing subsequent nodes. This ensures that the total number of CQEs returned in this poll does not exceed the user's request.

[0104] Waiting for the next polling: When the upper-layer application calls the polling interface again, the driver will reread the hardware CQE from the first buffer (which may be empty), and then enter step S300 again. At this time, the driver will continue to construct the remaining refresh error CQE from the nodes that were not processed last time, and process the subsequent nodes in the linked list.

[0105] For example: Suppose a user requests 16 CQEs, and the driver has already returned 3 hardware CQEs (polled=3), leaving 13 slots remaining. The current node A's pending_flush_cnt is 20, which is greater than 13. Therefore, the driver only constructs 13 flush error CQEs to fill this request, polled becomes 16, and the function returns. Node A's pending_flush_cnt is updated to 20-13=7 and retained in the linked list. The next time the application calls ibv_poll_cq, the driver will continue constructing from the remaining 7 slots on node A.

[0106] This batch processing mechanism ensures that the driver will not occupy the CPU for an extended period due to a single node requiring a large number of CQEs, nor will it block user requests. It strictly adheres to the maximum number of requests allowed per poll by the user, avoiding returning more CQEs than the user expects. Unfinished construction tasks are retained in the nodes, and the driver maintains the state, preventing data loss. When multiple nodes have pending CQEs, if a single poll can only partially complete the task, the driver processes them sequentially according to the linked list, ensuring that each queue pair gradually obtains the necessary refresh error CQEs.

[0107] S400, the final returned count is returned to the upper layer application as the return value of the polling interface.

[0108] The accumulated polled value is returned to the upper-layer application as the return value of the polling interface. The upper-layer application uses the return value to determine the total number of CQEs obtained in this poll (including hardware CQEs and software-constructed refresh error CQEs), and then processes each CQE in sequence.

[0109] For example: return value 16. The application iterates through wc[0] to wc

[15] . The first 3 are hardware CQEs, and the last 13 are CQEs in the IBV_WC_WR_FLUSH_ERR state. The application's original error handling logic (such as the handling of IBV_WC_WR_FLUSH_ERR in SPDK's poll_cq) will normally decrement current_recv_depth, eventually satisfying the condition for destroying the QP.

[0110] This step ensures that the return behavior of the polling interface is completely consistent with the behavior when the hardware normally generates a refresh error (CQE). The upper-layer application can continue to work normally without any modifications, achieving transparent compatibility of the driver layer with hardware function changes.

[0111] Furthermore, the method also includes the following steps: In S500, when the upper-layer application calls the destroy queue pair interface, the driver checks whether the queue pair exists in any pending queue pair list of completed queues. If it exists, the node corresponding to the queue pair is deleted from all the linked lists and the node memory is released.

[0112] When an upper-layer application calls a Queue Pair (QP) interface (such as kernel-mode ibv_destroy_qp or user-mode ibv_destroy_qp), the driver must first check whether the QP has been recorded in the pending queue pair list (err_state_qp_list) of any completion queue (CQ) before releasing the resources associated with the QP. Since a QP can be associated with send CQs and receive CQs (which may be the same or different), it is necessary to traverse all associated CQs.

[0113] Retrieve the send CQ (qp->send_cq) and receive CQ (qp->recv_cq) from the context of the QP. If they are the same, only one CQ needs to be processed; otherwise, both CQs need to be processed separately.

[0114] For each associated CQ, perform node lookup and deletion: Traverse the err_state_qp_list linked list of the CQ. Compare the qp_id of each node with the number of the QP to be destroyed. If a matching node is found, remove the node from the linked list using a linked list deletion operation (such as list_del). Release the memory occupied by the node (such as kfree or free).

[0115] After the node cleanup is complete, the driver continues to execute the original QP destruction process (such as releasing the QP context, doorbell memory, and unmapping DMA).

[0116] This step ensures that when a queue pair is destroyed, its corresponding pending linked list nodes are synchronously cleared, avoiding dangling pointers and memory leaks. Furthermore, because the driver cleans up the nodes before destruction, any subsequent polling operation on that queue will not attempt to construct a refresh error CQE for the destroyed queue, guaranteeing system stability and resource integrity.

[0117] S510: When the upper-layer application calls the destroy completion queue interface, the driver traverses the pending queue list of the completion queue, releases all nodes in the list, clears the list, and releases the memory of the first and second buffers.

[0118] When an upper-layer application calls the Destroy Complete Queue (CQ) interface (such as ibv_destroy_cq), the driver needs to release all resources associated with that CQ, including all nodes in the queue list to be processed, the memory of the first buffer and the second buffer.

[0119] This step ensures that all associated resources (including dynamically allocated linked list nodes and two buffers) are completely released when the queue is destroyed, preventing memory leaks. Especially when the system runs for a long time and a large number of QPs and CQs are created and destroyed, this step ensures that the driver does not accumulate invalid pending nodes. Simultaneously, because the linked list is cleared before destroying the CQ, any subsequent possible dangling pointer accesses are avoided. Furthermore, the proper release of the first buffer (DMA memory) also avoids IOMMU mapping remnants, ensuring hardware resource reclamation.

[0120] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0121] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0122] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0123] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0124] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0125] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0126] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0127] The electronic device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments in this application.

[0128] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and a bus connecting different system components (including memory and processor).

[0129] The memory stores program code that can be executed by the processor, causing the processor to perform the steps in the various embodiments described in this specification.

[0130] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0131] The memory may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0132] A bus can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus structures.

[0133] Electronic devices can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable user interaction with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be achieved through input / output (I / O) interfaces. Furthermore, electronic devices can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapters. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0135] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0136] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A method of constructing an RDMA completion queue element based on software, the method comprising: The method includes the following steps: ​ S100, allocate a first buffer and a second buffer to each completion queue; wherein, the first buffer is used to store the completion queue elements written by RDMA hardware, and the second buffer is used to store the completion queue elements constructed by the driver; the driver maintains a linked list of queue pairs to be processed, each node in the linked list of queue pairs to be processed corresponds to a queue pair that has entered an error state and still needs to report refresh error completion queue elements; each node records the identifier of the queue pair and the number of refresh error completion queue elements to be constructed in the queue pair. In response to the upper-layer application calling the polling interface and passing in the number of elements in the completion queue, the following steps are executed: S200: Read hardware completion queue elements from the first buffer and return them to the upper layer application until there are no unread hardware completion queue elements in the first buffer or the returned count reaches the requested number of completion queue elements; proceed to S300. S300, if the count has not reached the number of completed queue elements requested, then obtain the queue pair node to be processed from the queue pair list to be processed, construct a corresponding number of software completion queue elements with the status of work request refresh error for each queue pair to be processed, store the constructed software completion queue elements into the second buffer, read the software completion queue elements from the second buffer and return them to the upper layer application, and update the remaining number of nodes to be constructed. S400, the final returned count is returned to the upper layer application as the return value of the polling interface.

2. The method of claim 1, wherein the software-based constructing an RDMA completion queue element comprises: Step S300 includes the following steps: S310, traverse each node in the pending queue list in sequence, construct a software completion queue element with a status of work request refresh error each time according to the remaining number of elements to be constructed recorded in the node, and store the software completion queue element into the second buffer. S320: For each software-completed queue element stored, decrement the remaining number of elements to be constructed for that node by one, and increment the returned count by one; S330, repeat S310 and S320 until the remaining number of nodes to be constructed is zero, or the count has reached the number of completed queue elements requested. S340, if the remaining number of nodes to be constructed becomes zero, then the node is removed from the linked list and released.

3. The method of claim 2, wherein the software-based constructing of the RDMA completion queue element comprises: The generation trigger conditions for nodes in the linked list of the queue to be processed include a first trigger condition and a second trigger condition: The first triggering condition is: when the driver detects that the upper layer application has modified the state of a certain queue pair to an error state by modifying the queue pair state interface, it obtains the sum of the number of work queue elements in the sending queue and receiving queue of the queue pair that have not yet been consumed by the hardware, and uses it as the number of refresh error completion queue elements to be constructed. It creates a node containing the queue pair identifier and the number of elements, and adds it to the list of pending queue pairs of all completion queues associated with the queue pair. The second triggering condition is as follows: When the driver reads a completion queue element with a status of general error and not a work request refresh error during the process of reading hardware completion queue elements from the first buffer, it determines the queue pair to which the completion queue element belongs, obtains the sum of the number of work queue elements that have not yet been consumed by the hardware in the sending queue and receiving queue of the queue pair, creates a node and adds it to the pending queue pair linked list of each completion queue associated with the queue pair.

4. The method of claim 3, wherein the software-based constructing of the RDMA completion queue element comprises: The method further includes: When a completion queue is created, the driver records whether the completion queue has registered a completion handler function and whether the upper-layer application is waiting for completion queue elements through an event mechanism. After the first or second triggering condition occurs and the node is added to the pending queue of the completion queue, if the completion queue is in event notification mode, the driver actively calls the completion processing function of the completion queue to wake up the upper-layer application blocked on the completion queue event, prompting the upper-layer application to call the polling interface, thereby triggering step S200, so that the upper-layer application obtains the refresh error completion queue element constructed by the software.

5. The method of claim 4, wherein the software-based constructing of the RDMA completion queue element comprises: The drivers include user-mode drivers and kernel-mode drivers; When the driver is a user-space driver and the completion queue is in event notification mode, the user-space driver cannot directly call the completion handling function of the upper-layer application; the user-space driver calls the query queue pair interface and passes a private attribute mask to the query queue pair interface, which indicates that the completion queue completion event needs to be triggered. After receiving the query request, the kernel-mode driver finds the corresponding completion queue based on the private attribute mask and executes the completion processing function of the completion queue, thereby indirectly waking up the upper-layer application in user mode.

6. The method of claim 2, wherein the software constructs an RDMA completion queue element, and In step S300, when traversing the queue list to be processed and constructing software completion queue elements, if the remaining number of elements to be constructed for the current node is greater than the difference between the number of completed queue elements requested and the returned count, then only the number of software completion queue elements equal to the difference is constructed and filled. The remaining number of elements to be constructed for the node is subtracted from the difference and the node is kept in the linked list. Traversal is stopped immediately, and the node and subsequent nodes in the linked list are processed again when the next polling interface call is made.

7. The method of claim 3, wherein the software-based constructing an RDMA completion queue element further comprises: The step of obtaining the sum of the number of work queue elements in the send queue and receive queue of the queue pair that have not yet been consumed by the hardware includes: For the send queue, read the producer pointer and consumer pointer in the send queue context of the queue pair. The producer pointer indicates the last work queue element that the driver has sent to the hardware, and the consumer pointer indicates the last work queue element that the hardware has taken away. The difference between the producer pointer and the consumer pointer is taken as the number of work queue elements that have not yet been consumed in the send queue, NUM1. For the receive queue, read the producer pointer and consumer pointer in the receive queue context of the queue pair, and use the difference between the producer pointer and the consumer pointer as the number of work queue elements that have not yet been consumed in the receive queue, NUM2. Add NUM1 and NUM2 to get the total number of refresh error completion queue elements that need to be constructed for this queue pair.

8. The method of claim 1, wherein the software-based constructing of the RDMA completion queue element comprises: The method further includes the following steps: S500, when the upper application calls the destroy queue pair interface, the driver checks whether the queue pair exists in any pending queue pair list of a completion queue, if it exists, the node corresponding to the queue pair is deleted from all the lists where it exists and the node memory is released; S510, when the upper application calls the destroy completion queue interface, the driver traverses the pending queue pair list of the completion queue, releases all the nodes in the list, clears the list, and releases the memory of the first buffer and the second buffer. 9.A non-transitory computer-readable storage medium having stored therein at least one instruction or at least one piece of program, characterized in that, The at least one instruction or the at least one program is loaded and executed by the processor to implement the method for constructing an RDMA completion queue element based on software as claimed in any one of claims 1-8.

10. An electronic device, comprising: The non-transitory computer readable storage medium of claim 9 is included in a processor. The non-transitory computer readable storage medium of claim 9 is included in a processor.

Images