A lossless and efficient data processing method based on RMDA and network interface card

Abstract

The application discloses a lossless and efficient data processing method and network interface card based on RMDA, and belongs to the field of data communication of a data center. When a large-scale QP is concurrent in the RMDA, the efficient scheduling processing problem of different QP requests is solved, the lossless request signal, the out-of-order QP request and the waste-free scheduling period are ensured, the scheduling fairness problem of different QPs is effectively solved in combination with congestion control, the blocking of other QP processing by a large message and the excessive number of requests needing to be processed in a single scheduling are avoided, and the head blocking problem is solved.

Description

Technical Field

[0001] This invention belongs to the field of data communication in data centers, and particularly relates to a lossless and efficient data processing method and network interface card based on RMDA. Background Technology

[0002] RDMA technology is being deployed more and more widely in data centers to achieve lossless networks with "high bandwidth, low latency, high throughput, and zero packet loss". However, when handling large-scale concurrent I / O requests, RDMA-based network interface cards (RNICs) suffer from coarse-grained I / O request scheduling, which can lead to severe head-blocking problems, with requests from different QPs blocking each other. Large message I / O requests require significant resource consumption, impacting the processing of smaller messages and failing to guarantee fairness among multiple QPs. Furthermore, under conditions of large-scale concurrent I / O requests, untimely request processing can easily result in I / O loss, severely impacting the performance of upper-layer applications. Summary of the Invention

[0003] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a network interface card and data processing method based on RDMA technology. When there is a large number of concurrent QPs in the host, it can efficiently solve the scheduling and processing problem of requests from different QPs, ensuring that request signals are not lost, QP requests are not out of order, and scheduling cycles are not wasted. At the same time, combined with congestion control, it effectively ensures the scheduling fairness of different QPs and solves the head-blocking problem.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0005] A network interface card based on RDMA includes a PCIe BAR register processing module, a first-stage scheduling module, a QPC status table, a DB verification module, a QPN-QID mapping module, a second-stage scheduling module, a scheduling policy configuration module, a congestion control module, and a DMA engine.

[0006] The PCIe BAR register processing module is responsible for parsing and processing the doorbell signal DB in the software and hardware interaction.

[0007] The first-stage scheduling module is responsible for scheduling the doorbell signal database based on PageID and priority in the first stage, ensuring that the doorbell signal database is not lost.

[0008] QPC status table, which caches QPC status information;

[0009] The DB verification module is used to verify the validity of the DB and whether the status information of the corresponding QP is correct.

[0010] The QPN-QID mapping module is used to assign enqueue IDs to DBs to enter the second-stage scheduling module;

[0011] The second-stage scheduling module is responsible for scheduling the DB based on hostID and QPN in the second stage, ensuring that different QPs of different hosts can be scheduled fairly.

[0012] The scheduling strategy configuration module is used to configure the strategies such as the number of scheduling levels, the size of each scheduler, and the scheduling algorithm in the second-stage scheduling module.

[0013] The congestion control module is used to allocate Credit to each QP and control the amount of messages that each QP can send in one scheduling cycle;

[0014] The WQE processing module is used for WQE prefetching and processing. If WQE cannot be processed completely, it returns the breakpoint information of DB processing to the second-stage scheduling module.

[0015] The DMA engine is used for data transfer between the RNIC and the Host.

[0016] Where PageID is the page ID, representing the ID of the BAR space address corresponding to the software's DB; QPC represents the QP context information, used to cache QP address information; QP represents the RDMA connection queue; hostID represents the host's serial number ID; QPN represents the QP's serial number ID; Credit is the credit; WQE represents an RDMA request; RNIC represents the RDMA network card;

[0017] A lossless and efficient data stream processing method based on RDMA network interface card RDMA specifically includes the following steps:

[0018] Step 1: When a new WQE is generated in the SQ of the target host, the host will generate a doorbell signal DB and send it to the RNIC; that is, the DB information is written to the Doorbell space allocated by the RNIC for this QP through the PCIe interface.

[0019] Where SQ represents the sending queue;

[0020] Step 2: RNIC resolves the PageID of the address in the Doorbell register, as well as the Doorbell's QPN and priority CoS information, and adds the Doorbell to the first-stage scheduler. The first-stage scheduler adopts a hierarchical scheduling structure, supporting M groups. Each group can be further divided into 4 priority queues. The SP+WRR scheduling algorithm is used in each group, and the RR scheduling algorithm is used between different groups. SP is the strict priority scheduling algorithm; WRR is the weighted round-robin scheduling algorithm.

[0021] Step 3: The first-level scheduler of the first-stage scheduling module selects the queue to be scheduled using the configured scheduling algorithm and adds the DB to be processed to the scheduler's output queue.

[0022] Step 4: The second-level scheduler of the first-stage scheduling module uses RR scheduling to write the DB output by the first-level scheduler into the output queue;

[0023] Step 5: The DB verification module retrieves the DB from the head of the second-level output queue of the first-level scheduler and verifies its validity. It checks whether the DB's PageID matches the PageID in the QPC bound to the DB's QPN and whether the QP's status is normal. If they match, the DB is input into the QPN-QID mapping module; otherwise, the DB is discarded and an error is returned to the target host. Here, QPN represents the QP's sequence number ID; QID represents the local ID assigned to the QP in the RNIC, which is unique in the RDMA system. The QPN-QID mapping module is used for mapping and looking up the mapping between the QPN and the local QID, using the QID as an index for the local QP Context.

[0024] Step 6: After the DB inputs to the QPN-QID mapping module, it looks up the corresponding GroupID based on the HostID in the DB, and maps the QPN to the input queue of the corresponding GroupID to ensure that QP and DB with the same HostID are placed in the same scheduling group.

[0025] GroupID represents the ID of the scheduling group; different HostIDs can be placed in different scheduling groups.

[0026] HostID represents the host ID. In a virtualization-enabled scenario, each VM corresponds to a unique HostID.

[0027] Step 7: In the second-level scheduling module, each level scheduler outputs the DB to the final output queue of the module.

[0028] Step 8: The WQE processing module retrieves the DB from the head of the output queue and reads the QPC status required for processing WQE according to the QPN. The QPC contains Max_Burst_Size and Max_Batch_WQE_count, which represent the maximum number of messages that a single WQE can send and the maximum number of WQEs that can be processed at one time, respectively. Max_Burst_Size represents the maximum number of bytes allowed to be sent in a single scheduling cycle. Max_Batch_WQE_count represents the maximum number of WQEs allowed to be acquired in a single scheduling cycle, which is also the maximum number of messages that can be processed in a single scheduling cycle.

[0029] Step 9: The WQE processing module requests Credit from the congestion control module; where Credit is the size of the sendable message allocated to each QP by the congestion control algorithm.

[0030] Step 10: The WQE processing module obtains no more than N WQEs from SQ via the DMA engine based on Max_Batch_WQE_count and the current cacheable number of WQEs, WQE_Available_Count, where N = min(WQE_Available_Count, Max_Batch_WQE_count); WQE_Available_Count represents the maximum number of available WQEs; N represents the number of WQEs that can be processed in a single scheduling cycle, where N is the minimum value of WQE_Available_Count and Max_Batch_WQE_count.

[0031] Step 11: The WQE processing module processes the cached WQEs one by one. For each WQE processed, the consumption pointer of the WQE in the QPC is updated, and the Credit value is updated. The Credit value is the current value minus the Credit value consumed by the WQE. If, during the processing of a WQE, the remaining Credit is insufficient to process a complete WQE, the interrupted status of the WQE processing will be returned after the remaining Credit is consumed. This information is cached in the Interupted DB State table in the second scheduling module, and the bitmap corresponding to this queue is set to 1. The Interupted DB State includes HostID, QPN, Produce_Index, Target_WQE_Index, and Walk_Offset. Produce_Index represents the current WQE_Index being processed, Walk_Offset represents the pointer to the data already sent for the currently unprocessed WQE, and Target_WQE_Index represents the WQE position that the current DB needs to process. Credit is a token value representing the number of bytes sent by the current QP. State is the interrupt DB information table, used to cache interrupt information of DB scheduling; bitmap is a bitmap, each bit in the bitmap corresponds to QPN. If the QPN information is valid, the corresponding bit is set to 1, and if it is invalid, it is set to 0.

[0032] Step 12: When the second-stage scheduling module schedules a queue in the next cycle, it first uses Bitmap to determine whether there is an interrupted DB status in the queue, that is, whether the bitmap corresponding to the queue is 1; if bitmap = 0, a new DB is read from the scheduling input queue for processing; if bitmap = 1, the DB interrupt status is read first, and the DB interrupt status information is combined into a new DB and sent to the WQE processing module.

[0033] Step 13: Based on the information in the DB, the WQE processing module continues to process the MR data starting from the Walk_Offset of the WQE pointed to by Produce_Index. It determines whether the WQE can be completely processed based on the Credit, and repeats steps 11-13 until the DB can be completely sent out without interruption, and then sets the bitmap to 0. Here, Produce_Index represents the consumption pointer, indicating the current position of SQ processing; Walk_Offset represents the virtual address of the interrupt position of a single WQE processing.

[0034] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, the basic steps of Doorbell adding the first-stage scheduler include:

[0035] Step 2.1: The first-stage scheduling module obtains the GroupID of the target queue based on the Hash(PageID) and selects the Doorbell to add to the corresponding priority queue based on the CoS in the DB. If the queue is not full, the Doorbell is directly written to the tail of the queue. If the queue contains the Doorbell of the QPN, they can be merged into 1 DB. If the queue is full, proceed to step 2.2.

[0036] CoS stands for Channel of Service, which represents the priority channel and typically supports eight priority levels.

[0037] Step 2.2: The first-stage scheduling module records Doorbell information in the Overflow Buffer; the Overflow Buffer is a shared cache for all first-stage scheduling queues; in the Overflow Buffer, an entry is allocated for each priority of each GroupID, and the DB information is cached sequentially in a linked list; if a Doorbell is added to the Overflow Buffer, and the information of the QPN to which the Doorbell belongs already exists in the Overflow Buffer, then the new Doorbell replaces the old Doorbell, that is, in the buffer, only the latest Doorbell is cached for the same QP; in the Overflow Buffer, a bitmap is used to record whether there is a Doorbell for each priority of GroupID. If there is Doorbell information, the bit is set to 1, otherwise it is set to 0.

[0038] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, only the QPN of the DB is cached in the Overflow Buffer. When the QPN is scheduled, the latest WQE_Index is obtained by reading the DBRecord cached on the target host, and a new DB is generated and added to the first-level scheduling output queue of the scheduler to execute the next level of scheduling. The DB Record is the content cached on the host side, which is used to record the SQ's producer index and consumer index.

[0039] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, the scheduling method of the first-layer arbiter or scheduler for different CoS queues within the same Group is as follows:

[0040] Step 3.1: If the current polling reaches the queue with GroupID=n and CoS=m, first determine if the queue is empty; if it is not empty, take the DB at the head of the queue and add it to the scheduler's output queue; if the queue is empty, proceed to step 3.2.

[0041] Step 3.2: Read the bitmap of the queue from the Overflow Buffer and determine whether the DB information of the queue is cached. If it exists, retrieve the DB information from the head of the linked list, add it to the corresponding scheduling output queue, and delete the DB information from the linked list. The current scheduling ends, and the system waits for the next scheduling cycle. If it does not exist, skip the queue and poll the next queue. The current scheduling cycle ends. The Overflow Buffer is an overflow buffer. When the scheduling queue is full, the DB information is cached in the overflow buffer. This buffer is a shared buffer for all scheduling queues.

[0042] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, HostID can be represented by PF+VF ID; where PF ID represents physical function ID, which represents the ID of the physical channel in PCIe; VFID represents virtual function ID, which represents the ID of the virtual channel in PCIe, and one virtual channel represents one virtual machine.

[0043] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, the second-stage scheduling module also adopts a hierarchical multi-level scheduling method. Its scheduling level, the group size of each level, and the scheduling algorithm of each level scheduler can be configured through the scheduling strategy module.

[0044] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, the scheduling algorithm includes, but is not limited to, SP, RR, WRR, and DWRR scheduling algorithms;

[0045] SP stands for strict priority scheduling algorithm;

[0046] RR stands for Round Robin algorithm;

[0047] WRR is a weighted round-robin scheduling algorithm;

[0048] DWRR is a differential weighted round-robin algorithm.

[0049] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, the first-level input queue of the second-stage scheduling module contains two cache lines (Entries) for caching databases (DBs), wherein the Entry at the head of the queue represents the DB being scheduled, and the second Entry represents the next DB to be scheduled.

[0050] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, the DB is added to the input queue in the following manner: if the queue is empty, the new DB is added to the head of the queue; if the queue has only 1 DB, the new DB is added to the tail of the queue; if there is also a DB at the tail of the queue, the new DB replaces the old DB and is cached in the queue.

[0051] As a further preferred embodiment of the lossless and efficient data stream processing method based on RDMA of the present invention, in step 11, the Interupted DB State includes HostID, QPN, Produce_Index, Target_WQE_Index, and Walk_Offset. Produce_Index represents the WQE_Index currently being processed, Walk_offset represents the pointer to the data that has been sent for the currently unprocessed WQE, and Target_WQE_Index represents the WQE position that the current DB needs to process.

[0052] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0053] 1. Multi-stage scheduling: Design scheduling queues based on different scheduling granularities to avoid placing all databases together for serial execution, which would cause a serious head-blocking problem;

[0054] 2. The first-stage scheduler module uses a general DB allocation method based on process Page ID. Unlike the traditional method of binding DB type to address, it decouples DB type from address, thereby improving the utilization of DB space.

[0055] 3. In the first-stage scheduler module, a DB queue management and scheduling method is designed based on comprehensive factors such as Host, process, and priority. This method can achieve scheduling at different granularities. While ensuring that the same QP is not out of order, high-priority requests can be effectively scheduled, effectively solving the head-blocking problem under large-scale QP concurrency conditions.

[0056] 4. In the first-stage scheduler, the lossless DB processing method using the shared Overflow Buffer ensures that DB information is not lost under high concurrency conditions, thus achieving lossless request processing; at the same time, only DB status information is cached, and lost DB information is obtained by reading the DB Record, effectively saving cache.

[0057] 5. In the second-stage scheduling module, the maximum amount of data generated by WQEs that can be processed in each scheduling cycle is limited by constraints such as Credit + number of WQEs + message size. This ensures the fairness of multi-QP scheduling, prevents large messages from occupying resources and blocking other QPs, and alleviates the head-blocking problem.

[0058] 6. In the second-stage scheduling module, by adding an Interupted DB state cache to each scheduling queue to handle interrupted status, WQE breakpoint resume capability is supported during scheduling, realizing fine-grained QP scheduling and ensuring efficient and lossless request processing. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of a lossless and efficient data processing method for bare metal scenarios provided in Embodiment 1 of the present invention;

[0061] Figure 2 This is a schematic diagram of a lossless and efficient data processing method for virtual machine scenarios provided in Embodiment 2 of the present invention. Detailed Implementation

[0062] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0064] The method provided in this disclosure is applied to a network interface card (RNIC) in a data storage device, which is based on remote direct memory access (RDMA).

[0065] The target host supports the RDMA protocol, using queue pairs (QPs) for data transmission and reception. Each QP contains a send queue (SQ) and a receive queue (RQ). The SQ is responsible for sending messages, and the RQ is responsible for receiving messages. Each QP's SQ and RQ can be associated with a completion queue (CQ). Each QP has a locally unique QP number (QPN). QP status information, including QPN, CQN, QP address, and QP length, is stored in the QP Context (QPC). The RNIC maintains a QPC table, caching registered QPC information.

[0066] The target host may include multiple transmit queues, such as SQ1, SQ2, and SQ3. When the target host issues a Work Request (WR), it writes a Work Queue Element (WQE) into the available space in the SQ and sends a doorbell (DB) signal to the RNIC bound to that SQ via the driver. When the RNIC receives the DB, it reads the QPN, WQE_index, and other information carried in the DB through DMA, reads the QPC Table, calculates the address of the WQE in the SQ, and then initiates a DMA request to read the contents of the WQE for further processing.

[0067] The RNIC and the target host are connected via a communication bus (PCIe), and database entries (DBs) are written to the PCIe BAR registers. In the RNIC, the PCIe BAR space is managed in units of pages. Different processes are allocated different BAR space registers, and the BAR spaces are isolated from each other. Each page contains N general-purpose DBs. If the target host sends a doorbell to the RNIC, the doorbell is written to the available doorbell in the BAR space bound to that process via the PCIe interface. The doorbell has a 2-bit DB_type field that identifies the doorbell type. DB_type = 00 is a reserved value, DB_type = 01 represents a CQ doorbell, DB_type = 10 represents an SQ doorbell, and DB_type = 11 represents an EQ_AEQ doorbell.

[0068] The essence of a Database (DB) is a request notification signal sent by the target host. The RNIC processes the corresponding WR (Write-Out) or WQE (Write-Out-of-Queries) based on the DB's sending order and the parameters it carries. Processing a DB is essentially processing a WR. This addresses the request scheduling and processing issues under large-scale QP (Queries Per Request) concurrency conditions, effectively alleviates the head-blocking problem caused by different QP scheduling, ensures scheduling fairness across different QPs, and guarantees that DBs are not lost even under bursty request conditions.

[0069] like Figure 1 and Figure 2 As shown, the lossless and efficient DB signal processing method proposed in this invention includes the following basic steps:

[0070] Step 1: When a new WQE is generated in the SQ of the target host, the host will generate a doorbell signal DB and send it to the RNIC. That is, the DB information is written to the Doorbell space allocated by the RNIC for this QP through the PCIe interface.

[0071] Where SQ represents the sending queue;

[0072] Step 2: RNIC resolves the PageID, QPN, and priority (CoS) information of the Doorbell address based on the address in the Doorbell register, and adds the Doorbell to the first-stage scheduler. The first-stage scheduler adopts a hierarchical scheduling structure, supporting M groups. Each group can be further divided into 4 priority queues. Within each group, the SP+WRR scheduling algorithm is used, while between different groups, the RR scheduling algorithm is used. SP is the strict priority scheduling algorithm; WRR is the weighted round-robin scheduling algorithm.

[0073] The basic steps for Doorbell to join the first-stage scheduler include:

[0074] Step 2.1: In the first stage, the scheduling module obtains the GroupID of the target queue based on the Hash(PageID) and selects the Doorbell to add to the corresponding priority queue based on the CoS in the DB. If the queue is not full, the Doorbell is directly written to the tail of the queue. If the queue already contains a Doorbell for the QPN, they can be merged into one DB. If the queue is full, proceed to step 2.2.

[0075] Step 2.2: The first-stage scheduling module records Doorbell information in the Overflow Buffer. The Overflow Buffer is a shared cache for all first-stage scheduling queues. In the Overflow Buffer, an entry is assigned to each priority for each GroupID, and the DB information is cached sequentially in a linked list. If a Doorbell is added to the Overflow Buffer, and the Overflow Buffer already contains information about the QPN to which that Doorbell belongs, the new Doorbell replaces the old one. That is, in the buffer, only the latest Doorbell is cached for the same QP.

[0076] In the Overflow Buffer, a bitmap is used to record whether each priority of GroupID has Doorbell information. If Doorbell information is present, the bit is set to 1; otherwise, it is set to 0.

[0077] Optionally, to save on cache, only the QPN of the database is cached in the Overflow Buffer. When the QPN is scheduled, the latest WQE_Index is obtained by reading the DB Record cached in the target host, and a new DB is generated and added to the first-level scheduling output queue of the scheduler to execute the next level of scheduling.

[0078] Step 3: The first-level scheduler of the first-stage scheduler selects the queue to be scheduled using the configured scheduling algorithm and adds the DB to be processed to the output queue of the scheduler. The scheduling algorithm includes, but is not limited to, Round Robin (RR) scheduling.

[0079] The first-level Arbiter schedules different CoS queues within the same Group as follows:

[0080] Step 3.1 If the current polling operation reaches the queue with GroupID=n and CoS=m, first determine if the queue is empty. If it is not empty, remove the DB at the head of the queue and add it to the scheduler's output queue; if the queue is empty, proceed to step 3.2.

[0081] Step 3.2: Read the bitmap of the queue from the Overflow Buffer and determine whether the database information of the queue is cached. If it exists, retrieve the database information from the head of the linked list, add it to the corresponding scheduling output queue, and remove the database information from the linked list. The current scheduling ends, and the system waits for the next scheduling cycle. If it does not exist, skip the queue, poll the next queue, and the current scheduling cycle ends.

[0082] Step 4: The second-level Arbiter of the first-stage scheduler uses RR scheduling to write the DB output by the first-level Arbiter into the output queue.

[0083] Step 5: The DB verification module retrieves the DB at the head of the output queue of the second-level Arbiter from the first-level scheduler and verifies its validity. It checks whether the DB's PageID matches the PageID in the QPC bound to the DB's QPN and whether the QP is in a normal state. If they match, the DB is input into the QPN-QID mapping module; otherwise, the DB is discarded and an error is returned to the target host. Here, QPN represents the QP's sequence number ID; QID represents the local ID assigned to the QP in the RNIC, which is unique in the RDMA system; the QPN-QID mapping module is used for mapping and looking up the QPN and the local QID, using the QID as an index for the local QP Context.

[0084] Step 6: After the DB is input into the QPN-QID mapping module, this module looks up the corresponding GroupID based on the HostID in the DB, and then maps it to the corresponding GroupID's input queue according to the QPN, ensuring that QP DBs with the same HostID are placed in the same scheduling group. Here, GroupID represents the scheduling group ID; different HostIDs can be placed in different scheduling groups.

[0085] HostID represents the host ID. In a virtualization-enabled scenario, each VM corresponds to a unique HostID.

[0086] Specifically, HostID can be represented by PF+VF ID. PF ID stands for Physical Function ID, representing the ID of the physical channel in PCIe; VF ID stands for Virtual Function ID, representing the ID of the PCIe virtual channel. One virtual channel represents one virtual machine.

[0087] The second-stage scheduling module also employs a hierarchical, multi-level scheduling approach. Its scheduling levels, the group size at each level, and the scheduling algorithm for each scheduler can be configured through the scheduling policy module. Scheduling algorithms include, but are not limited to, SP, RR, WRR, and DWRR.

[0088] The first-level input queue of the second-stage scheduling module contains two Entries for caching databases (DBs). The Entry at the head of the queue represents the DB being scheduled, and the second Entry represents the next DB to be scheduled. DBs are added to the input queue according to the following method:

[0089] If the queue is empty, add the new database to the head of the queue; if the queue has only one database, add the new database to the tail of the queue; if there is also a database at the tail of the queue, replace the old database with the new database and cache it in the queue.

[0090] Step 7: In the second-level scheduling module, each level scheduler outputs DB to the final output queue of the module.

[0091] Step 8: The WQE processing module retrieves the DB from the head of the output queue and reads the QPC status required for processing the WQE according to the QPN. The QPC includes Max_Burst_Size and Max_Batch_WQE_count, representing the maximum number of messages a single WQE can send and the maximum number of WQEs that can be processed at once, respectively. Max_Burst_Size represents the maximum number of bytes allowed to be sent in a single scheduling cycle; Max_Batch_WQE_count represents the maximum number of WQEs allowed to be acquired in a single scheduling cycle, which is also the maximum number of messages allowed to be processed in a single scheduling cycle.

[0092] Step 9: The WQE processing module requests Credit from the congestion control module based on (HostID, QPN). Credit is the size of the message that can be sent allocated by the congestion control algorithm for each QP.

[0093] Step 10: The WQE processing module, based on Max_Batch_WQE_count and the current cacheable number of WQEs, WQE_Available_Count, obtains no more than N WQEs from SQ via the DMA engine, where N = min(WQE_Available_Count, Max_Batch_WQE_count). Specifically, WQE_Available_Count represents the maximum number of available WQEs, and N represents the number of WQEs that can be processed in a single scheduling cycle, where N is the minimum of WQE_Available_Count and Max_Batch_WQE_count.

[0094] Step 11: The WQE processing module processes the cached WQEs one by one. For each WQE processed, it updates the WQE consumption pointer in the QPC and updates the Credit value. The Credit value is the current value minus the Credit value consumed by the WQE. If, during the processing of a WQE, the remaining Credit is insufficient to process a complete WQE, the interrupt status of the WQE processing will be returned after the remaining Credit is consumed. This status will be cached in the Interupted DB State table in the second scheduling module, and the bitmap corresponding to that queue will be set to 1. The Interupted DB State includes HostID, QPN, Produce_Index, Target_WQE_Index, and Walk_Offset. Produce_Index represents the current WQE_Index being processed, Walk_Offset represents the pointer to the data already sent for the currently unprocessed WQE, and Target_WQE_Index represents the WQE position that the current DB needs to process. Credit is a token value representing the number of bytes sent by the current QP. The Interupted DB State is an interrupted DB information table used to cache interrupt information for DB scheduling. The bitmap is a bitmap in which each bit corresponds to a QPN. If the QPN information is valid, the corresponding bit is set to 1; otherwise, it is set to 0.

[0095] Step 12: When the second-stage scheduling module schedules a queue in the next cycle, it first checks the bitmap to determine if there is an interrupted DB status in the queue, i.e., whether the bitmap corresponding to the queue is 1. If bitmap = 0, a new DB is read from the scheduling input queue for processing; if bitmap = 1, the DB interrupt status is read first, and the DB interrupt status information is combined into a new DB and sent to the WQE processing module.

[0096] Step 13: Based on the information from the database (DB), the WQE processing module continues processing the MR data starting from the Walk_Offset of the WQE pointed to by Produce_Index. It then uses Credit to determine if the WQE can be completely processed, repeating steps 11-13 until the DB can be completely sent without interruption, at which point the bitmap is set to 0. Here, Produce_Index represents the consumer pointer, indicating the current position of the SQ processing; Walk_Offset represents the virtual address of the interrupt position for a single WQE processing step.

[0097] This method is mainly for situations where a WQE corresponds to a large message or a DB needs to process multiple WQEs. It consumes a lot of Credit, so it may take multiple scheduling cycles to process a DB.

[0098] This concludes the entire two-stage database scheduling and processing method. This method ensures no database loss while effectively reducing the head-blocking problem in QP scheduling, guaranteeing fair WQE processing in multi-QP concurrent scenarios.

[0099] This invention proposes a lossless and efficient data processing method and network interface card based on RDMA, providing two implementations for bare metal scenarios and cloud host scenarios, respectively. The basic modules and methods are largely the same in both scenarios, as shown in the technical solution in Section 5, but the processing of each module in the RNIC differs slightly between the two scenarios.

[0100] Regardless of whether it is a bare metal scenario or a cloud host scenario, the BAR space bound to each host by RNIC is unique, and therefore the PageID is unique. The process of adding the generated DB to the first-stage scheduler is the same under both conditions.

[0101] In bare metal scenarios, only one host creates a QP and issues requests within the host machine, and the QPN is unique. In cloud host scenarios, there are multiple virtual machines (VMs) on the host machine, which can be considered as multiple hosts. Each VM can create a QP and issue requests, and the QPN is unique within the VM, although different VMs can have the same QPN. Therefore, the QPC table needs to be maintained and managed at the host level.

[0102] When mapping QPN to QID, if there is only one host in a bare metal scenario, that host can occupy all the scheduling resources of the group; if there are multiple hosts in a cloud host scenario, groups need to be allocated based on the host.

[0103] Similarly, in the congestion control module, a Credit needs to be maintained for each QP. The management and maintenance of this Credit should also be based on the Host and QPN. Here, PageID is the page ID, representing the ID of the BAR space address corresponding to the software DB; QPC represents the QP's context information, used to cache QP address information; QP represents the RDMA connection queue; hostID represents the host's sequence number ID; QPN represents the QP's sequence number ID; Credit is the credit; WQE represents an RDMA request; and RNIC represents the RDMA network interface card.

[0104] Multi-stage scheduling, with scheduling queues designed based on different scheduling granularities, avoids placing all databases together for serial execution, which can cause severe head-blocking problems.

[0105] The first-stage scheduler module uses a general DB allocation method based on process Page IDs. Unlike the traditional method of binding DB types to addresses, it decouples DB types from addresses, thereby improving the utilization of DB space.

[0106] In the first-stage scheduler module, a DB queue management and scheduling method is designed based on comprehensive factors such as host, process, and priority. This allows for scheduling at different granularities, ensuring that high-priority requests are effectively scheduled while maintaining order within the same QP, thus effectively solving the head-blocking problem under large-scale QP concurrency conditions.

[0107] In the first phase of the scheduler, the lossless DB processing method using the shared Overflow Buffer ensures that DB information is not lost under high concurrency conditions, thus achieving lossless request processing; at the same time, only DB status information is cached, and lost DB information is obtained by reading the DB Record, effectively saving cache.

[0108] In the second-stage scheduling module, the number of data packets generated by WQEs that can be processed in each scheduling cycle is limited by the constraint of the number of Credits + WQEs. This ensures the fairness of multi-QP scheduling, prevents large messages from occupying resources and blocking other QPs, and alleviates the head-blocking problem.

[0109] In the second-stage scheduling module, by adding an Interupted DB state cache to each scheduling queue to handle interrupted states, WQE breakpoint resume capability is supported during scheduling, achieving fine-grained QP scheduling and ensuring efficient and lossless request processing.

[0110] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0111] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A network interface card based on RDMA, characterized in that: Includes a PCIe BAR register processing module, a first-stage scheduling module, a QPC status table, a DB verification module, a QPN-QID mapping module, a second-stage scheduling module, a scheduling policy configuration module, a congestion control module, and a DMA engine. The PCIe BAR register processing module is responsible for parsing and processing the doorbell signal DB in the software and hardware interaction. The first-stage scheduling module is responsible for scheduling the doorbell signal database based on PageID and priority in the first stage, ensuring that the doorbell signal database is not lost. QPC status table, which caches QPC status information; The DB verification module is used to verify the validity of the DB and whether the status information of the corresponding QP is correct. The QPN-QID mapping module is used to assign enqueue IDs to DBs to enter the second-stage scheduling module; The second-stage scheduling module is responsible for scheduling the DB based on hostID and QPN in the second stage, ensuring that different QPs of different hosts are scheduled fairly. The scheduling strategy configuration module is used to configure the strategies such as the number of scheduling levels, the size of each scheduler, and the scheduling algorithm in the second-stage scheduling module. The congestion control module is used to allocate Credit to each QP and control the size of messages sent by each QP in a scheduling cycle. The WQE processing module is used for WQE prefetching and processing. If WQE cannot be processed completely, it returns the breakpoint information of DB processing to the second-stage scheduling module. The DMA engine is used for data transfer between the RNIC and the Host. Where PageID is the page ID, representing the ID of the BAR space address corresponding to the software's DB; QPC represents the context information of QP, used to cache QP address information; QP represents the RDMA connection queue; hostID represents the host's serial number ID; QPN represents the QP's serial number ID; Credit is the credit; WQE represents an RDMA request; and RNIC represents the RDMA network card.

2. A lossless and efficient data stream processing method for RDMA in a network interface card based on the RDMA described in claim 1, characterized in that: Specifically, it includes the following steps: Step 1: When a new WQE is generated in the SQ of the target host, the host will generate a doorbell signal DB and send it to the RNIC; that is, the DB information is written to the Doorbell space allocated by the RNIC for this QP through the PCIe interface. Where SQ represents the sending queue; Step 2: RNIC resolves the PageID of the address in the Doorbell register, as well as the Doorbell's QPN and priority CoS information, and adds the Doorbell to the first-stage scheduler. The first-stage scheduler adopts a hierarchical scheduling structure, supporting M groups. Each group can be further divided into 4 priority queues. The SP+WRR scheduling algorithm is used in each group, and the RR scheduling algorithm is used between different groups. SP is the strict priority scheduling algorithm; WRR is the weighted round-robin scheduling algorithm. Step 3: The first-level scheduler of the first-stage scheduling module selects the queue to be scheduled using the configured scheduling algorithm and adds the DB to be processed to the scheduler's output queue. Step 4: The second-level scheduler of the first-stage scheduling module uses RR scheduling to write the DB output by the first-level scheduler into the output queue; Step 5: The DB verification module retrieves the DB from the head of the second-level output queue of the first-level scheduler and verifies its validity. It checks whether the DB's PageID matches the PageID in the QPC bound to the DB's QPN and whether the QP's status is normal. If they match, the DB is input into the QPN-QID mapping module; otherwise, the DB is discarded and an error is returned to the target host. Here, QPN represents the QP's sequence number ID; QID represents the local ID assigned to the QP in the RNIC, which is unique in the RDMA system. The QPN-QID mapping module is used for mapping and looking up the mapping between the QPN and the local QID, using the QID as an index for the local QP Context. Step 6: After the DB inputs to the QPN-QID mapping module, it looks up the corresponding GroupID based on the HostID in the DB, and maps the QPN to the input queue of the corresponding GroupID to ensure that QP and DB with the same HostID are placed in the same scheduling group. GroupID represents the ID of the scheduling group, and different HostIDs are placed in different scheduling groups; HostID represents the host ID. In a virtualization-enabled scenario, each VM corresponds to a unique HostID. Step 7: In the second-level scheduling module, each level scheduler outputs the DB to the final output queue of the module. Step 8: The WQE processing module retrieves the DB from the head of the output queue and reads the QPC status required for processing WQE according to the QPN. The QPC contains Max_Burst_Size and Max_Batch_WQE_count, representing the maximum number of messages sent by a single WQE and the maximum number of WQEs processed in a single operation, respectively. Max_Burst_Size represents the maximum number of bytes allowed to be sent in a single scheduling cycle; Max_Batch_WQE_count represents the maximum number of WQEs allowed to be acquired in a single scheduling cycle, which is also the maximum number of messages allowed to be processed in a single scheduling cycle. Step 9: The WQE processing module requests Credit from the congestion control module; where Credit is the size of the sendable message allocated to each QP by the congestion control algorithm. Step 10: The WQE processing module obtains no more than N WQEs from SQ via the DMA engine based on Max_Batch_WQE_count and the current cacheable number of WQEs, WQE_Available_Count, where N = min(WQE_Available_Count, Max_Batch_WQE_count); WQE_Available_Count represents the maximum number of available WQEs; N represents the number of WQEs processed in a single scheduling cycle, where N is the minimum value of WQE_Available_Count and Max_Batch_WQE_count. Step 11: The WQE processing module processes the cached WQEs one by one. For each WQE processed, the consumption pointer of the WQE in the QPC is updated, and the Credit value is updated. The Credit value is the current value minus the Credit value consumed by the WQE. If, during the processing of a WQE, the remaining Credit is insufficient to process a complete WQE, the interrupted status of the WQE processing will be returned after the remaining Credit is consumed. This information is cached in the Interupted DB State table in the second scheduling module, and the bitmap corresponding to this queue is set to 1. The Interupted DB State includes HostID, QPN, Produce_Index, Target_WQE_Index, and Walk_Offset. Produce_Index represents the current WQE_Index being processed, Walk_Offset represents the pointer to the data already sent by the currently unprocessed WQE, and Target_WQE_Index represents the WQE position that the current DB needs to process. Credit is a token value representing the number of bytes sent by the current QP. State is the interrupt DB information table, used to cache interrupt information of DB scheduling; bitmap is a bitmap, each bit in the bitmap corresponds to QPN. If the QPN information is valid, the corresponding bit is set to 1, and if it is invalid, it is set to 0. Step 12: When the second-stage scheduling module schedules a queue in the next cycle, it first uses Bitmap to determine whether there is an interrupted DB status in the queue, that is, whether the bitmap corresponding to the queue is 1; if bitmap=0, a new DB is read from the scheduling input queue for processing; if bitmap=1, the DB interrupt status is read first, and the DB interrupt status information is combined into a new DB and sent to the WQE processing module. Step 13: Based on the information in the DB, the WQE processing module continues to process the MR data starting from the Walk_Offset of the WQE pointed to by Produce_Index. It determines whether the WQE has been completely processed based on the Credit, and repeats steps 11-13 until the DB no longer interrupts and all data is sent out, at which point the bitmap is set to 0. Here, Produce_Index represents the consumption pointer, indicating the current position of the SQ processing; Walk_Offset represents the virtual address of the interrupt position of a single WQE processing.

3. The lossless and efficient data stream processing method based on RDMA according to claim 2, characterized in that: The basic steps for Doorbell to join the first-stage scheduler include: Step 2.1: The first-stage scheduling module obtains the GroupID of the target queue based on the Hash(PageID) and selects the Doorbell to add to the corresponding priority queue based on the CoS in the DB. If the queue is not full, the Doorbell is directly written to the tail of the queue. If the queue contains the Doorbell of the QPN, they are merged into 1 DB. If the queue is full, proceed to step 2.

2. CoS stands for Channel of Service, which represents the priority channel and typically supports eight priority levels. Step 2.2: The first-stage scheduling module records Doorbell information in the Overflow Buffer; the Overflow Buffer is a shared cache for all first-stage scheduling queues; in the Overflow Buffer, an entry is allocated for each priority of each GroupID, and the DB information is cached sequentially in a linked list; if a Doorbell is added to the Overflow Buffer, and the information of the QPN to which the Doorbell belongs already exists in the Overflow Buffer, then the new Doorbell replaces the old Doorbell, that is, in the buffer, only the latest Doorbell is cached for the same QP; in the Overflow Buffer, a bitmap is used to record whether there is a Doorbell for each priority of GroupID. If there is Doorbell information, the bit is set to 1, otherwise it is set to 0.

4. The lossless and efficient data stream processing method based on RDMA according to claim 2, characterized in that: The Overflow Buffer only caches the QPN of the DB. When the QPN is scheduled, the latest WQE_Index is obtained by reading the DBRecord cached on the target host, and a new DB is generated and added to the first-level scheduling output queue of the scheduler to execute the next level of scheduling. The DB Record is the content cached on the host side, which is used to record the producer index and consumer index of the SQ.

5. The lossless and efficient data stream processing method based on RDMA according to claim 2, characterized in that: The first-level arbiter or scheduler Arbiter uses the following method to schedule different CoS queues within the same Group: Step 3.1: If the current polling reaches the queue with GroupID=n and CoS=m, first determine if the queue is empty; if it is not empty, take the DB at the head of the queue and add it to the scheduler's output queue. If the queue is empty, proceed to step 3.2; Step 3.2: Read the bitmap of the queue from the Overflow Buffer and determine whether the DB information of the queue is cached. If it exists, retrieve the DB information from the head of the linked list, add it to the corresponding scheduling output queue, and delete the DB information from the linked list. The current scheduling ends, and the system waits for the next scheduling cycle. If it does not exist, skip the queue and poll the next queue. The current scheduling cycle ends. The Overflow Buffer is an overflow buffer. When the scheduling queue is full, the DB information is cached in the overflow buffer. This buffer is a shared buffer for all scheduling queues.

6. The lossless and efficient data stream processing method based on RDMA according to claim 2, characterized in that: HostID is represented by PF+VF ID; where PF ID stands for physical function ID, which represents the ID of the physical channel in PCIe; VFID stands for virtual function ID, which represents the ID of the virtual channel in PCIe, and one virtual channel represents one virtual machine.

7. The lossless and efficient data stream processing method based on RDMA according to claim 2, characterized in that: The second-stage scheduling module also adopts a hierarchical multi-level scheduling approach. Its scheduling level, the group size of each level, and the scheduling algorithm of each level scheduler can be configured through the scheduling strategy module.

8. The lossless and efficient data stream processing method based on RDMA according to claim 2, characterized in that: Scheduling algorithms include, but are not limited to, SP, RR, WRR, and DWRR scheduling algorithms; SP stands for strict priority scheduling algorithm; RR stands for Round Robin algorithm; WRR is a weighted round-robin scheduling algorithm; DWRR is a differential weighted round-robin algorithm.

9. The lossless and efficient data stream processing method based on RDMA according to claim 2, characterized in that: The first-level input queue of the second-stage scheduling module contains two cached lines of entries for caching databases (DBs). The entry at the head of the queue represents the DB being scheduled, and the second entry represents the next DB to be scheduled.

10. The lossless and efficient data stream processing method based on RDMA according to claim 2, characterized in that: When a database (DB) is added to the input queue, the following method is followed: if the queue is empty, the new DB is added to the head of the queue; if the queue contains only one DB, the new DB is added to the tail of the queue; if the tail of the queue also contains a DB, the new DB replaces the old DB and is cached in the queue.

Images