A doorbell control method, system and medium for an RDMA virtualization scenario

Abstract

A doorbell control method, system and medium of an RDMA virtualization scene, the method comprising: creating a producer-consumer control model of a function-based QP doorbell, the producer-consumer control model being a ring-shaped storage space; and implementing doorbell control in an RDMA network based on the producer-consumer control model. Through the method, the register resources of the hardware can be reduced, and the user experience can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to, but is not limited to, the technical field of Remote Direct Data Read (RDMA), and in particular to a doorbell control method, system and medium for an RDMA virtualization scenario. BACKGROUND

[0002] Single Root Input / Output Virtualization (SR-IOV) technology is to simulate a physical device into multiple virtual devices, each of which can be bound to a virtual machine, thereby facilitating multiple virtual machines to access the same physical device. SR-IOV introduces two new peripheral component interconnect express functions (PCIe Functions), namely physical function (physical function) and virtual function (virtual function), wherein the physical function is a complete PCIe function, and the virtual function is a lightweight PCIe function.

[0003] RDMA is a message-based transmission protocol, and data transmission is an asynchronous operation. An RDMA operation can be understood as follows: a host (Host) submits a work request (WR: Work Request) to a work queue (WQ: Work Queue), wherein the work queue includes a send queue (SQ: Send Queue) and a receive queue (RQ: Receive Queue). Each element of the work queue is called a work queue element (WQE: Work Queue Element). The host obtains a work completion (WC: Work Completion) from a completion queue. Each element in the completion queue is called a completion queue element (CQE: Completion Queue Element).

[0004] Hardware with an RDMA engine is a queue element processor. The RDMA hardware continuously takes a work request (WR) from a work queue (WQ) to execute, and after execution, places a work completion (WC) in a completion queue (CQ).

[0005] With the network scale of Remote Direct Memory Access (RDMA) getting larger and the demand of RDMA for virtualization, the number of queue pairs (QP) of RDMA is also increasing. On a traditional RDMA network card, the driver controls the network card through writing the internal registers of the network card, which is simple and efficient. However, with the increasing number of QPs, the number of registers for doorbell control will also increase exponentially, which will waste a lot of hardware resources. SUMMARY

[0006] The following is a summary of the subject matter of the detailed description herein. This summary is not intended to limit the scope of the claims.

[0007] An embodiment of the present disclosure provides a doorbell control method in a Remote Direct Memory Access (RDMA) virtualization scenario, the method comprising:

[0008] creating a producer-consumer control model of QP doorbell based on functions, the producer-consumer control model being a ring memory space;

[0009] implementing doorbell control in the RDMA network based on the producer-consumer control model.

[0010] An embodiment of the present disclosure also provides a doorbell control system in an RDMA virtualization scenario, comprising a host installed with a network card driver and a host memory and a network card with an RDMA engine, wherein: the network card driver is configured to perform the processing performed by the driver in the doorbell control method according to any embodiment of the present disclosure, and the network card is configured to perform the processing performed by the network card in the doorbell control method according to any embodiment of the present disclosure.

[0011] An embodiment of the present disclosure also provides a non-transitory computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, can implement the doorbell control method in an RDMA virtualization scenario according to any embodiment of the present disclosure.

[0012] The above embodiments of the present disclosure can reduce the register resources of hardware and improve user experience.

[0013] Other aspects can become apparent after consideration of the drawing and detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 a flowchart of a Remote Direct Memory Access (RDMA) method according to an embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram of a doorbell control system in an embodiment of RDMA virtualization scenario of the present disclosure. DETAILED DESCRIPTION

[0016] The present disclosure describes a number of embodiments, but the description is illustrative rather than limiting and many other embodiments and implementations will be apparent to those of ordinary skill in the art in view of this disclosure. Although a number of possible combinations of features have been set forth herein, many other combinations will be apparent to those of ordinary skill in the art and can be made without departing from the spirit and scope of embodiments of the disclosure. Unless specifically intended otherwise, any feature or element in any embodiment can be utilized in any other embodiment, whether or not that particular combination of features is specifically stated in the embodiments. In addition, some of these combinations will also be explicitly evident to those of ordinary skill in the art from an inspection of the disclosure, and can be subject to abrogation, either explicitly stated or otherwise inherent in the disclosure.

[0017] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The presently disclosed embodiments, features and elements can also be combined with any conventional feature or element to form a unique application of the presently claimed disclosure that is not specifically disclosed. Any feature or element of any embodiment can also be combined with features or elements from other applications to form another unique application of the presently claimed disclosure that is not specifically disclosed. Thus, it should be understood that any feature shown and / or discussed in the present disclosure can be used, either alone or in any combination, in the practice of the present disclosure. Therefore, the embodiments are not to be limited by other than in accordance with the restrictions shown and described in the appended claims, and equivalents thereof. Further, various modifications and changes can be made thereto within the scope and spirit of claims that follow this description.

[0018] Further, in describing representative embodiments, the specification can have presented the method and / or process as a particular sequence of steps. However, to the extent that the method or process depends on more than one step, the method or process should not be limited to the particular sequence of steps described. Other sequences of steps can be possible, and are within the spirit and scope of the embodiments. Therefore, the particular order in which the steps of the specification are presented is not construed as limiting. Further, the specification's recitation of certain methodologies should not be limiting, as other methodologies can be employed, and the described ones can vary from the methodologies written. The claims should not be limited by the steps of the methods in the specification, as the specific sequence of steps in the claims is not required to invoke their application.

[0019] The "network card" in the present disclosure can be an Ethernet card with RDMA engine (RNIC), and the "driver" can be a host-installed network card driver.

[0020] The function described in the present disclosure refers to the PCIE function in the above, and the resources of each function can be considered to be independent of each other, so the doorbell needs to be designed with an independent doorbell control logic for different PCIE functions.

[0021] Regarding the doorbell described in the present disclosure, when the QP needs to send data, the driver generates a WQE (Work Queue Element), and then notifies the network card to obtain the WQE to send or receive RDMA messages. The process of notifying the network card is called doorbell. The information carried in the doorbell can indicate the storage location of the WQE in the host memory. The WQE is usually placed in the host memory in the producer-consumer control model, so the latest producer pointer can be placed in the doorbell to notify the network card of the storage location of the WQE to be processed. In addition, the QP belongs to a certain function, and the resources between functions are isolated, so the QP can only send doorbells to the network card through the doorbell model of the function to which the QP belongs.

[0022] In order to reduce the register resources of the hardware, the doorbell of the QP needs to be transmitted from the driver to the network card through the producer-consumer model. On the other hand, in order to meet the virtualization requirements, each function needs to design a set of independent producer-consumer control models to control the transmission of the doorbell of the QP, in order to meet the resource isolation between each function.

[0023] An embodiment of the present disclosure provides a doorbell control method in a remote direct data read (RDMA) virtualization scenario, as shown in Figure 1 The method can be executed in the following steps:

[0024] Step S110, creating a producer-consumer control model of a function-based QP doorbell, the producer-consumer control model being a ring-shaped storage space;

[0025] Step S120, implementing doorbell control in an RDMA network based on the producer-consumer control model.

[0026] The ring-shaped storage space is that when the pointer points to the last piece of area of the ring-shaped storage space, the next storage space pointed to by the pointer is the first piece of area of the ring-shaped storage space.

[0027] The embodiment is especially suitable for implementing doorbell control in an RDMA network supporting a large number of QP connections. By creating a producer-consumer control model of a function-based QP doorbell, the register resources of the hardware can be reduced, and the user experience can be greatly improved.

[0028] The reason that the embodiment can reduce the register resource of the hardware can be specifically explained by the following example. It is assumed that there are 1K QPs, each of which uses a doorbell, and the size of each doorbell information is 16 bits (the doorbell information is 16 bits is only an assumption, and actually each doorbell information is greater than 16 bits). Therefore, the total consumed resource is 16*1K = 16K bits. If the producer-consumer control model is used for control, it is assumed that the physical address of the host is 64 bits, the depth of the model is 64K at most, and the size of each element of the model is 16 bits. Therefore, the network card only needs to design a 64-bit register recording the starting address of the model, a 16-bit register of the size of the producer-consumer control model, a 16-bit producer pointer and a 16-bit consumer pointer, and a total of 112-bit register resource. Therefore, when there are 1K QPs, 16K bits-112 bits = 15.9K bits of register resource can be saved, and the more the number of QPs is, the more register resource can be saved.

[0029] In an example embodiment of the present disclosure, the producer-consumer control model of the function-based QP doorbell is created, and the method comprises the steps of:

[0030] The producer-consumer control model of the function-based QP doorbell is created, and the method comprises the steps of:

[0031] In an example embodiment of the present disclosure, the producer-consumer control model of the function-based QP doorbell is created, and the method comprises the steps of:

[0032] The control information of the producer-consumer control model of the QP doorbell created for each function is refreshed into the corresponding register space of the network card by the driver.

[0033] The function comprises a PCIe function, and the control information of the producer-consumer control model can comprise the storage address of the host memory space of the producer-consumer control model and the size of the storage space.

[0034] In an example of the embodiment, when the driver needs to trigger a certain QP doorbell in a certain function, the doorbell control in the RDMA network is realized based on the producer-consumer control model, and the method comprises the steps of:

[0035] When the driver needs to issue the QP doorbell of the first function, the consumer pointer of the first producer-consumer control model maintained by the network card is read, and the consumer pointer is compared with the producer pointer of the first producer-consumer control model maintained by the driver.

[0036] The drive writes the bell information of the QP bell to be issued into the storage space of the first producer-consumer control model according to the comparison result when judging that there is space in the first producer-consumer control model to issue a new QP bell, and updates the producer pointer of the first producer-consumer control model;

[0037] The first producer-consumer control model is a producer-consumer control model of QP bells created for the first function.

[0038] The producer pointer is moved one bit backward after adding one WQE to the QP when the drive is responsible for adding the WQE to the QP, and the consumer pointer is moved one bit after executing one WQE when the network card is responsible for executing the WQE request, and the producer pointer and the consumer pointer are used in the same way, that is, starting from 0 and accumulating, and when the value accumulates to the maximum value, it will cycle to 0 and continue to accumulate.

[0039] In an example of the embodiment, when the producer pointer is equal to the consumer pointer, the number of QP bell information stored in the first producer-consumer control model is 0; when the producer pointer plus 1 is equal to the consumer pointer, the number of QP bell information stored in the first producer-consumer control model reaches the upper limit of the storage of the first producer-consumer control model; and the number of QP bells that can be stored in the first producer-consumer control model is the value of the consumer pointer minus the producer pointer minus 1.

[0040] In an example of the embodiment, the producer-consumer control model is used to realize the bell control in the RDMA network, and further includes:

[0041] The drive synchronously refreshes the producer pointer of the first producer-consumer control model updated by the function to the register space corresponding to the network card;

[0042] After receiving the update of the producer pointer of the first producer-consumer control model, the network card determines the number of QP bells that can be consumed according to the updated producer pointer and the consumer pointer of the first producer-consumer control model;

[0043] The network card acquires the corresponding number of QP bell information from the storage space of the first producer-consumer control model according to the control information of the first function;

[0044] The network card executes corresponding actions according to the content of each QP bell information acquired, and updates the consumer pointer of the first producer-consumer control model, at this time, the storage space occupied by the corresponding QP bell in the first producer-consumer control model is released.

[0045] The embodiment creates a producer-consumer control model of a function-based QP doorbell in a host space, and can realize control when the QP doorbell data is large based on sufficient host memory space.

[0046] In an example of the embodiment, the implementation of the doorbell control in the RDMA network based on the producer-consumer control model in the host memory can include the following steps:

[0047] Step (1): In the initialization stage, a piece of storage space of the producer-consumer control model of the QP doorbell is applied for each activated function in the host memory, and the control information of the QP doorbell created by the function is flushed to the register space corresponding to the network card.

[0048] Step (2): When the driver needs to trigger a certain QP doorbell in a certain function, the driver first needs to read the consumer pointer of the QP doorbell producer-consumer control model maintained by the network card through the register, and then compare the producer pointer of the model maintained by the driver to determine whether there is space for a new QP doorbell in the model.

[0049] Step (3): When the driver determines that there is enough write space in the QP doorbell producer-consumer control model of the function, the QP doorbell information of the QP to be generated is written into the storage space of the QP doorbell producer-consumer control model of the function, and the producer pointer of the producer-consumer control model is updated.

[0050] Step (4): The driver synchronously flushes the producer pointer of the updated QP doorbell producer-consumer control model of the function to the register of the network card.

[0051] Step (5): The network card maintains the consumer pointer of the QP doorbell producer-consumer control model of a certain function, and after receiving the producer pointer update of the QP doorbell producer-consumer control model of a certain function, the network card determines the number of unconsumed QP doorbells (the number of unconsumed QP doorbell information is the difference between the producer pointer and the consumer pointer) through the difference between the producer pointer and the consumer pointer, and acquires a corresponding number of QP doorbell information from the storage space of the QP doorbell producer-consumer control model according to the current consumable QP doorbell number of the network card, and the network card executes corresponding actions according to the content of each doorbell information.

[0052] Step (6): The subsequent QP doorbell information transmission is performed through steps (2) to (5).

[0053] In another example embodiment of the disclosure, the creation of the function-based QP doorbell producer-consumer control model includes:

[0054] A storage space for storing QP doorbell information is created in the network card, and the driver initializes a producer-consumer control model storage space for each activated function in the storage space;

[0055] The driver flushes the control information of the producer-consumer control model of the QP doorbell created for each function to the corresponding register space of the network card.

[0056] The function includes a PCIe function, and the control information of the producer-consumer control model can include the address of the producer-consumer control model in the storage space of the network card and the size of the storage space.

[0057] Since the internal resources of the network card are limited, creating a model requires a corresponding resource cost, so as to balance performance and resources, the embodiment creates a producer-consumer control model in a small space on the network card, which can realize low-latency control when the QP doorbell data is small.

[0058] In an example of the embodiment, the producer-consumer control model is used to realize doorbell control in an RDMA network, including:

[0059] When the driver needs to issue a QP doorbell of a first function, the driver reads the consumer pointer of the first producer-consumer control model maintained by the network card and compares it with the producer pointer of the first producer-consumer control model maintained by the driver;

[0060] When the driver determines according to the comparison result that there is space in the first producer-consumer control model to issue a new QP doorbell, the driver writes the doorbell information of the QP doorbell to be issued into the storage space of the first producer-consumer control model, and updates the producer pointer of the first producer-consumer control model;

[0061] The first producer-consumer control model is a producer-consumer control model of a QP doorbell created for the first function.

[0062] The producer pointer is moved one bit backward each time a WQE is added to the QP when the driver is responsible for adding the WQE to the QP, and the consumer pointer is moved one bit each time a WQE request is executed by the network card, and the producer pointer and the consumer pointer are used in the same way, that is, they start from 0 and accumulate, and when the value accumulates to the maximum value, it will cycle to 0 and continue to accumulate.

[0063] In an example of the embodiment, when the producer pointer is equal to the consumer pointer, the number of QP doorbell information stored in the first producer-consumer control model is 0; when the producer pointer plus 1 is equal to the consumer pointer, the number of QP doorbell information stored in the first producer-consumer control model reaches the upper limit of the storage of the first producer-consumer control model; and the remaining storable QP doorbells of the first producer-consumer control model are the value of the consumer pointer minus the producer pointer minus 1.

[0064] In an example of the embodiment, the producer-consumer control model is used to realize doorbell control in an RDMA network, and further includes:

[0065] The driver synchronously updates and flushes the producer pointer of the first producer-consumer control model after the function update to the register space corresponding to the network card;

[0066] After receiving the producer pointer update of the first producer-consumer control model, the network card determines the number of currently consumable QP doorbells according to the updated producer pointer and the consumer pointer of the first producer-consumer control model;

[0067] The network card acquires a corresponding number of QP doorbell information from the storage space of the first producer-consumer control model according to the control information of the first function;

[0068] The network card performs corresponding actions according to the content of each acquired QP doorbell information, and updates the consumer pointer of the first producer-consumer control model, which releases the storage space occupied by the corresponding QP doorbell in the first producer-consumer control model.

[0069] In an example of the embodiment, the producer-consumer control model in the storage space of the network card is used to realize doorbell control in an RDMA network, and can include the following steps:

[0070] Step (1): A storage space is designed in the network card to store QP doorbell information, and the storage space is initialized and managed by the driver. In the initialization stage, a piece of producer-consumer control model storage space of QP doorbell is applied for each activated function in the storage space of the network card, and the control information of the producer-consumer control model of the QP doorbell created by these functions is flushed to the register space corresponding to the network card.

[0071] Step (2): When a QP doorbell in a function needs to be triggered, the driver first needs to read the producer-consumer control model of the QP doorbell maintained by the network card through a register, and then compare the consumer pointer of the model with the producer pointer maintained by the driver to determine whether there is enough space in the producer-consumer control model of the QP doorbell of the function to generate a new QP doorbell.

[0072] Step (3): When the driver determines that there is enough space in the producer-consumer control model of the QP doorbell of the function to generate a new QP doorbell, the driver writes the doorbell information of the QP that needs to be generated into the storage space of the producer-consumer control model of the QP doorbell of the function, and updates the producer pointer of the producer-consumer control model of the QP doorbell of the function.

[0073] Step (4): The driver synchronously flushes the updated producer pointer of the producer-consumer control model of the QP doorbell of the function to the register of the network card.

[0074] Step (5): The network card maintains the consumer pointer of the producer-consumer control model of the QP doorbell of each function. After the network card receives the update of the producer pointer of the producer-consumer control model of the QP doorbell of a function, the network card determines the number of unconsumed QP doorbells by the difference between the producer pointer and the consumer pointer (the number of unconsumed QP doorbell information is the producer pointer minus the consumer pointer), and acquires a corresponding number of QP doorbell information from the storage space of the producer-consumer control model of the QP doorbell according to the current number of consumable QP doorbells of the network card, and executes corresponding actions according to the content of each doorbell information.

[0075] Step (6): Repeat steps (2) to (5) to transmit subsequent QP doorbell information.

[0076] In another embodiment of the present embodiment, the creation of the function-based QP doorbell producer-consumer control model comprises:

[0077] In the initialization phase, the driver applies a piece of storage space of the producer-consumer memory control model of the QP doorbell in the host memory for each activated function, and flushes the control information of the QP doorbell producer-consumer memory control model created for each function to the corresponding register space of the network card.

[0078] In the network card, a storage space for storing QP doorbell information is created. In the initialization phase, the driver applies a piece of storage space of the producer-consumer network card control model of the QP doorbell in the storage space for each activated function, and flushes the control information of the QP doorbell producer-consumer network card control model created for each function to the corresponding register space of the network card.

[0079] The storage size of the producer-consumer network card control model created in the storage space of the network card is smaller than the storage size of the producer-consumer memory control model created in the host space.

[0080] In an example of the embodiment, the doorbell control in the RDMA network is implemented based on the producer-consumer control model, including:

[0081] When the driver needs to issue a QP doorbell of the first function, if it is determined that there is unconsumed QP doorbell information in the first producer-consumer memory control model or the unconsumed QP doorbell information in the first producer-consumer network card control model reaches the upper limit of the storage space, the doorbell information of the QP doorbell to be issued is written into the storage space of the first producer-consumer memory control model, and the producer pointer of the first producer-consumer memory control model is updated.

[0082] The first producer-consumer memory control model is a producer-consumer memory control model of the QP doorbell created for the first function, and the first producer-consumer network card control model is a producer-consumer network card control model of the QP doorbell created for the first function.

[0083] When the driver determines that there is unconsumed QP doorbell information in the first producer-consumer memory control model, the doorbell information of the new QP doorbell is written into the first producer-consumer memory control model, which can enable the network card to execute each doorbell in the order in which the driver issues the doorbell. Because the network card preferentially acquires the QP doorbell in the storage space of the first producer-consumer network card control model, when the number of stored doorbell information in the first producer-consumer memory control model is not 0, if the driver writes the doorbell information into the storage space of the first producer-consumer network card control model, the doorbell generated later will be used first by the network card, and the order of doorbell use cannot be preserved. Therefore, the doorbell information of the new QP doorbell is written into the first producer-consumer memory control model for storage at this time.

[0084] In an example of the embodiment, the doorbell control in the RDMA network is implemented based on the producer-consumer control model, including:

[0085] When the driver needs to issue a QP doorbell of the first function, if it is determined that there is unconsumed QP doorbell information in the first producer-consumer memory control model or the unconsumed QP doorbell information in the first producer-consumer network card control model reaches the upper limit of the storage space, the doorbell information of the QP doorbell to be issued is written into the storage space of the first producer-consumer memory control model, and the producer pointer of the first producer-consumer memory control model is updated.

[0086] The first producer-consumer memory control model is a producer-consumer memory control model of the QP doorbell created for the first function, and the first producer-consumer network card control model is a producer-consumer network card control model of the QP doorbell created for the first function.

[0087] In the example, when there is space to write QP doorbell information in the first producer-consumer network card control model, and the number of QP doorbell information stored in the first producer-consumer memory control model is 0, the new QP doorbell information is written into the storage space of the first producer-consumer network card control model, which will not damage the in-order execution of the QP doorbell information, and because the QP doorbell information is stored in the first producer-consumer network card control model first, the processing speed of the network card for the doorbell information will be faster, and the latency will be lower.

[0088] In an example of the embodiment, the producer pointer is moved back by one bit when the driver is responsible for adding a WQE to the QP, and the consumer pointer is moved by one bit when the network card is responsible for executing the operation of the WQE request, and the producer pointer and the consumer pointer are used in the same way, that is, from 0 to accumulate, and when the value accumulates to the maximum value, it will loop to 0 to continue to accumulate.

[0089] In an example of the embodiment, the producer-consumer control model is used to implement doorbell control in an RDMA network, and further includes:

[0090] The driver synchronously refreshes the producer pointer of the first producer-consumer memory control model and the first producer-consumer network card control model to the register space corresponding to the network card after the function is updated;

[0091] After receiving the producer pointer update of the first producer-consumer memory control model or the first producer-consumer network card control model, the network card determines the number of currently consumable QP doorbells according to the updated producer pointer and the consumer pointer of the first producer-consumer memory control model or the first producer-consumer network card control model;

[0092] The network card obtains a corresponding number of QP doorbell information from the storage space of the first producer-consumer memory control model or the first producer-consumer network card control model according to the control information of the first function;

[0093] The network card executes corresponding actions according to the content of each obtained QP doorbell information, and updates the consumer pointer of the first producer-consumer memory control model or the first producer-consumer network card control model.

[0094] In the embodiment, when the producer-consumer memory control model and the producer-consumer network card control model are used together, the doorbell information is written into the producer-consumer network card control model first, and the doorbell information is written into the producer-consumer memory control model only when the doorbell information stored in the producer-consumer network card control model reaches the upper limit. Since the network card can process the doorbell information faster and has a lower processing delay, the low-delay control can be realized when the amount of QP doorbell data is small. When the amount of QP doorbell data is large, the doorbell information is written into the producer-consumer memory control model, so that the use scenario when a large amount of doorbell information is met can also be met.

[0095] In an example of the embodiment, the producer-consumer memory control model in the host memory and the producer-consumer network card control model in the storage space of the network card are used together to realize the doorbell control in the RDMA network, which can include the following steps:

[0096] Step (1): A storage space is designed in the network card to store QP doorbell information, and the storage space is initialized and managed by the driver. In the initialization stage, a piece of storage space of the producer-consumer network card control model of the QP doorbell is applied for each activated function in the storage space of the network card, and the control information of the producer-consumer network card control model of the QP doorbell created by the function is refreshed to the corresponding register space of the network card.

[0097] Step (2): In the initialization stage, a piece of storage space of the producer-consumer memory control model of the QP doorbell is applied for each activated function in the host memory, and the control information of the producer-consumer memory control model of the QP doorbell created by the function is refreshed to the corresponding register space of the network card.

[0098] Step (3): When the driver needs to trigger a QP doorbell in a function, the driver first needs to read the consumer pointers of the producer-consumer network card control model of the QP doorbell in the network card storage space and the consumer pointers of the producer-consumer memory control model of the QP doorbell in the host memory storage space through the register, and then compare the producer pointers of the two pieces of space maintained by the driver to determine whether there is space for a new QP doorbell in the model.

[0099] Step (4): When the driver determines that there is unconsumed QP doorbell information in the producer-consumer memory control model of the host memory storage space of the function or the unconsumed QP doorbell information in the producer-consumer network card control model of the network card storage space reaches the upper limit of the storage space, the new QP doorbell information is transmitted to the producer-consumer memory control model of the host memory storage space.

[0100] When the producer-consumer network card control model of the function's network card storage space's QP doorbell has space to write QP doorbell information and the number of QP doorbell information stored in the producer-consumer memory control model of the host memory storage space is 0, the new QP doorbell information is transmitted to the producer-consumer network card control model of the network card storage space, and the producer pointer of the producer-consumer network card control model or the producer-consumer memory control model of the function is updated.

[0101] Step (5): The driver synchronously refreshes the producer pointer of the updated producer-consumer memory control model and the producer-consumer network card control model of the function to the register of the network card.

[0102] Step (6): The network card maintains the consumer pointer of the producer-consumer memory control model and the producer-consumer network card control model of each function. After receiving the update of the producer pointer of the producer-consumer memory control model and the producer-consumer network card control model of a certain function, the network card determines the number of QP doorbells in the host memory space or the network card storage space that have not been consumed (the difference between the producer pointer and the consumer pointer is the number of QP doorbell information that has not been consumed) according to the difference between the producer pointer and the consumer pointer, and obtains a corresponding number of QP doorbell information from the storage space of the producer-consumer memory control model or the producer-consumer network card control model according to the number of QP doorbells that can be consumed by the network card currently. Among them, the QP doorbell in the network card storage space is preferentially obtained, and the network card performs corresponding actions according to the content of each doorbell information.

[0103] Step (7): The steps (2) to (6) are repeated to transmit subsequent QP doorbell information.

[0104] It should be noted that the determination logic of the number of consumable QP doorbells in any of the above embodiments can refer to the following manner, for example, the driver maintains the producer pointer, and the producer pointer is incremented by one after each new doorbell is issued. The network card maintains the consumer pointer, and the consumer pointer is incremented by one after the network card consumes each doorbell. Assuming that the producer pointer and the consumer pointer are both 0 at initialization, the driver continuously creates or accumulatively creates 10 doorbells, at which time the producer pointer is updated to 10, while the consumer pointer remains 0 at initialization, so the number of consumable QP doorbells is 10 (the producer pointer minus the consumer pointer, that is, 10 minus 0). If the network card obtains a doorbell from the model and uses it, the consumer pointer will be updated to 1, and the number of consumable QP doorbells will be updated to 9 (the producer pointer minus the consumer pointer, that is, 10 minus 1).

[0105] The present disclosure also provides a doorbell control system in an RDMA virtualization scenario, as shown in Figure 2As shown, including the host installed with the network card driver and the host memory and the network card with the RDMA engine, wherein: the network card driver is configured to perform the processing performed by the driver in the doorbell control method as described in any embodiment of the present disclosure, and the network card is configured to perform the processing performed by the network card in the doorbell control method as described in any embodiment of the present disclosure.

[0106] The processor of the above-mentioned embodiments of the present disclosure can be a general processor, including a central processing unit (CPU), a network processor (NP), a microprocessor, etc., and can also be other conventional processors, etc.; the processor can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA), discrete logic or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or other equivalent integrated or discrete logic circuit, and can also be a combination of the above devices. That is, the processor of the above-mentioned embodiments can be any processor or combination of devices that implement the methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure. If the embodiments of the present disclosure are implemented partially in software, instructions for the software can be stored in a suitable non-volatile computer readable storage medium, and one or more processors can be used to execute the instructions in hardware to implement the methods of the embodiments of the present disclosure. The term "processor" used herein can refer to the above structures or any other structure suitable for implementing the techniques described herein.

[0107] An embodiment of the present disclosure also provides a non-transitory computer readable storage medium, the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the doorbell control method in the RDMA virtualization scenario as described in any embodiment of the present disclosure.

[0108] In one or more exemplary embodiments described above, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally can correspond to non-transitory computer-readable storage media or communication media (e.g., a signal or carrier wave). Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product can include a computer-readable medium.

[0109] By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program codes in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. While the disclosed embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Any modification and changes of the disclosed embodiments are contemplated as falling within the scope of the present disclosure. Although the present disclosure has been described in detail with respect to particular embodiments, it should be understood that the various embodiments were merely for the purpose of illustration and explanation and were not intended to limit the scope of the present disclosure in any way.

Claims

1. A doorbell control method for a remote direct data read (RDMA) virtualization scenario, applied to a doorbell control system, the doorbell control system comprising a host and a network card with an RDMA engine, the host comprising a host memory and a network card driver; the method comprising: creating a producer-consumer control model of QP doorbells based on functions, the producer-consumer control model being a ring-shaped storage space; implementing doorbell control in an RDMA network based on the producer-consumer control model, comprising: when the network card driver needs to issue a first function of QP doorbells, reading a consumer pointer of a first producer-consumer control model maintained by the network card, and comparing the consumer pointer with a producer pointer of the first producer-consumer control model maintained by the network card driver; when the network card driver determines that there is space in the first producer-consumer control model to issue a new QP doorbell according to the comparison result, writing doorbell information of the QP doorbell to be issued into the storage space of the first producer-consumer control model, and updating the producer pointer of the first producer-consumer control model; wherein the first producer-consumer control model is a producer-consumer control model of QP doorbells created for the first function; and the network card driver synchronously refreshing the producer pointer of the first producer-consumer control model after function update to a register space corresponding to the network card; after receiving the producer pointer update of the first producer-consumer control model, the network card determines the number of currently consumable QP doorbells according to the updated producer pointer and a consumer pointer of the first producer-consumer control model; the network card acquires a corresponding number of QP doorbell information from the storage space of the first producer-consumer control model according to control information of the first function; and the network card executes corresponding actions according to the content of each acquired QP doorbell information, and updates the consumer pointer of the first producer-consumer control model. 2.The method of claim 1, wherein: the creating of the producer-consumer control model of QP doorbells based on functions comprises: in an initialization stage, the network card driver applies a piece of storage space of the producer-consumer control model of QP doorbells in the host memory for each activated function; the network card driver refreshes control information of the producer-consumer control model of QP doorbells created for each function to a register space corresponding to the network card. 3.The method of claim 1, wherein: the creating of the producer-consumer control model of QP doorbells based on functions comprises: creating a piece of storage space for storing QP doorbell information in the network card, and in an initialization stage, the network card driver applies a piece of storage space of the producer-consumer control model of QP doorbells in the storage space for each activated function; the network card driver refreshes control information of the producer-consumer control model of QP doorbells created for each function to a register space corresponding to the network card. 4.The method of claim 2 or 3, wherein: the function comprises a PCIe function. The control information of the producer-consumer control model includes an address of a storage space of the producer-consumer control model and a size of the storage space.

5. The method of claim 1, wherein: The producer-consumer control model of the QP doorbell is created based on functions, including: In the initialization phase, the network card driver applies a piece of storage space of the producer-consumer memory control model of the QP doorbell for each activated function in the host memory, and the network card driver flushes the control information of the producer-consumer memory control model of the QP doorbell created for each function to the corresponding register space of the network card; In the initialization phase, the network card driver applies a piece of storage space of the producer-consumer memory control model of the QP doorbell for each activated function in the host memory, and the network card driver flushes the control information of the producer-consumer memory control model of the QP doorbell created for each function to the corresponding register space of the network card; 6. The method of claim 5, wherein: The doorbell control in the RDMA network is implemented based on the producer-consumer control model, including: When the network card driver needs to issue a QP doorbell of a first function, if it is determined that there is unconsumed QP doorbell information in the first producer-consumer memory control model or the unconsumed QP doorbell information in the first producer-consumer network card control model reaches the upper limit of the storage space, the network card driver writes the doorbell information of the QP doorbell to be issued into the storage space of the first producer-consumer memory control model, and updates the producer pointer of the first producer-consumer memory control model.

7. The method of claim 5, wherein: The doorbell control in the RDMA network is implemented based on the producer-consumer control model, including: When the network card driver needs to issue a QP doorbell of a first function, if it is determined that there is unconsumed QP doorbell information in the first producer-consumer memory control model or the unconsumed QP doorbell information in the first producer-consumer network card control model reaches the upper limit of the storage space, the network card driver writes the doorbell information of the QP doorbell to be issued into the storage space of the first producer-consumer memory control model, and updates the producer pointer of the first producer-consumer memory control model.

8. The method of claim 5, wherein: The function includes a PCIe function. ​ ​ The control information of the producer-consumer memory control model comprises an address of a storage space of the producer-consumer memory control model and a size of the storage space. The control information of the producer-consumer network card control model comprises an address of a storage space of the producer-consumer network card control model and a size of the storage space.

9. A doorbell control system for an RDMA virtualization scenario, the system comprising: The computer program is executed by a processor to implement the doorbell control method of the RDMA virtualization scenario as claimed in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing a computer program, wherein, The computer program is executed by a processor to implement the doorbell control method of the RDMA virtualization scenario as claimed in any one of claims 1 to 8.

Images