Apparatus and method for remote direct memory access

Abstract

The present invention relates to an apparatus and method for RDMA. In particular, the present invention provides an entity for RDMA. The entity is configured to maintain a time-based QP, wherein the time-based QP comprises a first area associated with a time-based indication, wherein the first area is configured to hold one or more first WQEs, and wherein the time-based indication is configured to indicate that the one or more WQEs in the first area are to be processed periodically; and periodically process the one or more first WQEs in the first area in accordance with the time-based indication.

Description

Technical Field

[0001] This invention relates to high-performance computing technology, and more particularly to remote direct memory access (RDMA) technology. Specifically, this invention relates to transmitting RDMA transactions over packet-based networks. To this end, the invention provides an apparatus, method, and packet format for RDMA. Background Technology

[0002] RDMA is a technology widely used in modern data centers and computer clusters to achieve low-latency and high-bandwidth networking. RDMA offloads memory operations from the central processing unit (CPU) to the RDMA network interface card (NIC) (RDMA NIC, RNIC), which can directly access memory. This offload saves CPU time, allowing it to perform other tasks.

[0003] The software layer of RDMA uses verbs to execute RDMA operations, which are then translated into work requests written to the RNIC queue. Such work requests are called work queue elements (WQEs). RDMA operations are either tagged (one-sided write / read / atomic) or untagged (two-sided send / receive). RDMA peers communicate through queue pairs (QPs) that provide various transport services. Common QP types (published by the IB specification) include reliable connection (RC) QP, reliable datagram (RD) QP, extended reliable connection (XRC) QP, unreliable datagram (UD) QP, unreliable connection (UC) QP, and dynamic connection (DC) QP, etc.

[0004] Many applications or tasks require periodic messages with fixed content, such as heartbeat / keep-alive messages, messages reporting machine load or storage usage status, messages distributing configuration changes, or messages reporting round-trip time (RTT) measurements (when there is no active traffic). Currently, RDMA technology does not support automatic reuse of WQEs for any QP type. That is, software needs to reissue a WQE after the previous one has been processed. Therefore, applications and higher-level protocols use software-based schemes to report periodic messages in the current RDMA application. However, this scheme is more complex, less efficient, less accurate, and consumes more CPU time. Summary of the Invention

[0005] In view of this, embodiments of the present invention aim to provide an apparatus and method for RDMA. The objective is to create a novel time-scheduled RDMA QP behavior, particularly supporting the transmission of pre-configured WQEs on each rescheduling event. One goal is to support the repeated reuse of WQEs without software intervention.

[0006] These or other objectives can be achieved by the embodiments of the invention described in the appended independent claims. Advantageous implementations of the embodiments of the invention are further defined in the dependent claims.

[0007] A first aspect of the present invention provides an entity for RDMA. The entity is configured to: maintain a time-based QP, wherein the time-based QP includes a first region associated with a time-based indication, wherein the first region is configured to store one or more first WQEs, the time-based indication indicating that the one or more WQEs in the first region will be processed periodically; and process the one or more first WQEs in the first region periodically according to the time-based indication.

[0008] This invention provides a scheme for an entity of a first aspect to periodically schedule WQEs without involving software. This scheme allows entities to repeatedly reuse WQEs in time-based QPs without requiring upper-layer applications / software to redistribute WQEs. It should be noted that WQE and QP are terms in the RDMA specification. A WQE is an RDMA operation or transaction that is pushed into a QP, transmitted to a peer, or received from a peer. A QP represents a communication endpoint, which consists of a send queue (SQ) and a receive queue (RQ), or alternatively, a shared RQ. The time-based QP defined in this invention can also be referred to as a periodically-auto-scheduled (PAS) QP.

[0009] In one implementation, the time-based indication includes at least one of the following indications:

[0010] - The periodic interval T used to process the one or more WQEs in the first region;

[0011] - The number of cycles N used to process the one or more WQEs in the first region, where N is a positive integer;

[0012] - The duration of periodically processing one or more WQEs in the first region;

[0013] - Before receiving a stop command, continuously issue one or more WQEs in the first region.

[0014] Specifically, the number of times this time-based QP needs to be rescheduled can be finite or infinite (e.g., until the QP is removed or instructed to stop).

[0015] In another implementation, the time-based QP types include one of the following: RC QP, RD QP, XRC QP, UD QP, UC QP, and DC QP.

[0016] It should be noted that the QP types mentioned above are merely examples. The novel method described in this invention is also applicable to other existing QP types.

[0017] In another implementation, the one or more first WQEs include at least one of SQ WQE, RQ WQE, and SRQ WQE.

[0018] In another implementation, the one or more first WQEs are associated with at least one of a first destination address and a second destination address, wherein the entity is used to periodically process the one or more first WQEs to at least one of a first destination entity corresponding to the first destination address and a second destination entity corresponding to the second destination address.

[0019] Periodic QP can be used for a single destination or multiple destinations.

[0020] In another implementation, the entity is also used to maintain a completion queue (CQ), wherein the first region is also associated with a completion indicator, which indicates how completion is generated.

[0021] Traditionally, RDMA QP always generates completion (RQ processing, etc.) on some opcode. There is no control to operate conditional completion. According to embodiments of the present invention, completion can be configurable.

[0022] In another implementation, the completion indication includes at least one of the following indications:

[0023] - Generation completes after a predefined number of processed first WQEs;

[0024] - Generation is completed after a predefined number of cycles;

[0025] - Generation completes when an error or a predefined type of error occurs;

[0026] - In response to notification completion;

[0027] - Do not generate complete.

[0028] In another implementation, the entity is also configured to stop processing the one or more first WQEs in the first region when an error occurs.

[0029] In another implementation, the time-based QP includes a second region, wherein the second region includes one or more second WQEs, wherein the one or more second WQEs are processed after the ringing.

[0030] It's possible that a time-based QP might also have a section for periodic WQE. That is, some WQEs within the QP can be periodic, while the rest can be periodic. Only periodic WQEs are processed after the bell rings.

[0031] In another implementation, the entity is further configured to: add one or more third WQEs to the first region; and periodically process the one or more third WQEs in the next cycle.

[0032] Optionally, the SQ and RQ of a time-based QP are dynamic. This means that when a WQE is issued, it is not necessary to pause the QP scheduling, expand the SQ / RQ, and then resume it. According to the invention, in the next cycle, the QP will use the newly issued WQE for scheduling.

[0033] In another implementation, the entity is also used to obtain time-based QPs from at least one of the first and second applications.

[0034] Optionally, a single PAS QP can be shared by multiple applications (multiplexing).

[0035] In another implementation, the entity is further configured to: maintain a memory region for storing specific data for the entity; monitor the memory region and determine whether the specific data has changed; and when it is determined that the specific data has changed, provide the content of the specific data to one or more other entities for RDMA using one or more WQEs of the time-based QP.

[0036] The method provided in this invention can be used to distribute changes in memory, such as configuration changes.

[0037] A second aspect of the invention provides a method for RDMA. The method includes: maintaining a time-based QP, wherein the time-based QP includes a first region associated with a time-based indication, wherein the first region is used to store one or more first WQEs, the time-based indication indicating that the one or more WQEs in the first region will be processed periodically; and processing the one or more first WQEs in the first region periodically according to the time-based indication.

[0038] In another implementation, the time-based indication includes at least one of the following indications:

[0039] - The periodic interval T used to process the one or more WQEs in the first region;

[0040] - The number of cycles N used to process the one or more WQEs in the first region, where N is a positive integer;

[0041] - The duration of periodically processing one or more WQEs in the first region;

[0042] - Before receiving a stop command, continuously issue one or more WQEs in the first region.

[0043] In another implementation, the time-based QP type includes one of the following: reliable connection QP, reliable datagram QP, extended reliable connection QP, unreliable datagram QP, unreliable connection QP, and dynamic connection QP.

[0044] In another implementation, the one or more first WQEs include at least one of a transmit queue WQE, a receive queue WQE, and a shared receive queue WQE.

[0045] In another implementation, the one or more first WQEs are associated with at least one of a first destination address and a second destination address, wherein the entity is used to periodically process the one or more first WQEs to at least one of a first destination entity corresponding to the first destination address and a second destination entity corresponding to the second destination address.

[0046] In another implementation, the method further includes maintaining a completion queue, wherein the first region is also associated with a completion indication, wherein the completion indication is used to indicate the generation method of completion.

[0047] In another implementation, the completion indication includes at least one of the following indications:

[0048] - Generation completes after a predefined number of processed first WQEs;

[0049] - Generation is completed after a predefined number of cycles;

[0050] - Generation completes when an error or a predefined type of error occurs;

[0051] - In response to notification completion;

[0052] - Do not generate complete.

[0053] In another implementation, the method further includes stopping processing of the one or more first WQEs in the first region when an error occurs.

[0054] In another implementation, the time-based QP includes a second region, wherein the second region includes one or more second WQEs, wherein the one or more second WQEs are processed after the ringing.

[0055] In another implementation, the method further includes: adding one or more third WQEs in the first region; and periodically processing the one or more third WQEs in the next cycle.

[0056] In another implementation, the method further includes obtaining a time-based QP from at least one of the first and second applications.

[0057] In another implementation, the method further includes: maintaining a memory region for storing specific data for the entity; monitoring the memory region and determining whether the specific data has changed; and when it is determined that the specific data has changed, providing the content of the specific data to one or more other entities for RDMA using one or more WQEs of the time-based QP.

[0058] The second aspect and its implementation methods offer the same advantages and effects as those described above for the entities relating to the first aspect and its corresponding implementation methods.

[0059] A third aspect of the invention provides a computer program including program code, which, when implemented on a processor, is used to perform the method according to the second aspect or any implementation thereof.

[0060] A fourth aspect of the present invention provides a computer-readable storage medium comprising computer-executable computer program code instructions, which, when executed on a computer, are used to perform a method according to the second aspect or any implementation thereof.

[0061] It should be noted that all devices, elements, units, and modules described in this application can be implemented by software or hardware elements or any combination thereof. All steps performed by the various entities described in this application and the functions described to be performed by the various entities are intended to indicate that the respective entities are used to perform the corresponding steps and functions. Although the specific functions or steps performed by external entities are not reflected in the detailed description of the specific elements of the entities performing the specific steps or functions in the following description of specific embodiments, those skilled in the art will understand that these methods and functions can be implemented by corresponding hardware or software elements or any combination thereof. Attached Figure Description

[0062] The above aspects and implementation methods are illustrated in the following detailed description of specific embodiments, with reference to the accompanying drawings.

[0063] Figure 1 An entity according to an embodiment of the present invention is shown.

[0064] Figure 2 A schematic diagram according to an embodiment of the present invention is shown.

[0065] Figure 3 A signaling flowchart at the transmitting node according to an embodiment of the present invention is shown.

[0066] Figure 4 A signaling flowchart at the receiving node according to an embodiment of the present invention is shown.

[0067] Figure 5 The following illustration shows the exchange of data packets between multiple senders and receivers according to an embodiment of the present invention.

[0068] Figure 6 The exchange of data packets between multiple destinations according to an embodiment of the present invention is illustrated.

[0069] Figure 7 A signaling flowchart at a load balancer node according to an embodiment of the present invention is shown.

[0070] Figure 8 A signaling flowchart at a load balancer node according to an embodiment of the present invention is shown.

[0071] Figure 9 A signaling flowchart at a server node according to an embodiment of the present invention is shown.

[0072] Figure 10 The diagram illustrates packet exchange between a load balancer node and a server node according to an embodiment of the present invention.

[0073] Figure 11 The diagram illustrates packet exchange between a load balancer node and multiple server nodes according to an embodiment of the present invention.

[0074] Figure 12 A signaling flowchart for distributing changes according to an embodiment of the present invention is shown.

[0075] Figure 13 A method according to an embodiment of the present invention is shown. Detailed Implementation

[0076] Illustrative embodiments of methods, apparatus, and program products for time-based periodic QP scheduling in communication systems are described with reference to the accompanying drawings. While this description provides detailed examples of possible implementations, it should be noted that these details are intended to be exemplary and do not limit the scope of this application.

[0077] Furthermore, one embodiment / example may refer to multiple other embodiments / examples. For example, any descriptions mentioned in one embodiment / example, including but not limited to terms, elements, processes, explanations, and / or technical advantages, are applicable to multiple other embodiments / examples.

[0078] Figure 1An entity 100 for RDMA according to an embodiment of the present invention is illustrated. Entity 100 may include processing circuitry (not shown) for performing, implementing, or initiating various operations of entity 100 as described herein. The processing circuitry may include hardware and software. The hardware may include analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may include components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. Entity 100 may also include memory circuitry storing one or more instructions that can be executed by a processor or processing circuitry (specifically, under the control of software). For example, the memory circuitry may include a non-transitory storage medium storing executable software code that, when executed by the processor or processing circuitry, causes entity 100 to perform various operations. In one embodiment, the processing circuitry includes one or more processors and a non-transitory memory connected to one or more processors. Non-transient memory can store executable program code that, when executed by one or more processors, causes entity 100 to perform, conduct, or initiate the operations or methods described herein.

[0079] Specifically, entity 100 is used to maintain a time-based QP 101, wherein the time-based QP 101 includes a first region 102 associated with a time-based indication. Specifically, the first region 102 is used to store one or more first WQEs 103, 103', and the time-based indication is used to indicate that one or more WQEs in the first region 102 will be processed periodically. Entity 100 is also used to periodically process one or more first WQEs 103, 103' in the first region 102 according to the time-based indication.

[0080] This invention introduces a novel method in which any RDMAQP type can be a periodically scheduled WQE, SQ WQE, or RQ WQE. WQEs in time-based QPs are reused repeatedly, and no parameters (e.g., producer index (PI) or consumer index (CI)) need to be manipulated to redistribute the WQE.

[0081] For example, the time-based QP 101 type can be one of the following: RC QP, RD QP, XRC QP, UD QP, UCQP, and DC QP. It should be noted that the QP types listed above are merely examples. The novel method described in this invention is also applicable to any other QP type. That is, according to embodiments of the invention, data packets transmitted over the network based on this new QP behavior can be identical to any existing QP type (RC / UD / RD / XRC / UC / DC, etc.) without requiring changes to the network data packets.

[0082] Rescheduling of the same WQE can be limited to a certain number of iterations or can be unlimited. According to embodiments of the present invention, time-based indications include at least one of the following indications:

[0083] - The periodic interval T used to process the one or more WQEs in the first region 102;

[0084] - The number of cycles N used to process the one or more WQEs in the first region 102, where N is a positive integer;

[0085] - The duration of periodically processing the one or more WQEs in the first region 102;

[0086] - Before receiving a stop command, continuously publish one or more WQEs in the first region 102.

[0087] According to embodiments of the present invention, one or more first WQEs 103, 103' include at least one of SQ WQE, RQ WQE, and shared RQ (SRQ) WQE. New QP behaviors can use either RQ or SRQ. WQEs on RQ / SRQ are generally referred to as RQ elements (RQEs). Multiple RQEs can be published to RQs that will be reused.

[0088] According to an embodiment of the present invention, one or more first WQEs 103, 103' may be associated with at least one of a first destination address and a second destination address, wherein entity 100 is used to periodically process one or more first WQEs 103, 103' to at least one of a first destination entity 100 corresponding to the first destination address and a second destination entity 100 corresponding to the second destination address.

[0089] Alternatively, for example, when multiple destinations are specified for the message or the QP (when one or more first WQE 103, 103' are associated with a first destination address and a second destination address), one or more first WQE 103, 103' may be processed more than once in each rescheduling event.

[0090] According to an embodiment of the present invention, Figure 1 The entity 100 shown can also be used to maintain CQ, wherein the first region 102 is also associated with a completion indicator, wherein the completion indicator is used to indicate the generation method of completion.

[0091] Optionally, the completion instruction may include at least one of the following:

[0092] - Generation is completed after a predefined number of processed first WQE 103, 103';

[0093] - Generation is completed after a predefined number of cycles;

[0094] - Generation completes when an error or a predefined type of error occurs;

[0095] - In response to notification completion;

[0096] - Do not generate complete.

[0097] Traditionally, RDMA QPs always generate completion on some opcode (RQ processing, etc.). There is no control to manipulate conditional completion. According to the present invention, completion can be generated according to configuration, for example, based on WQE, based on the number of rescheduled events, when the PI pointer of the RQ is reached, only in case of error, etc.

[0098] Optionally, if periodic completion events are necessary for a specific QP type, these events are discarded. Completed generation can be scheduled continuously, not scheduled at all, or rescheduled when an error occurs, allowing the application to handle reported errors, such as deleting a WQE, removing a QP, etc.

[0099] According to an embodiment of the present invention, entity 100 can be used to stop processing one or more first WQEs 103, 103' in the first region 102 when an error occurs.

[0100] Optionally, the time-based QP 101 may also include a second region, wherein the second region includes one or more second WQEs, wherein the one or more second WQEs are processed after the ringing. It should be noted that the same QP (PAS QP) may have a special region for periodic WQEs, or it may have a region for regular WQEs (regular WQEs are processed after the ringing as defined in the existing RDMA specification).

[0101] According to an embodiment of the present invention, entity 100 can be used to add one or more third WQEs in the first region 102 and process one or more third WQEs periodically in the next cycle.

[0102] Optionally, the SQ / RQ of a PAS QP is dynamic, meaning that when a WQE is released, there is no need to pause the QP's scheduling, expand the SQ / RQ, and then resume it. Instead, a new periodic WQE can be added to the first region, and then the QP will schedule the new periodic WQE along with the existing WQE in the next (and subsequent) rescheduling event.

[0103] Optionally, a single PAS QP can be shared by multiple applications (multiplexing). According to an embodiment of the present invention, entity 100 can be used to obtain a time-based QP 101 from at least one of the first and second applications.

[0104] Figure 2 A schematic diagram of an RNIC according to an embodiment of the present invention is shown. The PAS QP includes SQ, RQ / SRQ, and CQ. Any single WQE included in this QP can include multiple destinations in each (re)scheduling event.

[0105] The "tail" of a queue (SQ or RQ / SRQ) can be indicated using the parameter PI, and the "head" of the queue can be indicated using the parameter CI. It's important to note that when PI ≠ CI, it means there are one or more WQEs in the queue that should be processed; when PI = CI, it means the work has been completed. The "tail" is only updated when an additional WQE is published. The "head" is only updated when a previously published WQE is deleted or a new WQE is added.

[0106] The timer in the RNIC can be set to a periodic interval T for processing one or more WQEs in the PAS QP. Each time the timer expires, the RNIC processes all WQEs between the "header" and "tail" of the SQ. According to an embodiment of the invention, the data packets to the network are formatted according to existing QP types.

[0107] It should be noted that RDMA operations include read, write, send, receive, and atomic operations. RQ / SRQ is only needed when an immediate write or send operation is required. CQ is needed when generation is complete or an error occurs.

[0108] An RNIC can be either a requester RNIC or a responder RNIC. Specifically, the requester RNIC processes all WQEs published to the SQ in each rescheduled event. Each time a send or immediate write request is received, the responder RNIC uses the next WQE published to the RQ / SRQ. After reaching the PI pointer of the RQ, it continues in a loop from the RQE pointed to by the CI pointer.

[0109] like Figure 1 The entity 100 shown can be either the requester RNIC or the responder RNIC.

[0110] As mentioned above, many applications or tasks require periodic messages with fixed content. The new PAS QP, as defined in this embodiment of the invention, is particularly advantageous when applied to these use cases. Specific embodiments based on different use cases are described in detail below.

[0111] A "heartbeat" is a periodic signal generated by hardware or software to indicate normal operation or to synchronize other parts of a computer system. Heartbeat messages are typically exchanged between endpoints (sending and receiving nodes) at regular intervals on the order of seconds or milliseconds. If a receiving node does not receive a heartbeat for a period of time (usually several heartbeat intervals), it is assumed that the sending node has failed.

[0112] Heartbeat messages can also be used for high availability and fault tolerance purposes.

[0113] From the start-up of the endpoint (sending node) to its shutdown, heartbeat messages are typically sent in a continuous, cyclical, and periodic manner. When the destination endpoint (receiving node) identifies a lost heartbeat message during the expected arrival period, it can be determined that the initiating endpoint (sending node) has failed, is shut down, or is generally no longer available.

[0114] Figure 3 A signaling flowchart at a transmitting node according to an embodiment of the present invention is shown. Here, the transmitting node can be an endpoint that needs to send a "heartbeat" message. It should be noted that the transmitting node can also be any other device that needs to send periodic signals.

[0115] In this embodiment, the application creates a time-based QP and requests the QP to send a data packet P to destination D at intervals T. Here, data packet P can be a heartbeat signal. Accordingly, the NIC starts periodically sending data packet P to destination D according to a timer. That is, the NIC starts a timer, and in each cycle, after the interval T expires, it periodically sends data packet P to destination D. This will continue indefinitely until it is stopped (e.g., sender failure).

[0116] Figure 4 A signaling flowchart at a receiving node according to an embodiment of the present invention is shown. Here, the receiving node can be the destination endpoint receiving a "heartbeat" message. It should be noted that the receiving node can also be any other device that needs to receive periodic signals.

[0117] In this embodiment, the application creates a time-based QP and requests the QP to receive data packet P from destination D at interval T. Here, data packet P can be a heartbeat signal. Optionally, failure to receive data packet P from destination D within interval T may activate callback C. Therefore, the NIC waits for data packet P from destination D unless it receives a request from the application to delete the QP or delete destination D. Optionally, if data packet P from destination D does not arrive within interval T, the NIC can activate callback C. Depending on the QP configuration, the NIC will decide whether to continue waiting for data packet P or wait for the application to request the NIC QP to wait again for data packet P from destination D within interval T.

[0118] Figure 5 This illustration depicts packet exchange between multiple sending nodes and one receiving node according to an embodiment of the present invention. Sending nodes D1 and Dn can be... Figure 3 The sending node is shown. The receiving node can be... Figure 4 The receiving node is shown.

[0119] Each of the sending nodes D1...Dn periodically sends data packet P to the destination D (i.e., the receiving node). The receiving node's QP is requested to receive data packet P from the destination (sending node) D1...Dn within interval T. It should be noted that in step 7, data packet P does not arrive at the receiving node from sending node D1 within interval T. Therefore, the NIC at the receiving node activates callback C in the application on sending node D1. Based on the QP configuration, the NIC will decide whether to continue waiting for the next data packet P from sending node D1.

[0120] Figure 6 This illustration depicts packet exchange between multiple destinations according to an embodiment of the present invention. Each destination can be a sending node, a receiving node, or a sending-receiving node. Figure 6 In the example shown, each of nodes D1, D2, and D3 is a sending and receiving node.

[0121] For a specific NIC, a timer can be configured for the transmission interval St for sending data packet P and / or the reception interval Rt for receiving data packet P. Taking node D1 as an example, node D1's periodic QP is requested to send data packet P to nodes D2 and D3 at interval St, and is requested to receive data packet P from nodes D2 and D3 at interval Rt. Failure to receive data packet P from node D2 or D3 within interval T will activate callback C. It should be noted that nodes D2 and D3 can be configured in a similar manner.

[0122] Load balancing is the process of distributing a set of tasks across a set of resources (computing units) to improve overall processing efficiency. Load balancing techniques can optimize the response time of each task, thereby preventing uneven overload of computing units while other computing units remain idle.

[0123] There are two main methods for load balancing. One is called a static algorithm, which does not consider the state of different computing units. The other is called a dynamic algorithm, which is more general and efficient, but requires information exchange between different computing units, and risks lower efficiency.

[0124] Load balancers use various scheduling algorithms (also known as load balancing methods) to determine which compute unit to send a task to. Simple algorithms include random selection, round-robin, and least connections. More complex load balancers may consider additional factors such as the reported load of the compute unit, minimum response time, operational / shutdown status (determined by some form of monitoring polling), number of active connections, geographic location, functionality, and network traffic sent or received.

[0125] Figure 7 A signaling flowchart at a load balancer node according to an embodiment of the present invention is shown. In this embodiment, the application creates a time-based QP and requests the QP to issue RDMA read operations and load reports from destination D at intervals T. Accordingly, the NIC starts periodically issuing RDMA reads and load reports from destination D according to a timer of interval T. That is, the NIC starts a timer, and in each cycle, after the interval T expires, issues RDMA reads and load reports from destination D. This will continue indefinitely until the application stops, the QP is terminated, the timer (interval T) is changed, or the application is terminated.

[0126] Figure 8 A signaling flowchart at a load balancer node according to an embodiment of the present invention is shown. In this embodiment, the application creates a time-based QP and requests the QP to issue RDMA read operations and load reports from multiple destinations (i.e., D, E, and F) at intervals T. Accordingly, the NIC starts periodically issuing RDMA reads and load reports from each of destinations D, E, and F according to a timer of interval T. That is, the NIC starts a timer, and in each period, after the interval T expires, issues RDMA reads and load reports from each of destinations D, E, and F. Figure 7 Similar to the embodiments shown, this process will continue indefinitely until the application stops, the QP is terminated, the timer (interval T) is changed, or the application is terminated.

[0127] Figure 9A signaling flowchart at a server node according to an embodiment of the present invention is shown. In this embodiment, the application creates a periodic QP and requests a memory region that stores the server's load information and can be read by the load balancer. The application updates the load information locally from time to time. Incoming RDMA read requests from the load balancer node will read directly from the application's memory and return the server's load information.

[0128] Figure 10 The diagram illustrates packet exchange between a load balancer node and a server node according to an embodiment of the present invention. Figure 10 The load balancer node shown can be Figure 7 The load balancer node shown. Figure 10 The server node shown can be Figure 9 The server node shown.

[0129] Figure 11 The diagram illustrates packet exchange between a load balancer node and multiple server nodes according to an embodiment of the present invention. Figure 10 The load balancer node shown can be Figure 7 The load balancer node shown. Figure 10 Each of the server nodes D, E, and F shown can be Figure 9 The server node shown.

[0130] An application on a load balancer node creates a time-based Query Point (QP) and requests the QP to issue RDMA read operations and load reports from each of server nodes D, E, and F at intervals T. Accordingly, the NIC starts periodically issuing RDMA reads and load reports from destination D according to a timer at interval T. That is, the NIC starts a timer that, in each cycle, after interval T expires, issues RDMA reads and load reports from each of servers D, E, and F. This will continue indefinitely until the application stops, the QP is terminated, the timer (interval T) is changed, or the application is terminated.

[0131] Because configuration changes need to be distributed to all nodes in the cluster, configuration changes are applied to clusters with multiple nodes. Configuration changes can include node failures, new nodes joining the cluster, or changes to global parameters.

[0132] Configuration changes are typically managed by the cluster controller. When a node in the cluster receives a configuration change, it will take action based on the specific changes requested.

[0133] Distributing configuration changes requires significant CPU time. Within the cluster controller, this necessitates complex software coding and deployment, involving reliable network protocols. Offloading this process to another entity (such as a smart NIC) can drastically reduce the burden on application developers.

[0134] According to embodiments of the present invention, entity 100 can also be used to maintain a memory region for storing specific data for entity 100. Entity 100 can monitor the memory region and determine whether the specific data has changed. When it is determined that the specific data has changed, entity 100 can be used to provide the content of the specific data to one or more other entities used for RDMA using one or more WQEs based on time-based QP 101.

[0135] Figure 12 A signaling flowchart illustrating a change in the distribution of data by a sending node according to an embodiment of the present invention is shown. In this embodiment, the application of sending node D1 creates a time-based Request for Payment (QP) and requests the QP to send data packets to each of the receiving nodes D2, D3...Dn at intervals T. The NIC continuously checks whether some contents in memory (e.g., addresses A1 to An) have changed. The contents may include the configuration or parameters of the receiving nodes. If the contents or part of the contents have changed, the NIC will retrieve the entire contents in memory and send them to each of the receiving nodes D2, D3...Dn. This process is repeated until instructed to stop.

[0136] It should be noted that any data modification compared to the previous cycle will trigger transmission (i.e., altered distribution). If the data remains unchanged, it will enter a sleep state until the next cycle. This invention's novel method can be used to distribute configuration data, from one transmitter to one receiver, or from one transmitter to multiple receivers.

[0137] In summary, this invention provides a novel time-based scheduling-based RDMA QP behavior. Specifically, pre-configured WQEs in the QP are sent in each rescheduling event. WQEs in the QP are reused without needing to be republished. It should be noted that the number of times this QP needs to be rescheduled can be finite or infinite (i.e., until the QP is removed). Any existing type of QP can be used to implement this new QP behavior without requiring changes to network packets. Periodic QPs can be used for a single destination or multiple destinations, and different application messages can be multiplexed on the same QP. Messages can be unicast or multicast, depending on the capabilities of the QP type (only UD QPs can multicast).

[0138] It is worth noting that common existing periodic message programming models involve the main CPU executing this logic in upper-level applications, which makes it more complex, less efficient, less accurate, and consumes more CPU time. According to embodiments of the present invention, even non-RDMA applications can use this QP behavior to transfer periodic messages to hardware without software intervention.

[0139] Figure 13 A method 1300 for RDMA according to an embodiment of the present invention is illustrated. In one specific embodiment of the invention, method 1300 is performed by… Figure 1 The entity 100 shown is executed. Method 1300 includes step 1301: maintaining a time-based QP 101. Specifically, the time-based QP 101 includes a first region 102 associated with a time-based indication. The first region 102 is used to store one or more first WQEs 103, 103', and the time-based indication is used to indicate that one or more WQEs in the first region 102 will be processed periodically. Method 1300 also includes step 1302: periodically processing one or more first WQEs 103, 103' in the first region 102 according to the time-based indication.

[0140] Optionally, the time-based indication may include at least one of the following indications:

[0141] - The periodic interval T used to process the one or more WQEs in the first region 102;

[0142] - The number of cycles N used to process the one or more WQEs in the first region 102, where N is a positive integer;

[0143] - The duration of periodically processing the one or more WQEs in the first region 102;

[0144] - Before receiving a stop command, continuously publish one or more WQEs in the first region 102.

[0145] Optionally, the one or more first WQEs 103, 103' are associated with at least one of a first destination address and a second destination address, wherein the method further includes periodically processing the one or more first WQEs 103, 103' to at least one of a first destination entity 100 corresponding to the first destination address and a second destination entity 100 corresponding to the second destination address.

[0146] Optionally, the method 1300 further includes maintaining a CQ, wherein the first region 102 is also associated with a completion indication, the completion indication being used to indicate the generation method of completion.

[0147] Optionally, the completion instruction may include at least one of the following:

[0148] - Generation is completed after a predefined number of processed first WQE 103, 103';

[0149] - Generation is completed after a predefined number of cycles;

[0150] - Generation completes when an error or a predefined type of error occurs;

[0151] - In response to notification completion;

[0152] - Do not generate complete.

[0153] Optionally, method 1300 further includes stopping processing of the one or more first WQEs 103, 103' in the first region 102 when an error occurs.

[0154] Optionally, method 1300 further includes adding one or more third WQEs in the first region 102 and periodically processing one or more third WQEs in the next cycle.

[0155] Optionally, method 1300 further includes obtaining a time-based QP 101 from at least one of the first application and the second application.

[0156] Optionally, method 1300 further includes: maintaining a memory region for storing specific data for entity 100; monitoring the memory region and determining whether the specific data has changed; and when it is determined that the specific data has changed, providing the content of the specific data to one or more other entities for RDMA using one or more WQEs of the time-based QP 101.

[0157] The invention has been described in conjunction with various embodiments and implementations as examples. However, based on a study of the drawings, the invention, and the independent claims, those skilled in the art will be able to understand and implement other variations in practicing the claimed invention. In the claims and the description, the word "comprising" does not exclude other elements or steps, and "a" does not exclude a plurality. A single element or other unit may fulfill the function of several entities or items listed in the claims. Listing measures in dissimilar dependent claims does not imply that combinations of these measures cannot be used in advantageous implementations.

[0158] Furthermore, any method provided in the embodiments of the present invention can be implemented in a computer program having an encoding / decoding module. When a processing device runs the computer program, the computer program causes the processing device to perform the method steps. The computer program is included in a computer-readable medium of the computer program product. The computer-readable medium can substantially include any memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable PROM (EPROM), flash memory, electrically erasable PROM (EEPROM), or hard disk drive.

[0159] Furthermore, those skilled in the art will recognize that embodiments of entity 100 include the necessary communication capabilities in the form of functions, modules, units, elements, etc., for executing the scheme. Other examples of such modules, units, elements, and functions include: processors, memories, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selection units, switches, interleavers, deinterleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiving units, transmitting units, DSPs, trellis-coded modulation (TCM) encoders, TCM decoders, power supply units, power feeders, communication interfaces, communication protocols, etc., which are suitably arranged together to execute the scheme.

[0160] Specifically, the processor of entity 100 may include one or more instances of a central processing unit (CPU), processing unit, processing circuitry, processor, application-specific integrated circuit (ASIC), microprocessor, or other processing logic capable of interpreting and executing instructions. Therefore, the term "processor" can refer to processing circuitry comprising multiple processing circuits, such as any, some, or all of the aforementioned processing circuits. The processing circuitry may also perform data processing functions for inputting, outputting, and processing data, including data buffering and device control functions such as call processing control, user interface control, etc.

Claims

1. An entity (100) for Remote Direct Memory Access (RDMA), characterized in that, The entity (100) is used for: Maintain a time-based queue pair QP (101), wherein the time-based QP (101) includes a first area (102) associated with a time-based indication, wherein the first area (102) is used to store one or more first work queue elements WQE (103, 103'), and the time-based indication is used to indicate that the one or more WQE in the first area (102) will be processed periodically; According to the time-based instruction, the one or more first WQEs (103, 103') in the first region (102) are processed periodically.

2. The entity (100) according to claim 1, characterized in that, The time-based indication includes at least one of the following: The periodic interval T used to process the one or more WQEs in the first region (102); The number of cycles N used to process the one or more WQEs in the first region (102), where N is a positive integer; The duration of processing one or more WQEs in the first region (102) periodically; Before receiving a stop command, continuously publish one or more WQEs in the first region (102).

3. The entity (100) according to claim 1, characterized in that, The time-based QP (101) type includes one of the following: reliable connection QP, reliable datagram QP, extended reliable connection QP, unreliable datagram QP, unreliable connection QP, and dynamic connection QP.

4. The entity (100) according to any one of claims 1 to 3, characterized in that, The one or more first WQEs (103, 103') include at least one of a transmit queue WQE, a receive queue WQE, and a shared receive queue WQE.

5. The entity (100) according to any one of claims 1 to 3, characterized in that, The one or more first WQEs (103, 103') are associated with at least one of a first destination address and a second destination address, wherein the entity (100) is used to periodically process the one or more first WQEs (103, 103') to at least one of a first destination entity (100) corresponding to the first destination address and a second destination entity (100) corresponding to the second destination address.

6. The entity (100) according to any one of claims 1 to 3, characterized in that, Used for: Maintain a completion queue, wherein the first region (102) is also associated with a completion indication, wherein the completion indication is used to indicate the generation method of completion.

7. The entity (100) according to claim 6, characterized in that, The completion instruction includes any one of the following: Generation is completed after a predefined number of processed first WQEs (103, 103'); The generation is completed after a predefined number of cycles; Generation completes if an error or a predefined type of error occurs; In response to the completion of notification generation; The process is not complete.

8. The entity (100) according to any one of claims 1 to 3, characterized in that, Used for: When an error occurs, stop processing of the one or more first WQEs (103, 103') in the first region (102).

9. The entity (100) according to any one of claims 1 to 3, characterized in that, The time-based QP (101) includes a second region, wherein the second region includes one or more second WQEs, wherein the one or more second WQEs are processed after the ringing.

10. The entity (100) according to any one of claims 1 to 3, characterized in that, Used for: Add one or more third WQEs to the first region (102); The one or more third WQEs are processed periodically in the next cycle.

11. The entity (100) according to any one of claims 1 to 3, characterized in that, Used for: The time-based QP (101) is obtained from at least one of the first and second applications.

12. The entity (100) according to any one of claims 1 to 3, characterized in that, Used for: Maintain a memory region for storing specific data for the entity (100); Monitor the memory region and determine whether the specific data has changed; When it is determined that the specific data has changed, the content of the specific data is provided to one or more other entities for RDMA using one or more WQEs based on the time-based QP (101).

13. A method for Remote Direct Memory Access (RDMA), characterized in that, The method includes: Maintain a time-based queue pair QP, wherein the time-based QP (101) includes a first area (102) associated with a time-based indication, wherein the first area (102) is used to store one or more first work queue elements WQE, and the time-based indication is used to indicate that the one or more WQE in the first area (102) is processed periodically. According to the time-based instruction, the one or more first WQEs (103, 103') in the first region (102) are processed periodically.

14. The method according to claim 13, characterized in that, The time-based indication includes at least one of the following: The periodic interval T used to process the one or more WQEs in the first region (102); The number of cycles N used to process the one or more WQEs in the first region (102), where N is a positive integer; The duration of processing one or more WQEs in the first region (102) periodically; Before receiving a stop command, the one or more WQEs in the first region (102) are processed continuously.

15. The method according to claim 13, characterized in that, The one or more first WQEs (103, 103') are associated with at least one of a first destination address and a second destination address, wherein the method further includes periodically processing the one or more first WQEs (103, 103') to at least one of a first destination entity (100) corresponding to the first destination address and a second destination entity (100) corresponding to the second destination address.

16. The method according to any one of claims 13 to 15, characterized in that, Also includes: Maintain a completion queue, wherein the first region (102) is also associated with a completion indication, wherein the completion indication is used to indicate the generation method of completion.

17. The method according to claim 16, characterized in that, The completion instruction includes any one of the following: Generation is completed after a predefined number of processed first WQEs (103, 103'); The generation is completed after a predefined number of cycles; Generation completes if an error or a predefined type of error occurs; In response to the completion of notification generation; The process is not complete.

18. The method according to any one of claims 13 to 15, characterized in that, Also includes: When an error occurs, stop processing of the one or more first WQEs (103, 103') in the first region (102).

19. The method according to any one of claims 13 to 15, characterized in that, Also includes: Add one or more third WQEs to the first region (102); The one or more third WQEs are processed periodically in the next cycle.

20. The method according to any one of claims 13 to 15, characterized in that, Also includes: The time-based QP (101) is obtained from at least one of the first and second applications.

21. The method according to claim 15, characterized in that, Also includes: Maintain a memory region for storing specific data for the entity (100); Monitor the memory region and determine whether the specific data has changed; When it is determined that the specific data has changed, the content of the specific data is provided to one or more other entities for RDMA using one or more WQEs based on the time-based QP (101).

22. A computer program product, characterized in that, Includes program code for executing the method according to any one of claims 13 to 21 when implemented in a processor.

Images