RDMA-based distributed system multi-copy distribution method

Abstract

The invention relates to a distributed system multi-copy distribution method based on RDMA (Remote Direct Memory Access), belongs to the technical field of distributed systems, and solves the problem of relatively high write-in time delay of distributed system multi-copy distribution in the prior art. The method comprises the following steps: a client writes data corresponding to a plurality of I / O requests into a predetermined memory slot position of a master node and predetermined memory slot positions of slave nodes; the client sends the global metadata to the main node; the master node sends the local metadata aiming at the master node to the master node, and sends the local metadata aiming at each slave node to each slave node; each slave node persists data stored in a predetermined memory slot of the slave node to a nonvolatile storage medium of the slave node; and the master node persists data stored in a predetermined memory slot of the master node to a non-volatile storage medium of the master node, and the data persisted to the non-volatile storage medium of the master node and the data persisted to the non-volatile storage medium of each slave node are copies of each other. And the write-in time delay is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed system, and particularly relates to a distributed system multi-copy distribution method based on RDMA. BACKGROUND

[0002] Multi-copy distribution in a distributed system is a key technology to ensure high availability, high fault tolerance and high performance of data.

[0003] Master-slave replication is a widely used multi-copy distribution scheme. Its principle is to divide all storage nodes into a master node (Primary) and multiple slave nodes (Standby). The client sends data to the master node, the master node processes the write request, and synchronizes the data changes specified by the write request to multiple slave nodes.

[0004] In a high-concurrency or high-latency network environment, the write latency of the current multi-copy distribution technology scheme is difficult to meet the requirements. SUMMARY

[0005] In view of the above analysis, the embodiments of the present application aim to provide a distributed system multi-copy distribution method based on RDMA to solve the problem of high write latency in the multi-copy distribution of the existing distributed system.

[0006] The embodiments of the present application provide a distributed system multi-copy distribution method based on RDMA, comprising the following steps:

[0007] The client writes the data corresponding to the plurality of I / O requests into the predetermined memory slot of the master node and the predetermined memory slot of each slave node;

[0008] The client generates global metadata according to the persistent information corresponding to the plurality of I / O requests, the predetermined memory slot of the master node and the predetermined memory slot of each slave node, and sends the global metadata to the master node;

[0009] The master node determines local metadata for the master node and local metadata for each slave node according to the global metadata, sends the local metadata for the master node to the master node, and sends the local metadata for each slave node to each slave node respectively;

[0010] Each slave node uses the received local metadata to persist the data stored in the predetermined memory slot of the slave node to the non-volatile storage medium of the slave node, and the master node uses the received local metadata to persist the data stored in the predetermined memory slot of the master node to the non-volatile storage medium of the master node, wherein the data persisted to the non-volatile storage medium of the master node and the data persisted to the non-volatile storage medium of each slave node are mutual copies.

[0011] Further improvement based on the above method, after the data stored in the predetermined memory slot is persisted to the non-volatile storage medium, further includes:

[0012] The master node generates a confirmation submission response according to the received local metadata, and each slave node generates a confirmation submission response according to the received local metadata and sends it to the master node;

[0013] The master node generates a whole confirmation response for the global metadata according to the confirmation submission responses of each slave node and the master node, and sends it to the client;

[0014] After the client receives the whole confirmation response, it sets the predetermined memory slot of the master node and the predetermined memory slot of each slave node to the idle state.

[0015] Further improvement based on the above method, before the client writes the data corresponding to the plurality of I / O requests to the predetermined memory slot of the master node and the predetermined memory slot of each slave node, further includes:

[0016] The user layer generates a plurality of logically associated I / O requests for the data to be stored in multiple copies, and sends the generated I / O requests to the client;

[0017] The client receives the I / O request sent by the user layer, and records the cache location of the data of the I / O request and the persistence information of the data of the I / O request for each I / O request, wherein the data of the I / O request is a data block, and the data of the plurality of logically associated I / O requests is persisted according to the persistence information.

[0018] Further improvement based on the above method, before the client writes the data corresponding to the plurality of I / O requests to the predetermined memory slot of the master node and the predetermined memory slot of each slave node, further includes:

[0019] The client processes any I / O request received from the user layer into an I / O data list, and adds the processed I / O data list to the I / O sending queue; the I / O data list records a metadata part and a data part; the metadata part records the persistence information of the I / O request; the data part records the cache location of the data of the I / O request;

[0020] The client allocates the plurality of I / O data lists in the I / O sending queue to the predetermined memory slot of the master node and the predetermined memory slot of each slave node;

[0021] The client generates a master node storage copy according to the allocated predetermined memory slot of the master node, and adds the processed master node storage copy to the master node copy data buffer;

[0022] The client generates the storage copies of each slave node according to the predetermined memory slots of each slave node after the allocation, and adds the processed storage copies of each slave node to the storage copy data buffer of each slave node.

[0023] Based on the further improvement of the above method, the client writes the data corresponding to the plurality of I / O requests into the predetermined memory slots of the master node and the predetermined memory slots of each slave node, comprising:

[0024] The client writes the data corresponding to the plurality of I / O requests recorded in the master node storage copy from the cache location to the predetermined memory slots of the master node in batches through the RDMAWrite operation;

[0025] The client writes the data corresponding to the plurality of I / O requests recorded in the storage copy of each slave node from the cache location to the predetermined memory slots of each slave node in batches through the RDMAWrite operation.

[0026] Based on the further improvement of the above method, the client generates global metadata according to the persistent information corresponding to the plurality of I / O requests, the predetermined memory slots of the master node and the predetermined memory slots of each slave node, comprising:

[0027] The client obtains the persistent information of the plurality of I / O requests recorded in the I / O data list and the predetermined memory slots of the master node allocated for the plurality of I / O requests, and generates local metadata for the master node;

[0028] The client obtains the persistent information of the plurality of I / O requests recorded in the I / O data list and the predetermined memory slots of each slave node allocated for the plurality of I / O requests, and generates local metadata for each slave node;

[0029] The client combines the local metadata for the master node and the local metadata for each slave node into global metadata.

[0030] Based on the further improvement of the above method, each slave node uses the received local metadata to persist the data stored in the predetermined memory slots of the slave node to the non-volatile storage medium of the slave node, comprising:

[0031] Each slave node receives local metadata, determines the persistent information corresponding to the plurality of I / O requests and the predetermined memory slots of the slave node, and uses the persistent information to persist the data stored in the predetermined memory slots of each slave node to the non-volatile storage medium of each slave node.

[0032] The master node uses the received local metadata to persist the data stored in the predetermined memory slots of the master node to the non-volatile storage medium of the master node, comprising:

[0033] The master node receives the local metadata, determines the persistent information corresponding to the plurality of I / O requests and the predetermined memory slot of the master node, and uses the persistent information to persist the data stored in the predetermined memory slot of the master node to the non-volatile storage medium of the master node.

[0034] Based on the further improvement of the above method, the number of master node storage replicas processed by the client in one RDMAWrite operation is the same as the number of each slave node storage replica processed by the client in one RDMAWrite operation.

[0035] The number of master node storage replicas processed by the client in one RDMAWrite operation is determined by the minimum of the predefined maximum batch size, the minimum of the number of free memory slots available to the client master node and each slave node.

[0036] Based on the further improvement of the above method, before the client writes the data corresponding to the plurality of I / O requests to the predetermined memory slot of the master node and the predetermined memory slot of each slave node, the method further comprises:

[0037] The master node or each slave node allocates a predetermined memory pool, divides the predetermined memory pool into a plurality of memory slots of a specified capacity, allocates a unique slot identifier to each memory slot in the predetermined memory pool, and records each memory slot as a free state or an occupied state.

[0038] Register the predetermined memory pool as an RDMA accessible memory area; generate a bitmap for the predetermined memory pool.

[0039] After the client receives the I / O request sent by the user layer, the client specifies that a plurality of nodes in the distributed system participate in the replica update service, and specifies one of the plurality of nodes as a master node and the other nodes as slave nodes.

[0040] When the client first connects to the master node or each slave node, the master node or each slave node scans the free memory slots in the bitmap, allocates a specified number of memory slots in the predetermined memory pool to the client, and returns the slot identifiers and RDMA access information of the specified number of memory slots to the client.

[0041] Based on the further improvement of the above method, when the client maintains a connection with the master node or each slave node,

[0042] The master node or each slave node calculates the stress index of the client according to the number of RDMAWrite operations initiated by the client per unit time, the average filling time ratio of the current slot of the client, and the number of times the client triggers expansion within a preset time, and executes the memory slot expansion strategy for the client according to the stress index.

[0043] The master node or each slave node calculates a scaling index of the client according to the idle time ratio of the current slot of the client and the minimum time interval of two scaling operations of the client, and executes a memory slot number scaling strategy for the client according to the scaling index.

[0044] Compared with the prior art, the application can realize at least one of the following beneficial effects:

[0045] 1. The remote memory access technology based on the RDMA protocol optimizes the traditional synchronous replication scheme, the client decouples data writing and metadata submission, the client directly participates in data distribution between the master node and the slave node, the data writing time delay between the master node and the slave node is reduced, and the writing performance of the distributed system is improved.

[0046] 2. The utilization efficiency of the RDMA network is effectively improved, and the data transmission and coordination overhead of the client to the master node and the slave node is reduced by merging I / O requests or aggregating I / O requests in batches locally by the client.

[0047] 3. The client directly participates in master-slave copy distribution, and writes data to the master node and the slave node asynchronously through RDMA; compared with the traditional master-slave replication, the data writing path in the master-slave replication is shortened; the writing delay of the client to the master node and the slave node is effectively reduced.

[0048] 4. The client decouples data writing and metadata submission, and processes data writing and metadata submission respectively through the pipeline technology, so that the data writing efficiency is effectively improved while the data strong consistency between the master node and the slave node is ensured.

[0049] 5. Path optimization. The client directly writes remote memory, eliminates the transfer overhead of the master node to the slave node, such as the master node only needs to transfer metadata but not the data itself; the data transmission path is reduced from 2 hops to 1 hop, and the network delay is reduced by 50% in an ideal case.

[0050] 6. After decoupling, metadata and data are separated; the data stream aggregates multiple I / O requests through the RDMA write primitive, and the control stream realizes the RPC process through the RDMA SEND / RECV primitive, so that the high bandwidth and low delay characteristics of the RDMA network are fully utilized.

[0051] 7. Dynamically allocating memory slots can improve the memory utilization efficiency and reduce the generation of memory fragments.

[0052] In the application, the above technical methods can be combined with each other to realize more preferred combination methods. Other features and advantages of the application will be described in the subsequent specification, and some advantages will become apparent from the specification or be understood by implementing the application. The purposes and other advantages of the application can be realized and obtained from the contents specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and are not intended to limit the scope of the application, and together with the description serve to explain the principles of the application.

[0054] Figure 1 A flow chart of a multi-copy distribution method based on RDMA in a distributed system according to an embodiment of the application;

[0055] Figure 2 A schematic diagram of data interaction between a client and a master node and two slave nodes in a multi-copy distribution method based on RDMA in a distributed system according to an embodiment of the application;

[0056] Figure 3 A schematic diagram of a I / O sending queue of a client, a replica data buffer of a master node and a slave node, and global metadata in a multi-copy distribution method based on RDMA in a distributed system according to an embodiment of the application;

[0057] Figure 4 A schematic diagram of sending and receiving local metadata and persistence of a master node and a slave node in a multi-copy distribution method based on RDMA in a distributed system according to an embodiment of the application. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present application will be described in detail with reference to the drawings, in which the same or similar components are denoted by the same reference numerals, and therefore the same or similar components are not described in detail again, and the principles of the present application are explained with reference to embodiments of the present application, which form a part of this application, and are used to explain the principles of the present application, but are not used to limit the scope of the present application.

[0059] The background of the embodiments of the present application will be described in more detail below.

[0060] With reference to the foregoing description, master-slave replication is a multi-copy distribution scheme widely used in the fields of databases, distributed storage, etc. In master-slave replication, the data synchronization mode is divided into synchronous replication and asynchronous replication according to the consistency, delay and fault tolerance requirements of the system.

[0061] Asynchronous replication refers to that after the client writes data to the master node, the master node directly returns the write success to the client, and then the master node performs data replication to the slave node through a background task. The distributed system using this asynchronous replication method has low write delay, but can only guarantee the eventual consistency of data between the master node and the slave node, and there is a risk of data loss when the master node fails.

[0062] Synchronous replication refers to that after the master node receives the data of the client, the master node synchronously replicates the data to all slave nodes, and waits for the confirmation of all slave nodes before confirming the write success to the client. It can guarantee the consistency of data between the master node and the slave node at any time, and is widely used in various high-reliability application systems.

[0063] In the current synchronous replication scheme, data transmission needs to be sent from the client to the master node, and then copied from the master node to the slave node. This serial data transmission method causes the delay of two network transmissions to be superimposed. In addition, although the data consistency between the master and slave nodes at each moment is guaranteed, the master node must wait for all slave nodes to confirm the write completion before returning success to the client. Therefore, the write latency of the current synchronous replication scheme is high, that is, the write latency is the time experienced from the client writing data to the master node to the data consistency of all master and slave nodes.

[0064] The following lists the special terms related to the embodiments of the application:

[0065] Remote Procedure Call (RPC).

[0066] Remote Direct Memory Access (RDMA) protocol.

[0067] Using the RDMA protocol, a node configured with an RDMA dedicated network card (such as RNIC) can remotely and directly access the local memory of another node configured with an RDMA dedicated network card in the RDMA network without the participation of the computer operating system kernel of the other node, and has a kernel bypass feature.

[0068] The following specifically describes the embodiments of the application.

[0069] One specific embodiment of the application discloses a distributed system multi-copy distribution method based on RDMA, as shown in Figure 1 The method comprises the following steps:

[0070] The client writes data corresponding to a plurality of I / O requests into a predetermined memory slot of the master node and a predetermined memory slot of each slave node;

[0071] The client generates global metadata according to the persistent information corresponding to the plurality of I / O requests, the predetermined memory slot of the master node, and the predetermined memory slot of each slave node, and sends the global metadata to the master node;

[0072] The master node determines local metadata for the master node and local metadata for each slave node according to the global metadata, sends the local metadata for the master node to the master node, and sends the local metadata for each slave node to each slave node, respectively;

[0073] Each slave node utilizes the received local metadata to persist data stored in a predetermined memory slot of the slave node to a non-volatile storage medium of the slave node; the master node utilizes the received local metadata to persist data stored in a predetermined memory slot of the master node to a non-volatile storage medium of the master node, wherein the data persisted to the non-volatile storage medium of the master node and the data persisted to the non-volatile storage medium of each slave node are mutual copies.

[0074] The embodiment method can be abstracted into three stages of data writing, metadata distribution, data coordination submission, etc.

[0075] The data writing stage is executed by the client. The client directly accesses the memory slots of the master and slave nodes through RDMA remote memory access based on the RDMA network, and writes the data corresponding to multiple I / O requests in batches and in parallel into the predetermined memory slots of the master node and the predetermined memory slots of each slave node, so that the slave node (Slave / Standby) and the master node (Master / Primary) store exactly the same data, ensuring the consistency of the memory data between the master and slave nodes; without generating data copies, the zero-copy function is realized.

[0076] Specifically, as shown in Figure 2 In the RDMA-based distributed system, the RDMA-based client accesses the memory of the master node and two slave nodes configured with RDMA dedicated network cards (RNIC) in the RDMA network in parallel and in batches, and writes data into the memory of the master node or the slave node.

[0077] The metadata distribution stage is executed by the client. As shown in Figure 2 The client generates global metadata meta-all according to the persistent information corresponding to multiple I / O requests, the predetermined memory slots of the master node and the predetermined memory slots of each slave node, and sends the global metadata meta-all to the master node by using a lightweight RPC request.

[0078] The data coordination submission stage is responsible for coordinating all slave nodes to submit data by the master node. The master node parses the lightweight RPC request sent by the client, coordinates all slave nodes to submit data, collects the confirmation submission responses of all slave nodes, generates an overall confirmation response such as rsp-all, and confirms to the client whether the data is successfully submitted. All slave nodes parse the coordination submission messages such as meta-rep1 and meta-rep2 sent by the master node, persist the data in the local memory slot to the local hard disk, and then send the confirmation submission responses such as rsp1 and rsp2 to the master node to confirm the completion of data submission.

[0079] Specifically, after persisting the data stored in the predetermined memory slot to the non-volatile storage medium, the method further includes:

[0080] The master node generates a confirmation submission response according to the received local metadata, and each slave node generates a confirmation submission response according to the received local metadata and sends the confirmation submission response to the master node.

[0081] The master node generates an overall confirmation response for the global metadata according to the confirmation submission responses of each slave node and the master node, and sends the overall confirmation response to the client.

[0082] After the client receives the overall confirmation response, the client sets the predetermined memory slot of the master node and the predetermined memory slot of each slave node to an idle state.

[0083] At this point, the multi-copy distribution for the plurality of I / O requests is completed, and the write latency for the plurality of I / O requests can be determined. The write latency includes two network latencies, i.e., a data write latency and a data coordination submission latency.

[0084] Based on the further improvement of the above method, before the client writes the data corresponding to the plurality of I / O requests into the predetermined memory slot of the master node and the predetermined memory slot of each slave node, the method further includes:

[0085] The user layer generates a plurality of I / O requests logically associated with the data to be stored in multiple copies, and sends the generated I / O requests to the client.

[0086] The client receives the I / O requests sent by the user layer, and records, for each I / O request, the cache location of the data of the I / O request and the persistence information of the data of the I / O request, wherein the data of the I / O request is a data block, and the data of the plurality of I / O requests logically associated are persisted according to the persistence information.

[0087] Specifically, as shown in Figure 3 , Figure 4 The persistence information of the data of the I / O request includes an offset. The offset is an offset assigned by the user layer to each data block of the plurality of I / O requests logically associated when the user layer generates the plurality of I / O requests. The offset is used to indicate the offset of each data block relative to a reference benchmark in the data to be stored in multiple copies. When the data is persisted in a hard disk, each node persists the data block from each I / O request according to the offset, assembles the data blocks of the plurality of I / O requests logically associated into the data to be stored in multiple copies, and stores the data to be stored in multiple copies in the local hard disk. Figure 4 In the above, storage is used to indicate the local hard disk of each node; commit is used to indicate the operation of writing data in the memory slot of the node to the local hard disk of the node.

[0088] Specifically, as shown in Figure 3 , Figure 4As shown, the persistence information of the data of the I / O request can further include: length. The length is the data length of the data block determined by the user layer when generating each data block for the to-be-multi-copy-stored data, which is the absolute capacity of the data or the number of memory slots occupied by the data. Specifically, the data length is the minimum unit when the subsequent client writes data to the predetermined memory slot of the master node and the predetermined memory slot of each slave node.

[0089] Specifically, in a high concurrency scenario, multiple user layers in the distributed system generate multiple logically associated I / O requests for the received to-be-multi-copy-stored data, and send the generated multiple I / O requests to the selected client.

[0090] After the selected client receives the I / O request sent by the user layer, according to the pre-set business rule, such as the three-copy distributed storage mode, in the RDMA distributed system, a plurality of nodes reachable through the RDMA network are determined and the plurality of nodes are controlled to perform multi-copy distribution.

[0091] Specifically, when the client controls the plurality of nodes to perform multi-copy distribution, one of the plurality of nodes is designated as a master node, and the other nodes in the plurality of nodes are designated as slave nodes.

[0092] Specifically, the distributed system determines a node in the distributed system as a task node according to the current load balancing and other constraints, and sends the logically associated multiple I / O requests generated by the user layer to the client of the task node, which will not be described in detail.

[0093] Specifically, as shown in the above embodiment, before the client writes the data corresponding to the multiple I / O requests into the predetermined memory slot of the master node and the predetermined memory slot of each slave node, the client further includes: Figure 3

[0094] The client processes any I / O request received from the user layer into an I / O data list, and adds the processed I / O data list to an I / O sending queue; the I / O data list records a metadata part and a data part; the metadata part records the persistence information of the I / O request; and the data part records the cache location of the data of the I / O request.

[0095] The client allocates the multiple I / O data lists in the I / O sending queue to the predetermined memory slot of the master node and the predetermined memory slot of each slave node, wherein each I / O data list corresponds to a memory slot of the master node and a memory slot of each slave node.

[0096] The client generates a master node storage copy according to the allocated predetermined memory slot of the master node, and adds the processed master node storage copy to a master node copy data buffer.

[0097] ​The client generates the storage replica of each slave node according to the predetermined memory slot of each slave node after the allocation, and adds the processed storage replica of each slave node to the replica data buffer of each slave node.

[0098] Specifically, as shown in Figure 3 , the client processes the I / O request received from the user layer, such as I / O1 or I / O2, into an I / O data list in parallel, and adds the processed I / O data list to the I / O sending queue send queue, and the I / O data list corresponding to I / O1 or I / O2 is identified by io1 or io2 in the send queue; the I / O data list records the metadata part, such as meta1 or meta2, and the data part data, such as data1 or data2; the metadata part records the persistent information of the I / O request, such as offset1, offset2 or length1, length2; and the data part records the cache location of the data of the I / O request, such as buffer1 or buffer2.

[0099] Specifically, as shown in Figure 3 , Figure 4 , the client allocates the predetermined memory on the master node and the predetermined memory on each slave node for a plurality of I / O data lists (such as M1) in the I / O sending queue, wherein each I / O data list corresponds to a memory slot on the master node and a memory slot on each slave node, such as the predetermined memory slot id11 allocated for the io1 request on the master node, the predetermined memory slot id12 allocated for the io1 request on the slave node 1, and the predetermined memory slot id13 allocated for the io1 request on the slave node 2; and the predetermined memory slot id21 allocated for the io2 request on the master node, the predetermined memory slot id22 allocated for the io2 request on the slave node 1, and the predetermined memory slot id23 allocated for the io2 request on the slave node 2 are specifically shown, and thus each I / O request corresponds to a memory slot in the predetermined memory.

[0100] Specifically, as shown in Figure 3As shown, the primary node stores the replica record of the identification of the I / O request, such as io1 or io2; the cache location or data logical name of the data of the I / O request, such as data1 or data2; the predetermined memory slot of the primary node allocated for the I / O request, such as slot id11, slot id21; and the replica record of the identification of the I / O request, such as io1 or io2; the cache location or data logical name of the data of the I / O request, such as data1 or data2; the predetermined memory slot of the secondary node allocated for the I / O request, such as slot id12, slot id22 for the secondary node 1, and slot id13, slot id23 for the secondary node 2.

[0101] Specifically, as shown in FIG. 4, the buffer queue databuffer stores the primary node replica data buffer for the M1 I / O requests, which is identified by prim data, and specifically shows the primary node replica for the io1 request, which records io1, data1 and slot id11, and the primary node replica for the io2 request, which records io2, data2 and slot id21. Figure 3

[0102] Specifically, as shown in FIG. 4, the buffer queue databuffer stores the primary node replica data buffer for the M1 I / O requests, which is identified by prim data, and specifically shows the primary node replica for the io1 request, which records io1, data1 and slot id11, and the primary node replica for the io2 request, which records io2, data2 and slot id21. Figure 3

[0103] Specifically, as shown in FIG. 4, the buffer queue databuffer stores the primary node replica data buffer for the M1 I / O requests, which is identified by prim data, and specifically shows the primary node replica for the io1 request, which records io1, data1 and slot id11, and the primary node replica for the io2 request, which records io2, data2 and slot id21. Figure 3

[0104] Specifically, the client writes the data corresponding to the plurality of I / O requests into the predetermined memory slots of the primary node and the predetermined memory slots of each secondary node, including:

[0105] The client writes the data corresponding to the plurality of I / O requests recorded by the primary node replica in bulk from the cache locations thereof to the predetermined memory slots of the primary node through RDMAWrite operation; ​​​

[0106] The client writes the data corresponding to the plurality of I / O requests recorded in the storage copy of each slave node from the cache location thereof in batches to the predetermined memory slot of each slave node through an RDMAWrite operation.

[0107] Specifically, the client directly participates in data distribution between the master and slave nodes, executes a data write process to form a data stream, and aggregates a plurality of (e.g., M2) I / O requests using an RDMAWRITE primitive, which is conducive to fully utilizing the high-bandwidth and low-latency characteristics of the RDMA network.

[0108] Specifically, as shown in Figure 3 , Figure 4 , the client maintains a sending queue for the I / O requests received from the user layer using a pipeline technique; maintains a storage copy for each I / O data list in the sending queue; and writes the data corresponding to the plurality of I / O requests to the predetermined memory slot of the master node and the predetermined memory slot of each slave node through an RDMAWrite operation.

[0109] Specifically, as shown in Figure 3 , the client continuously polls the completion queue and the copy data buffer maintained for the master node and each slave node in parallel. After detecting that the last RDMAWrite operation triggered the update of the completion queue or the completion list of the copy data maintained for the master node and each slave node, the client writes the data corresponding to M2 I / O requests recorded in the storage copy of the master node from the cache location thereof in batches to the predetermined memory slot of the master node through an RDMAWrite operation, and writes the data corresponding to the plurality of I / O requests recorded in the storage copy of each slave node from the cache location thereof in batches to the predetermined memory slot of each slave node through an RDMAWrite operation, where M2 is a positive integer.

[0110] In this way, the client writes the data of M2 I / O requests in batches and in parallel to the idle memory slot of the master node and the idle memory slot of each slave node, without generating a data copy, thereby realizing a zero-copy function; the predetermined memory slot of each slave node and the predetermined memory slot of the master node store exactly the same data, thereby ensuring the consistency of the memory data between the master and slave nodes.

[0111] Specifically, the client processes all N requests received in batches using the sending queue. Specifically, after the client allocates the predetermined memory of the master node and the predetermined memory of each slave node for the M1 I / O requests corresponding to the M1 I / O data lists in the sending queue, respectively, the M1 I / O data lists will be removed from the sending queue. It should be understood that, for the convenience of display, Figure 3 io1 or io2 in the sending queue send queue is not removed.

[0112] Specifically, the data data of each I / O request occupies a slot in the memory. For example, the capacity of each slot can be 64K. Figure 3 The length of the meta can record the absolute capacity of the data block, such as 64K.

[0113] Specifically, using the replica data buffer, the client batch writes the data of multiple I / O requests in parallel to the free memory slots on the master node and the free memory slots on each slave node. Specifically, after the client batch writes the data of M2 I / O requests in parallel to the free memory slots on the master node and the free memory slots on each slave node through the RDMAWrite operation, the M2 master node storage replicas or slave node storage replicas are removed from the respective replica data buffer.

[0114] Specifically, the client also maintains a replica data waiting list and a replica data completion list for the RDMAWrite operation.

[0115] After the client batch writes the data of M2 I / O requests in parallel to the free memory slots on the master node and the free memory slots on each slave node through the RDMAWrite operation, the client transfers the M2 master node storage replicas or slave node storage replicas from the replica data buffer to the replica data waiting list (such as the wait list), which is used to track the write operation requests that have been issued but not yet confirmed.

[0116] Specifically, as shown in Figure 3 The two write operation requests that have been issued but not yet confirmed are identified as io3 and io4 in the replica data waiting list. After the write operation requests io3 and io4 are removed, the replica data buffer also records the I / O requests io5, io6, and io7 to be generated.

[0117] Further, the client asynchronously polls the RDMA completion queue (such as poll CQ) corresponding to the RDMAWrite operation for M2 I / O requests or the aforementioned replica data completion list complete list to detect whether the previously initiated RDMAWrite operation for M2 I / O data lists has been completed. The execution of the RMDAWrite instruction generates the RDMA completion queue (Completion Queue, CQ).

[0118] When a completion success event is obtained from the RDMA completion queue, the client locates the corresponding M2 I / O data lists (such as io3 and io4 identified in the replica data waiting list) in the replica data waiting list according to the work request ID recorded in the completion event, and moves the M2 I / O requests to the replica data completion list.

[0119] Naturally, when a failure event of obtaining completion is obtained from the RDMA completion queue, the client takes out the M2 I / Os from the replica data waiting list, and writes the data recorded by the M2 storage replicas in batch and in parallel into the free memory slots on the master node and the free memory slots on each slave node through the RDMAWrite operation again.

[0120] Specifically, the client continuously polls the replica data buffer, the replica data waiting list and a replica data completion list of the master node or each slave node in parallel. By using the replica data buffer, the replica data waiting list and the replica data completion list, the client can efficiently and reliably process the RDMAWrite in pipeline.

[0121] Specifically, the client continuously polls the replica data buffer, the replica data waiting list and a replica data completion list of the master node or each slave node in parallel. By using the replica data buffer, the replica data waiting list and the replica data completion list, the client can efficiently and reliably process the RDMAWrite in pipeline. Figure 3 The data buffer list (data list) shown in the figure, that is, the replica data buffer, the waiting list and the completion list in the databuffer, is used to efficiently and in parallel write data into the free memory of the master and slave nodes, and to ensure low latency and high throughput of data transmission.

[0122] Specifically, the client generates global metadata according to the persistent information corresponding to the plurality of I / O requests, the predetermined memory slots of the master node and the predetermined memory slots of each slave node, including:

[0123] The client obtains the persistent information of the plurality of I / O requests recorded in the I / O data list and the predetermined memory slots of the master node allocated for the plurality of I / O requests, and generates local metadata for the master node;

[0124] The client obtains the persistent information of the plurality of I / O requests recorded in the I / O data list and the predetermined memory slots of each slave node allocated for the plurality of I / O requests, and generates local metadata for each slave node;

[0125] The client combines the local metadata for the master node and the local metadata for each slave node into global metadata.

[0126] Specifically, each slave node uses the received local metadata to persist the data stored in the predetermined memory slots of the slave node to the non-volatile storage medium of the slave node, including:

[0127] Each slave node receives the local metadata, determines the persistent information corresponding to the plurality of I / O requests and the predetermined memory slot of the slave node, and uses the persistent information to persist the data stored in the predetermined memory slot of each slave node to the non-volatile storage medium of each slave node; wherein the data stored in the non-volatile storage medium of the slave node is logically associated and consistent with the logical association of the plurality of I / O requests.

[0128] The master node uses the received local metadata to persist the data stored in the predetermined memory slot of the master node to the non-volatile storage medium of the master node, comprising:

[0129] The master node receives the local metadata, determines the persistent information corresponding to the plurality of I / O requests and the predetermined memory slot of the master node, and uses the persistent information to persist the data stored in the predetermined memory slot of the master node to the non-volatile storage medium of the master node; wherein the data stored in the non-volatile storage medium of the master node is logically associated and consistent with the logical association of the plurality of I / O requests.

[0130] Specifically, after the master node receives the RPC request submitted by the client, the metadata of each I / O request in each slave node is parsed, and the parsed metadata of each I / O request in each slave node is sent to each slave node through the RPC request.

[0131] Specifically, after the slave node receives the RPC request sent by the master node, the physical address where the data is located is located according to the memory slot identifier and the data logical name corresponding to each I / O request recorded in the RPC request, a memory barrier is established, the I / O data is read from the memory slot corresponding to each I / O request, and the data length and data offset of each I / O request are used to persist the data to the non-volatile storage medium.

[0132] Above, the client coordinates the master node and each slave node to execute the metadata submission process to form a control flow; in the control flow, only the metadata meta of the I / O request is transmitted, and the data data of the I / O request is not transmitted, thereby reducing the data amount. Specifically, the control flow realizes the RPC process through the RDMA SEND / RECV primitive.

[0133] Specifically, the number of master node storage replicas handled by the client in one RDMAWrite operation is the same as the number of each slave node storage replica handled by the client in one RDMAWrite operation.

[0134] The number of master node storage replicas handled by the client in one RDMAWrite operation is determined by the minimum of the maximum batch size predefined, and the minimum of the number of free memory slots available to the client master node and each slave node.

[0135] Naturally, the number of I / O requests aggregated in a single RDMAWrite operation is related to, but not necessarily identical to, the number of I / O data lists batched when the client allocates the predetermined memory slots on the master node and the predetermined memory slots on each slave node for the multiple I / O data lists in the I / O sending queue. Naturally, before the client allocates the predetermined memory slots on the master node and the predetermined memory slots on each slave node for the multiple I / O data lists in the I / O sending queue, the client queries the free memory slot information of each node.

[0136] In the above, the batch parallel write size is dynamically adjusted based on the number of free slots, and the distributed system can adaptively cope with load changes as a whole, avoid memory resource competition, and thus improve overall write efficiency.

[0137] Specifically, before the client writes the data corresponding to the multiple I / O requests to the predetermined memory slots on the master node and the predetermined memory slots on each slave node, the method further includes:

[0138] The master node or each slave node allocates a predetermined memory pool, divides the predetermined memory pool into multiple memory slots of a specified capacity, allocates a unique slot identifier to each memory slot in the predetermined memory pool, and records each memory slot as a free state or an occupied state;

[0139] The predetermined memory pool is registered as an RDMA accessible memory area, and a bitmap is generated for the predetermined memory pool.

[0140] After the client receives the I / O request sent by the user layer, the client specifies that multiple nodes in the distributed system participate in the replica update service, and specifies one of the multiple nodes as a master node and the other nodes as slave nodes.

[0141] When the client first connects to the master node or each slave node, the master node or each slave node scans the free memory slots in the bitmap, allocates a specified number of memory slots in the predetermined memory pool to the client, and returns the slot identifiers and RDMA access information such as R_key or VirtualAddress of the specified number of memory slots to the client.

[0142] Specifically, each node determines the capacity or size of the memory slot or memory unit according to the actual business requirement (such as the capacity of the data block). When each node is initialized, the node pre-allocates a memory pool for RDMA, divides the memory pool into a plurality of memory slot positions slots according to the size of the memory slot, such as 4096 slots with a memory slot size of 64K can be obtained from a memory pool of 256M. Each memory slot of the memory pool is assigned a unique id, and its state is recorded in real time, such as free or busy. The memory pool for RDMA is registered with the operating system kernel as an RDMA accessible memory region (MemoryRegion, MR) for client remote writing.

[0143] In this way, the master node or each slave node generates a bitmap locally for the predetermined memory pool, and when a client first connects, the bitmap is used to allocate a specified number of memory slots in the predetermined memory pool for the client.

[0144] Specifically, before processing the I / O request, when the client first connects to the master node or the slave node, the master node or the slave node scans the free memory slot position of the bitmap, allocates a specified number of initial free slot positions (such as 64 slots with a memory slot size of 64K, a total capacity of 4M, and not necessarily contiguous memory slots) for the client, and returns the memory slot id and RDMA access information (such as remote key R_key, virtual address VirtualAddress) in the initial free slot position to the client.

[0145] In this way, the client can record the state of the memory slot position of the slave node in real time through the RDMA protocol, directly access the memory region of the master and slave nodes using the RDMA access information, and remotely write data to each memory slot in the initial free slot position of the slave node (usually, the amount of data written by the client will not fill the initial free slot position).

[0146] In the multi-concurrent scenario of the distributed system, a large number of or massive I / O requests from multiple user layers usually need to be processed. There are multiple clients in the distributed system, and each client determines a master node and each slave node to form a replica distribution group in the distributed system, that is, the same RDMA network in the physical sense, and can correspond to multiple replica distribution groups in the logical sense, that is, there are multiple combinations of master nodes and slave nodes. Therefore, the master node and each slave node in the same RDMA network need to serve multiple clients for the predetermined memory pool allocated by the RDMA protocol. Figure 2 and Figure 3 The three-replica mode is shown.

[0147] Specifically, according to the pressure index, a memory slot number expansion strategy for the client is executed, including:

[0148] According to the pressure index, the pressure level is classified as high pressure, medium pressure or low pressure; when the pressure level is high pressure, the number of current slots is extended by exponential growth; when the pressure level is medium pressure, the number of current slots is extended by linear growth; and when the pressure level is low pressure, the number of current slots is kept unchanged.

[0149] Specifically, according to the shrinkage index, the memory slot number shrinkage strategy for the client is executed, including:

[0150] The master node or each slave node calculates the shrinkage index according to the idle time ratio of the current slot of the client and the minimum time interval of two shrinkage operations of the client; when the shrinkage index is greater than a preset threshold, the shrinkage operation is triggered, wherein each shrinkage reduces the number of current slots by at most 50%; if the pressure index of the client increases after the shrinkage operation, the expansion operation is triggered and the expansion cool-off period limit is skipped.

[0151] The client determines the number of I / O requests taken out from the sending queue in a batch when allocating the predetermined memory according to the number of memory slots after expansion or shrinkage, that is, the number of I / O requests filled into the replica data buffer databuffer subsequently.

[0152] Naturally, the above expansion operation or shrinkage operation is performed in parallel by the master node and each slave node controlled by the same client, so as to realize that the slave node (Slave / Standby) and the master node (Master / Primary) store the same data, and ensure the consistency of memory data between the master node and the slave node.

[0153] In this way, the master node or each slave node generates a bitmap for the predetermined memory pool locally, and flexibly expands or shrinks for each client according to the processing pressure of the client when connected with the master node or each slave node, and dynamically allocates the memory slots of the predetermined memory pool among multiple clients.

[0154] Specifically, determining whether the client needs to trigger the memory slot expansion includes:

[0155] The three indexes of the total number of write initiated by the client in a unit of time (denoted as write frequency W, such as the number of RMDAwrite operations performed in a unit of time, and the number of I / O requests written by each RMDAwrite operation is accumulated; it should be understood that the amount of data M2 written by the client each time RMDAwrite operation is not fixed), the average filling time ratio of the current slot of the client (denoted as memory slot usage U; the ratio of the write delay of the memory slot to the unit of time when performing an I / O), and the number of times of triggering expansion of the client in a specified time (denoted as expansion frequency F, the interval between adjacent two expansions is usually a non-uniform time interval, and the number of times of triggering expansion in a specified time is counted to obtain the expansion frequency F) are calculated according to the write pressure index formula listed as follows to calculate the write pressure index (P) of the current client:

[0156] P = a (W / Wmax) + b (U / Umax) + g (F / Fmax);

[0157] Wherein, a, b, g are expansion weight coefficients, the default values are 0.4, 0.4 and 0.2 respectively, naturally, the expansion weight coefficients can be adjusted according to the actual business demand; Wmax, Umax and Fmax are the maximum threshold of each variable preset by the distributed system.

[0158] According to the write pressure index, the pressure level is divided into high pressure (P≥0.7), medium pressure (0.3≤P<0.7) and low pressure (P<0.3);

[0159] Under different pressure levels, different adjustment strategies are adopted to adjust the number of memory slots, wherein the number of memory slots before adjustment is denoted as Scurrent, and the number of memory slots after adjustment is denoted as Snew.

[0160] Specifically, when the pressure level is high pressure (P≥0.7), the exponential growth strategy is adopted to expand the memory pool capacity to quickly respond to burst traffic:

[0161] Snew = Scurrent x 2;

[0162] In this way, when expanding, the number of memory slots or the size of the memory pool increases exponentially with a power of 2, and doubles successively.

[0163] Specifically, when the pressure level is medium pressure (0.3≤P<0.7), the linear growth strategy is adopted to expand the memory pool capacity to balance performance and memory:

[0164] Snew = Scurrent + D,

[0165] Wherein, D is a positive integer, such as D is 25-50% of the current number of memory slots Scurrent; such as D is 4M.

[0166] Specifically, when the pressure level is low pressure (P < 0.3), the number of memory slots remains unchanged, such as Snew = Scurrent, to avoid memory waste.

[0167] Further, determining whether the client needs to trigger memory slot expansion further includes:

[0168] To avoid a single client occupying a large amount of memory, the following multiple conditions need to be met when expanding the memory slot: maximum slot limit Smax, minimum slot limit Smin, and expansion cooling-off period constraint.

[0169] Specifically, when the initial free slot is 4M, the maximum slot limit Smax can be 64MB (16 times the initial free slot 4M); the minimum slot limit Smin can be 4MB (the initial free slot 4M). Naturally, according to actual business needs, the maximum slot limit Smax and the minimum slot limit Smin can be flexibly configured.

[0170] Specifically, the expansion cooling-off period constraint includes: the interval between two expansions of the same client is not less than 1 second, such as ≥ 1 second. In this way, jitter can be prevented.

[0171] Further, determining whether the client needs to trigger memory slot expansion further includes:

[0172] If the client does not trigger expansion again within T time (such as 2s) after allocating new memory slots to increase the number of memory slots, it is considered that the allocation is reasonable, and the current expansion weight coefficient is maintained; if the memory slot is still quickly filled after expansion (the total number of write operations initiated by the client per unit time; the average filling time proportion of the current slot of the client is greater than a preset threshold, such as 0.9), then the α and β weights are increased, and the γ weight remains unchanged; if the memory slot is not fully utilized, then the α and β weights are reduced, and the γ weight remains unchanged.

[0173] Specifically, when the client pressure decreases (such as from high pressure or medium pressure to low pressure), the excess memory slots are gradually released by calculating the contraction index (S).

[0174] Specifically, the contraction index is calculated by the free time ratio of the current slot of the client (denoted as Ufree, that is, the difference between 1 and the average filling time proportion of the current slot of the client) and the length of time (Tidle) that the client has been in a low pressure state continuously.

[0175] S = λ * P + μ * Ufree + ν * (Tidle / Tmax),

[0176] Where λ, μ, and ν are contraction weight coefficients, and the default values are 0.5, 0.3, and 0.2, respectively. Naturally, the contraction weight coefficients can be adjusted according to actual business needs.

[0177] Tidle is the duration that the client is continuously in a low pressure state, and Tmax is the maximum threshold duration preset by the distributed system.

[0178] When the shrinkage index S is greater than or equal to 0.6, the shrinkage process is triggered, and each shrinkage reduces the current memory slot size by at most 50%.

[0179] The shrinkage cooling-off period (≥T cooldown ) is 30 seconds by default, and expansion is positive, while shrinkage is negative. The shrinkage operation is prohibited during the shrinkage cooling-off period. If the client pressure suddenly increases (such as the pressure level changes to medium pressure or high pressure) after shrinkage, expansion is triggered immediately and the expansion cooling-off period limit is skipped.

[0180] Embodiments of the present application are described further below with reference to the accompanying drawings.

[0181] As Figure 2 shown, the process of completely processing a plurality of I / O requests in a group includes:

[0182] The client decouples the plurality of I / O requests into a data write process and a metadata submission process, and decouples the data of the plurality of I / O requests into data parts or data (data) and metadata (meta). The client processes the data (data) of the I / O requests in the data write process first, and then processes the metadata (meta) of each I / O request in the metadata submission process.

[0183] Specifically, the client writes the data part (data) of the I / O request directly to the memory of each node (master node or slave node) in the distributed cluster based on RDMA to process the data part (data) of the request, completing the data write process.

[0184] Specifically, the client processes the metadata (meta) of the I / O request into global metadata (meta-all) and sends the global metadata (meta-all) to the master node. The master node processes the global metadata (meta-all) into metadata for each slave node (such as meta-rep1; meta-rep2) and sends it to each slave node. Each slave node receives the metadata (meta-rep1; meta-rep2) for each slave node and performs persistent storage of the memory data, completing the metadata submission process.

[0185] Specifically, a large number of even massive I / O requests sent by the user layer are processed, and the client of any node in the distributed system processes a plurality of I / O requests in each group through pipeline technology, effectively improving the data write efficiency.

[0186] To this end, as Figure 2As shown, the process of completely processing a plurality of I / O requests in a group also includes: after each slave node completes the persistence of the memory data, returning a persistence response (such as rsp1; rsp2) to the master node; the master node receiving the confirmation submission response (such as rsp1; rsp2) returned by each slave node, and returning an overall confirmation response (rsp-all) to the client; the client processing the overall confirmation response (rsp-all) and releasing the memory of each node.

[0187] Specifically, based on RDMA and RPC, after the client receives the overall confirmation response, it can confirm that the data submission is successful, that is, it can determine that the hard disk data is consistent between all master-slave nodes.

[0188] Specifically, as shown in Figure 3 The client local buffer is implemented using a dual-buffer separation architecture that separates metadata and data. The client decouples traditional monolithic I / O requests into data flow and control flow, which are processed in parallel to achieve efficient distribution of multiple copies.

[0189] As shown in Figure 3 The data flow corresponds to the copy data buffer databuffer. The copy data buffer databuffer uses multiple independent channels for each node in the distributed cluster to maintain a dedicated copy data buffer for the corresponding storage copy. Each copy data buffer has a node identifier indicating the master node or each slave node. At any time, each copy data buffer records a plurality of I / O data lists.

[0190] As shown in Figure 3 The control flow corresponds to the metadata buffer metabuffer. The metadata buffer records the metadata of each I / O request in the data buffer list, and the memory slot identifier or coordinate information slot id of each I / O request in all copy data buffers. Figure 3 The metabuffer shown in Figure 2 The meta-all shown in

[0191] As shown in Figure 3 The metadata buffer records three I / O metadata lists, which are respectively for the master node prim, the slave node rep1, and the slave node rep2 in the distributed cluster. The I / O metadata list for the master node prim, the I / O metadata list for the slave node rep1, and the I / O metadata list for the slave node rep2 are not described in detail.

[0192] The client maintains global metadata (meta-all) in the metadata buffer. After the client finishes writing the data buffer data into memory, it sends the global metadata (meta-all) maintained in the metadata buffer to the master node via an RPC request. The master node then coordinates the data submission of each master and slave node based on the global metadata information.

[0193] Furthermore, after the client's local buffer is established, the data writing phase begins. During this phase, the client efficiently and in parallel writes its data to the free memory of the master or slave node. This process sequentially includes data writing and request transfer, as well as event handling and state update. These two processes are described in detail below.

[0194] like Figure 4 As shown, during the data writing and request transfer process, the client continuously polls the data buffer queue or replica data buffer established for the target node (master node or slave node). Upon detecting new I / O data io5, io6, and io7, the client uses RDMAwrite to batch and parallelly write these three I / O data entries (io5, io6, and io7) into the pre-allocated free memory slots of the master or slave node. Subsequently, the three I / O data entries (io5, io6, and io7) are transferred from the data buffer queue to the waiting list.

[0195] During the event processing and status update process, the client asynchronously polls the RMDA completion queue (CQ, which stores RDMAwrite event information and records work request i, such as the identifier of I / O request) to detect whether the previously initiated RDMAwrite operation has been completed.

[0196] When a completion event is received from CQ, the client locates the corresponding I / O request in the wait list based on the event information (such as the job request ID), and moves the located I / O request from the wait list to the complete list, making at least one I / O request an entry in the complete list. The entry is then removed from the wait list.

[0197] At this point, in the client-maintained complete list, the I / O request is marked as successful (su, indicating success), indicating that the data has been successfully written into the target node's memory; the I / O request is marked as failed (fa, indicating failure), indicating that the batch writing of the M4 I / O data into the target node's memory failed.

[0198] The above event processing and state updating process ensures the reliability and sequence of write operations, while maximizing parallel processing capability through a non-blocking polling mechanism.

[0199] The above process takes full advantage of the zero-copy (relative to TCP / IP) and kernel bypass features of RMDA technology, significantly reducing the CPU overhead involved in copying data, and achieving parallel data writing between master and slave nodes.

[0200] The above is the data writing phase. The data coordination and submission phase realizes global consistent persistence of data in a distributed cluster through a self-defined three-level atomic submission protocol, specifically including a client submission sub-phase, a master node coordination sub-phase, and a slave node execution sub-phase.

[0201] As shown in Figure 2 , Figure 3 and Figure 4 , in the client submission sub-phase, the client completes meta buffer construction through polling the RMDA completion queue; and obtains the memory slot number of each I / O request in each master-slave node from the data buffer, which has completed RMDA Write data writing; merges it into global metadata meta-all, and sends it to the master node through an RPC request, such as encapsulating the global metadata after formatting, and sending the encapsulated data packet to the master node, fully utilizing the high bandwidth and low delay characteristics of the RDMA network.

[0202] As shown in Figure 2 , Figure 3 and Figure 4 , in the master node coordination sub-phase, the master node receives the RPC request submitted by the client, parses the local metadata of the I / O request in each master-slave node, and then sends the metadata of the I / O request in the slave node to each slave node through an RPC request, such as sending the local metadata in each master-slave node to the corresponding master-slave node according to the master-slave node identifier recorded in the local metadata, such as sending rep1, io1, meta1, and slot id12; rep1, io2, meta2, and slot id22 to slave node rep1 as meta-rep1; the data and process of meta-rep* sent to other slave nodes are implemented in the same way as slave node rep1, and are not repeated here.

[0203] As shown in Figure 2 , Figure 3 and Figure 4As shown, in the execution sub-stage, the slave node rep1 receives RPC requests (recording meta-rep1, such as rep1, io1, meta1 and slot id12; rep1, io2, meta2 and slot id22). Based on the memory slot id12 and slot id22, it locates the physical address and persistence information of data1 and data2, establishes a memory barrier, and uses the cache to read the data data1 and data2 for io1 and io2 from memory and persists the data to the non-volatile storage medium storage.

[0204] The slave node rep1 returns a persistent acknowledgment to the master node. rsp1 is used to acknowledge that the slave node has written the response returned for each I / O request within the processing of the RPC request, indicating whether the write to disk has failed.

[0205] The master node collects all RPC responses sent by the slave nodes, and finally commits all data that has been successfully persisted by the slave nodes and sends a response back to the client.

[0206] like Figure 2 , Figure 3 and Figure 4 As shown, after the client receives the master node's confirmation commit response rsp-all, it releases the remote memory slots of the master and slave nodes and changes their status to idle.

[0207] Specifically, each slave node maintains a persistent completion list locally, saving successfully persisted I / O requests into this list. The slave node polls the persistent completion list and sends the identifiers of successfully persisted I / O requests to the master node via an RPC response.

[0208] In this way, each slave node receives the RPC request sent by the master node, parses the metadata meta-rep* containing at least one I / O request, reads the memory data according to the memory slot ID and persistence information (such as the offset within the slot) specified by the metadata of at least one I / O request, and persists it to the hard disk.

[0209] In summary, in the RDMA-based distributed system multi-replica distribution method of this embodiment,

[0210] 1. Remote memory access technology based on RDMA protocol optimizes the traditional synchronous replication scheme, decoupling data writing and metadata submission on the client side; the client directly participates in data distribution between master and slave nodes, reducing data writing latency between master and slave nodes and improving the write performance of the distributed system.

[0211] 2. Through the client local buffering batch merging I / O request or aggregation I / O request, the utilization efficiency of the RDMA network is effectively improved, and the data transmission and coordination overhead of the client to the master-slave node are reduced.

[0212] 3. The client directly participates in the master-slave copy distribution, and writes data to the master-slave node through RDMA asynchronous writing; compared with the traditional master-slave replication, the data writing path in the master-slave replication is shortened; and the writing delay of the client to the master-slave node is effectively reduced.

[0213] 4. The client decouples data writing and metadata submission, and processes data writing and metadata submission through pipeline technology, so that the data writing efficiency is effectively improved while ensuring the strong consistency of data between the master-slave nodes.

[0214] 5. Path optimization. The client directly writes to the remote memory, eliminating the transit overhead of the master node to the slave node, such as the master node only needs to transit metadata but not the data itself; the transmission path of the data is reduced from 2 hops to 1 hop, and the network delay is reduced by 50% in an ideal case.

[0215] 6. After decoupling, metadata and data are separated; the data stream aggregates multiple I / O requests through the RDMA write primitive, and the control stream realizes the RPC process through the RDMA SEND / RECV primitive, so as to fully utilize the high-bandwidth and low-delay characteristics of the RDMA network.

[0216] 7. Dynamically allocating memory slots, fully improving the memory utilization efficiency and reducing the generation of memory fragmentation.

[0217] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium is a disk, an optical disk, a read-only memory or a random access memory, etc.

[0218] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought by those skilled in the art, and should be covered within the protection scope of the present application.

Claims

1. A method for RDMA-based distributed system multi-copy distribution, characterized in that, The method comprises the following steps: The client writes data corresponding to the plurality of I / O requests into the predetermined memory slot of the master node and the predetermined memory slot of each slave node; The client generates global metadata according to the persistent information corresponding to the plurality of I / O requests, the predetermined memory slot of the master node and the predetermined memory slot of each slave node, and sends the global metadata to the master node; The master node determines local metadata for the master node and local metadata for each slave node according to the global metadata, sends the local metadata for the master node to the master node, and sends the local metadata for each slave node to each slave node respectively; Each slave node uses the received local metadata to persist the data stored in the predetermined memory slot of the slave node to the non-volatile storage medium of the slave node, and the master node uses the received local metadata to persist the data stored in the predetermined memory slot of the master node to the non-volatile storage medium of the master node, wherein the data persisted to the non-volatile storage medium of the master node and the data persisted to the non-volatile storage medium of each slave node are mutual copies.

2. The method of claim 1, wherein, After persisting the data stored in the predetermined memory slot to the non-volatile storage medium, the method further comprises: The master node generates a commit confirmation response according to the received local metadata, and each slave node generates a commit confirmation response according to the received local metadata and sends it to the master node; The master node generates an overall commit confirmation response for the global metadata according to the commit confirmation responses of each slave node and the master node, and sends it to the client; After receiving the overall commit confirmation response, the client sets the predetermined memory slot of the master node and the predetermined memory slot of each slave node to an idle state.

3. The method of claim 1, wherein, Before the client writes data corresponding to the plurality of I / O requests into the predetermined memory slot of the master node and the predetermined memory slot of each slave node, the method further comprises: The user layer generates a plurality of logically associated I / O requests for the data to be stored in multiple copies, and sends the generated I / O requests to the client; The client receives the I / O requests sent by the user layer, and for each I / O request, records the cache location of the data of the I / O request and the persistent information of the data of the I / O request, wherein the data of the I / O request is a data block, and the data of the plurality of logically associated I / O requests is persisted according to the persistent information.

4. The method of claim 3, wherein, Before the client writes data corresponding to the plurality of I / O requests into the predetermined memory slot of the master node and the predetermined memory slot of each slave node, the method further comprises: The client processes any I / O request received from the user layer into an I / O data list, and adds the processed I / O data list to an I / O sending queue; the I / O data list records a metadata part and a data part; the metadata part records the persistent information of the I / O request; and the data part records the cache location of the data of the I / O request; The client allocates the plurality of I / O data lists in the I / O sending queue to the predetermined memory slot of the master node and the predetermined memory slot of each slave node; The client generates a master node storage copy according to the allocated predetermined memory slot of the master node, and adds the processed master node storage copy to a master node copy data buffer; and The client generates a slave node storage copy according to the allocated predetermined memory slot of each slave node, and adds the processed slave node storage copy to a slave node copy data buffer. The client generates the storage replicas of each slave node according to the predetermined memory slots of each slave node after the allocation, and increases the processed storage replicas of each slave node to the replica data buffer of each slave node.

5. The method of claim 3, wherein, The client writes the data corresponding to the plurality of I / O requests to the predetermined memory slots of the master node and the predetermined memory slots of each slave node, comprising: The client writes the data corresponding to the plurality of I / O requests recorded in the master node storage replica from the cache location to the predetermined memory slots of the master node in batches through the RDMAWrite operation; The client writes the data corresponding to the plurality of I / O requests recorded in the storage replica of each slave node from the cache location to the predetermined memory slots of each slave node in batches through the RDMAWrite operation.

6. The method of claim 5, wherein, The client generates the global metadata according to the persistent information corresponding to the plurality of I / O requests, the predetermined memory slots of the master node and the predetermined memory slots of each slave node, comprising: The client obtains the persistent information of the plurality of I / O requests recorded in the I / O data list and the predetermined memory slots of the master node allocated for the plurality of I / O requests, and generates the local metadata for the master node; The client obtains the persistent information of the plurality of I / O requests recorded in the I / O data list and the predetermined memory slots of each slave node allocated for the plurality of I / O requests, and generates the local metadata for each slave node; The client combines the local metadata for the master node and the local metadata for each slave node into the global metadata.

7. The method of claim 6, wherein The slave node uses the received local metadata to persist the data stored in the predetermined memory slots of the slave node to the non-volatile storage medium of the slave node, comprising: The slave node receives the local metadata, determines the persistent information corresponding to the plurality of I / O requests and the predetermined memory slots of the slave node, and uses the persistent information to persist the data stored in the predetermined memory slots of each slave node to the non-volatile storage medium of each slave node; The master node uses the received local metadata to persist the data stored in the predetermined memory slots of the master node to the non-volatile storage medium of the master node, comprising: The master node receives the local metadata, determines the persistent information corresponding to the plurality of I / O requests and the predetermined memory slots of the master node, and uses the persistent information to persist the data stored in the predetermined memory slots of the master node to the non-volatile storage medium of the master node.

8. The method of claim 5, wherein The number of master node storage replicas processed by the client in one RDMAWrite operation is the same as the number of slave node storage replicas processed by the client in one RDMAWrite operation; The number of master node storage replicas or slave node storage replicas processed by the client in one RDMAWrite operation is determined by the minimum value of the predefined maximum batch size, the number of free memory slots available to the master node and each slave node.

9. The method of claim 3, wherein, Before the client writes the data corresponding to the plurality of I / O requests to the predetermined memory slots of the master node and the predetermined memory slots of each slave node, further comprising: The master node or each slave node allocates a predetermined memory pool, divides the predetermined memory pool into a plurality of memory slots with a specified capacity, allocates a unique slot identifier to each memory slot in the predetermined memory pool, and records each memory slot as an idle state or an occupied state; The predetermined memory pool is registered as an RDMA accessible memory region, and a bitmap is generated for the predetermined memory pool; After the client receives an I / O request sent by the user layer, a plurality of nodes in a distributed system are specified to participate in replica update services, one of the plurality of nodes is specified as a master node, and the other nodes are specified as slave nodes; When the client first connects to the master node or each slave node, the master node or each slave node scans the bitmap for idle memory slots, allocates a specified number of memory slots in the predetermined memory pool to the client, and returns the slot identifiers and RDMA access information of the specified number of memory slots to the client.

10. The method of claim 9, wherein, Further comprising: When the client maintains a connection with the master node or each slave node, The master node or each slave node calculates a stress index of the client according to the number of RDMAWrite operations initiated by the client per unit time, the average filling time ratio of the current slot of the client, and the number of times the client triggers expansion within a preset time; According to the stress index, a memory slot expansion strategy for the client is executed; The master node or each slave node calculates a contraction index of the client according to the idle time ratio of the current slot of the client and the minimum time interval between two contraction operations of the client, and executes a memory slot contraction strategy for the client according to the contraction index.

Images