Distributed consistency method and device, equipment, storage medium and product

Abstract

The invention discloses a distributed consistency method, device and equipment, a storage medium and a product. Comprising the steps that in the process that a to-be-added node is added into a high-availability cluster, a main node and the to-be-added node exchange remote addresses of remote direct memory access of the main node and the to-be-added node; a two-stage submission protocol is initiated, and a main node encapsulates data sent by a client and writes the data into a remote address of remote direct content access of a newly added node; the newly added node polls and checks that a new distributed consistency sending message exists on a local address by receiving a coroutine of the remote direct memory access request, and then writes a processed reply message into a remote address of the remote direct memory access of the main node; and the main node polls and detects that a new distributed consistency reply message exists on the local address by replying the coroutine of the remote direct memory access request, and then processes the new distributed consistency reply message. The method can effectively reduce the consistency delay.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of data consistency, and in particular, to a distributed consistency method, device, equipment, storage medium and product. Background Art

[0002] The Paxos consensus algorithm is the theoretical basis of the distributed consistency algorithm, which is used to achieve data consistency among the majority of replicas in a distributed environment. The Paxos consensus algorithm mainly relies on the message passing model and uses a specific communication protocol to achieve consistency among distributed nodes.

[0003] The MGR Paxos algorithm optimizes the Paxos consensus algorithm. It mainly reduces the prepare phase and the log write disk latency by designating a master node, thereby improving the performance.

[0004] However, both the Paxos consensus algorithm and the MGR Paxos algorithm adopt the core communication protocol of the Internet. For example, the Transmission Control Protocol (TCP). Since the distributed consistency algorithm based on the TCP / IP communication protocol depends on the network layer of the operating system kernel during the communication process, it will cause TCP protocol stack latency. Summary of the Invention

[0005] The present invention provides a distributed consistency method, device, equipment, storage medium and product to solve the problem that the existing distributed consistency algorithm has a sharp increase in consistency latency due to the communication process depending on the network layer of the operating system kernel.

[0006] According to one aspect of the present invention, a distributed consistency method is provided, including:

[0007] During the process of a node to be added joining a highly available cluster, the master node and the node to be added exchange their remote direct memory access (RDMA) remote addresses, and the remote addresses are used to form a security mechanism for remote direct memory access;

[0008] Initiate a two-phase commit protocol. The master node encapsulates the data sent by the client and writes it to the remote address of the remote direct content access of the newly added node, where the node to be added becomes the newly added node after joining the highly available cluster;

[0009] The newly added node polls and checks whether there is a new distributed consistency send message on the local address through a coroutine that receives remote direct memory access requests, and then writes the processed reply message to the remote address of the remote direct memory access of the master node;

[0010] After the master node polls and detects a new distributed consistency reply message on the local address through the coroutine that replies to the remote direct memory access request, it processes the new distributed consistency reply message.

[0011] According to another aspect of the present invention, there is provided a distributed consistency device, including:

[0012] An exchange module 110, configured to exchange the remote addresses of their own remote direct memory access between the master node and the node to be joined during the process of the node to be joined joining the highly available cluster, where the remote addresses are used to constitute a security mechanism for remote direct memory access;

[0013] A first writing module 120, configured to initiate a two-phase commit protocol, and the master node encapsulates the data sent by the client and writes it to the remote address of the remote direct content access of the newly joined node, where the node to be joined serves as the newly joined node after joining the highly available cluster;

[0014] A second writing module 130, configured to, after the newly joined node polls and checks for a new distributed consistency send message on the local address through the coroutine that receives the remote direct memory access request, write the processed reply message to the remote address of the remote direct memory access of the master node;

[0015] A processing module 140, configured to, after the master node polls and detects a new distributed consistency reply message on the local address through the coroutine that replies to the remote direct memory access request, process the new distributed consistency reply message.

[0016] According to another aspect of the present invention, there is provided an electronic device, where the electronic device includes: at least one processor;

[0017] And a memory communicatively connected to the at least one processor;

[0018] Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed consistency method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, there is provided a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the distributed consistency method according to any embodiment of the present invention when executed by a processor.

[0020] According to another aspect of the present invention, there is provided a computer program product, which includes a computer program that, when executed by a processor, implements the distributed consistency method according to any embodiment of the present invention.

[0021] The technical solution of the embodiment of the present invention uses the unilateral read and write operations of remote direct memory access to solve the problem of a sharp increase in consistency latency, and achieves the beneficial effect of effectively reducing consistency latency.

[0022] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Description of the Drawings

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0024] Figure 1 It is a schematic flowchart of a distributed consistency method provided for Embodiment 1 of the present invention;

[0025] Figure 2 It is a schematic flowchart of a distributed consistency method provided for Embodiment 2 of the present invention;

[0026] Figure 3 It is a schematic structural diagram of a distributed consistency device provided for Embodiment 3 of the present invention;

[0027] Figure 4 It is a schematic structural diagram of an electronic device for a distributed consistency method according to an embodiment of the present invention. Detailed Embodiments

[0028] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. It should be understood that the various steps recorded in the method embodiments of the present invention can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0029] As used herein, the term "including" and its variations are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0031] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] Embodiment 1

[0034] Figure 1 It is a schematic flowchart of a distributed consistency method provided for Embodiment 1 of the present invention. This method is applicable to the situation of achieving data consistency among majority replicas in a distributed environment. This method can be executed by a distributed consistency device, where the device can be implemented by software and / or hardware and is generally integrated on an electronic device. In this embodiment, the electronic device includes but is not limited to: computer devices.

[0035] As Figure 1 shown, a distributed consistency method provided for Embodiment 1 of the present invention includes the following steps:

[0036] S110. During the process of a node to be joined joining a highly available cluster, the master node and the node to be joined exchange their remote direct memory access (RDMA) remote addresses, and the remote addresses are used to constitute a security mechanism for remote direct memory access.

[0037] Among them, Remote Direct Memory Access (RDMA) is a network technology that allows a computer to directly access the memory of another computer without the intervention of the operating system kernel and the central processing unit.

[0038] Among them, the remote address consists of raddr and rkey. Raddr and rkey are used together to form the security mechanism for RDMA remote memory access; raddr represents the memory address of the remote machine and is used to directly read and write the memory of the other party without the participation of the CPU; rkey is the security access credential in the RDMA technology and is equivalent to the "password" of the remote memory. It is used to authorize other hosts to operate on the locally registered memory area through RDMA.

[0039] In this embodiment, after the high-availability cluster selects the master node, it can initiate a cluster formation command to the master node and an add-to-cluster command to the new node to be added; the node to be added still uses the core communication protocol of the Internet to send an add request to the master node, and at this time, the initialization operation of RDMA is performed; after the master node monitors the add request, it executes the distributed consistency protocol to enable each node in the high-availability cluster to reach the consistency protocol; the master node sends its own RDMA remote address to the node to be added, the node to be added passes its own RDMA remote address to the master node, and the master node sends a snapshot of the status information of the high-availability cluster to the node to be added, so that the node to be added can determine the node pair according to the snapshot and realize the exchange of the RDMA remote addresses of the two nodes in each node pair.

[0040] In this embodiment, each time the master node monitors an add request, it creates a coroutine for receiving core communication protocol requests and a coroutine for receiving direct memory access requests. The coroutine for receiving remote direct memory access requests is used to receive remote direct memory access requests of the consistency protocol.

[0041] Among them, each time a new node, that is, the node to be added, wants to join the high-availability cluster, it will send an add request to the master node. Each time the master node monitors the add request, in addition to creating a coroutine acceptor_learner_task for receiving core communication protocol requests, it can also additionally create a coroutine rdma_learner_task for receiving direct memory access requests; acceptor_learner_task is used to process requests sent using the core communication protocol of the Internet, such as TCP / IP requests; rdma_learner_task is used to receive rdma requests of the consistency protocol.

[0042] In this embodiment, each time the master node monitors a join request, it creates a coroutine for replying to the core communication protocol request and a coroutine for replying to the direct memory access request. The coroutine for replying to the direct memory access request is used to process the remote direct memory access request of the coherence protocol.

[0043] Among them, each time a new node, that is, a node to be joined, wants to join the highly available cluster, it sends a join request to the master node. Each time the master node monitors a join request, in addition to creating a coroutine reply_handler_task for replying to the core communication protocol request, it can also additionally create a coroutine reply_rdma_handler_task for replying to the direct memory access request; reply_handler_task is used to process requests sent using the Internet core communication protocol, such as TCP / IP requests; reply_rdma_handler_task is used to process rdma requests sent using the coherence protocol.

[0044] S120. Initiate the two-phase commit protocol. The master node encapsulates the data sent by the client and writes it to the remote address of the remote direct content access of the newly joined node.

[0045] Among them, after the node to be joined joins the highly available cluster, it serves as the newly joined node.

[0046] Among them, the two-phase commit protocol (Two-Phase Commit Protocol, 2PC) is the core protocol for ensuring transaction atomicity in a distributed system, and is used to ensure that a transaction across multiple nodes is either all committed or all rolled back.

[0047] In this embodiment, the client sends an SQL request to the database read / write instance node, that is, the master node. The proposer_task coroutine fetches the data sent by the client from the pipeline and initiates the two-phase commit protocol. This process will loop through each node in the highly available cluster and wake up the corresponding sender_task of the node.

[0048] In this embodiment, the sender_task coroutine of the master node checks whether the RDMA flag bit is set. If so, it directly encapsulates the data sent by the client and writes it to the RDMA remote address of the newly joined node in the IBV_WR_RDMA_WRITE mode, and checks whether the write to the completion queue is successful. If the write is not successful, it still uses the Internet core communication protocol for communication.

[0049] S130. After the newly added node polls and checks whether there is a new distributed consistency send message at the local address through the coroutine that receives the remote direct memory access request, it writes the processed reply message to the remote address of the remote direct memory access of the master node.

[0050] Among them, the coroutine of the newly added node that receives the remote direct memory access request periodically polls whether there is a new paxos message at the configured local address. If so, it reads the message into the processing flow, returns the processed reply message, and writes the reply message to the remote address of the remote direct memory access of the master node to be added in the form of IBV_WR_RDMA_WRITE.

[0051] S140. After the master node polls and detects that there is a new distributed consistency reply message at the local address through the coroutine that replies to the remote direct memory access request, it processes the new distributed consistency reply message.

[0052] Among them, the master node polls and checks whether there is a new paxos message at the configured local address through the coroutine reply_rdma_handler_task that replies to the remote direct memory access request; if not, it means that no new data has arrived in this polling; if so, it reads the new paxos message of the specified length into the processing flow and clears this section of memory.

[0053] Furthermore, each node in the highly available cluster maintains a data structure, which is used to maintain the message written to the remote address of the remote direct memory access of the node in a preset writing manner; among them, the node that receives the message obtains the message length from the remote address of the received remote direct memory access, and dynamically adjusts the receive buffer offset according to the message length.

[0054] Among them, each node in the highly available cluster maintains a data structure, which is used to maintain the writing to the remote address of the remote direct memory access of the node in the form of IBV_WR_RDMA_WRITE. The node that receives the message can obtain the message length from the content parsed from the remote address of the remote direct memory access, and then the physical address for receiving the next message can increase the corresponding offset.

[0055] So far, a round of RDMA-based message processing mechanism is completed. It should be noted that after using the ibv_post_send function of RDMA, after the requesting end receives the confirmation message from the network card, it generates a completion queue entry and places it in the completion queue, and the task is completed.

[0056] A distributed consistency method provided in Embodiment 1 of the present invention. First, during the process of a node to be added joining a highly available cluster, the master node and the node to be added exchange their remote direct memory access (RDMA) remote addresses, and the remote addresses are used to constitute the security mechanism of remote direct memory access. Then, a two-phase commit protocol is initiated. The master node encapsulates the data sent by the client and writes it to the remote address of the newly added node's remote direct content access. After that, the newly added node, through the coroutine that receives the remote direct memory access request, polls to check if there is a new distributed consistency send message on the local address, and then writes the processed reply message to the remote address of the master node's remote direct memory access. Finally, the master node, through the coroutine that replies to the remote direct memory access request, polls to detect if there is a new distributed consistency reply message on the local address, and processes the new distributed consistency reply message. In the above method, each node processes requests through the coroutine that receives the remote direct memory access request and the coroutine that replies to the remote direct memory access request, without the intervention of the operating system kernel and CPU; by bypassing the traditional TCP / IP protocol stack, the latency is greatly reduced and the throughput is increased.

[0057] Embodiment 2

[0058] Figure 2 It is a schematic flowchart of a distributed consistency method provided in Embodiment 2 of the present invention. Embodiment 2 is optimized on the basis of the above embodiments. For the content not detailed in this embodiment, please refer to Embodiment 1.

[0059] As Figure 2 shown, a distributed consistency method provided in Embodiment 2 of the present invention includes the following steps:

[0060] S210. The node to be added sends a join request to the master node using the core communication protocol of the Internet and performs the initialization operation of remote direct memory access.

[0061] Among them, the highly available cluster can initiate a cluster formation command to the selected master node and a join cluster command to the node to be added. After receiving the command, the node to be added can send a join request of the add_node_type type to the master node using the TCP / IP protocol and perform the RDMA initialization operation.

[0062] S220. After the master node listens to the join request through the coroutine that receives the core communication protocol request, it executes the distributed consistency protocol on the join request. After the protocol is reached, each node in the highly available cluster executes the distributed consistency protocol.

[0063] Among them, each node in the highly available cluster executes the paxos consistency protocol to add the node information to the cluster state information, and the cluster state information records the survival status of each node.

[0064] S230. After the master node detects that the status information of the highly available cluster has changed, it sends the remote address of the remote direct memory access of the master node to the node to be joined.

[0065] Specifically, after the master node detects that the status information of the highly available cluster has changed through the health check coroutine, it sends a health check message to the node to be joined through the coroutine for sending the core communication protocol. The health check message includes the remote address of the remote direct memory access of the master node.

[0066] Further, sending a health check message to the node to be joined through the coroutine for sending the core communication protocol includes: the coroutine for sending the core communication protocol establishes a communication connection with the node to be joined, creates a reliable connection protection domain, creates a completion queue for the reliable connection, creates a work completion array, registers remote access to local memory, adds the remote address of the remote direct memory access of the master node to the health check message, and sends the health check message to the node to be joined.

[0067] S240. The node to be joined saves the remote address of the remote direct memory access of the master node into its own memory, passes the remote address of the remote direct memory access of the node to be joined to the master node, and requests to obtain the status information of the highly available cluster.

[0068] Specifically, after the node to be joined receives the health check message through the coroutine for receiving the core communication protocol request, it responds to the health check message, extracts the remote address of the remote direct memory access of the master node from the health check message, and saves the remote address of the remote direct memory access of the master node into its own memory; at the same time, the node to be joined creates a reliable connection protection domain, a completion queue for the reliable connection, and a work completion array, and registers remote access to local memory, adds the remote address of the remote direct memory access of the master node to the response to the health check message, and sends the health check message to the master node; at the same time, the node to be joined requests to obtain the status information of the highly available cluster.

[0069] S250. The master node saves the remote address of the remote direct memory access of the node to be joined into its own memory, and sends a snapshot of the status information of the highly available cluster to the node to be joined.

[0070] Among them, after the master node receives, through the coroutine that replies to the core communication protocol request, all node information in the highly available cluster obtained by the node to be joined, it obtains the remote direct memory access (RDMA) remote address of the node to be joined from the response heartbeat message and saves it in the memory; it sends a snapshot of the status information of the highly available cluster to the node to be joined, and subsequent messages can directly write messages to the RDMA remote address of the node to be joined by using RDMA.

[0071] S260. After the node to be joined receives the snapshot through the coroutine that replies to the core communication protocol request, for each pair of nodes, it executes steps S230 and S240 to implement the exchange of the RDMA remote addresses of the two nodes in each pair of nodes.

[0072] Among them, a pair of nodes consists of the node to be joined and each node in the highly available cluster.

[0073] In this embodiment, each time the master node monitors an addition request, it creates a coroutine for receiving core communication protocol requests and a coroutine for receiving direct memory access requests. The coroutine for receiving RDMA requests is used to receive RDMA requests of the consistency protocol; each time the master node monitors an addition request, it creates a coroutine for replying to core communication protocol requests and a coroutine for replying to direct memory access requests. The coroutine for replying to direct memory access requests is used to process RDMA requests of the consistency protocol.

[0074] S270. Initiate a two-phase commit protocol. The master node encapsulates the data sent by the client and writes it to the RDMA remote address of the newly joined node.

[0075] Among them, after the node to be joined joins the highly available cluster, it serves as the newly joined node.

[0076] S280. The newly joined node, through the coroutine for receiving RDMA requests, polls to check if there is a new distributed consistency send message on the local address, and then writes the processed reply message to the RDMA remote address of the master node.

[0077] S290. The master node, through the coroutine for replying to RDMA requests, polls to detect if there is a new distributed consistency reply message on the local address, and then processes the new distributed consistency reply message.

[0078] A distributed consistency method provided in Embodiment 2 of the present invention can significantly reduce latency and improve throughput by using this method.

[0079] Based on the technical solutions of the above embodiments, the present invention provides a specific implementation manner.

[0080] As a specific implementation of this embodiment, a distributed consistency method includes a cluster formation process and a protocol negotiation process.

[0081] Among them, the cluster formation process includes the following steps:

[0082] 1. The highly available cluster selects the master node node0 according to the transaction unique identifier.

[0083] 2. The highly available cluster sends a cluster formation command to node0.

[0084] 3. The highly available cluster sends a join cluster command to node1.

[0085] 4. After node1 joins the highly available cluster, it sends a join request of the add_node_type type to node0. At this time, the TCP / IP protocol is still used, and the RDMA initialization operation is performed.

[0086] 5. When node0 listens to the join request, the coroutine that listens to the peer request is still the acceptor_learner_task, that is, the coroutine that receives the core communication protocol request. The paxos consistency protocol is executed for this request. After the protocol is negotiated, each node in the highly available cluster executes the consistency protocol.

[0087] 6. When the alive_task of node0, that is, the liveness detection coroutine, detects that the cluster status information has changed, it sends a liveness message to node1. The liveness message is sent by the sender_task of node0, that is, the coroutine that sends the core communication protocol. This coroutine first establishes a TCP connection with node1 and sends the liveness message. At this time, a reliable connection protection domain will also be created, a completion queue for creating a reliable connection will be created, a work completion array will be created, and remote access to local memory will be registered. The (raddr, rkey) that allows node1 to directly access node0 is included in the liveness request message.

[0088] 7. After the acceptor_learner_task of node1, that is, the coroutine that receives the core communication protocol request, receives the liveness message sent by node0, node1 responds to the liveness message, extracts the (raddr, rkey) of node0 from the liveness message, and saves it to the memory of node1. At the same time, node1 creates a reliable connection protection domain, creates a reliable connection completion queue, creates a work completion array, registers remote access to local memory, includes the (raddr, rkey) that allows node0 to directly access node1 in the response liveness message, and requests to obtain the messages of all nodes in the highly available cluster from node0.

[0089] 8. After node0 receives a message from node1 requesting to obtain all nodes in the high-availability cluster in reply_handler_task, the coroutine that replies to core communication protocol requests, it obtains node1's (raddr, rkey) from the response detection message and saves it to node0's memory. It then sends a snapshot of node0's cluster to node1. Subsequent messages can be written directly to node1's raddr using RDMA.

[0090] 9. After node1 receives the status information of all nodes in the cluster from node0, it synchronizes the (raddr, rkey) of the peer node using steps 6 to 8 for the node pairs in the cluster.

[0091] 10. Every time node0 listens to a new node joining request, in addition to creating the acceptor_learner_task coroutine, it also creates the rdma_learner_task coroutine, which is a coroutine that receives direct memory access requests. This coroutine is used to receive RDMA requests of the consistency protocol.

[0092] 11. Every time node0 listens to a new node joining request, in addition to creating the reply_handler_task coroutine, it also creates the reply_rdma_handler_task coroutine, which is the coroutine that replies to the direct memory access request. This coroutine is used to process the RDMA request.

[0093] The agreement-making process includes the following steps:

[0094] 1. The client sends a SQL request to the database read / write instance node node0. The proposer_task coroutine takes the data sent by the client from the pipeline and initiates a two-phase commit protocol. This process loops through each node in the high-availability cluster and wakes up the corresponding sender_task coroutine of the node.

[0095] 2. The sender_task coroutine of node0 checks whether the RDMA flag is set. If so, it encapsulates the data and writes it to the raddr of node1 using the IBV_WR_RDMA_WRITE method, and checks whether the completion queue is written successfully. If not, it still uses TCP for communication.

[0096] 3. The rdma_learner_task coroutine of node1 periodically polls the configured local address for new paxos messages. If so, it reads the message for processing and returns a processed reply message, which is written to the raddr of node0 in the form of IBV_WR_RDMA_WRITE.

[0097] 4. The reply_rdma_handler_task polls to check if there is a new paxosreply message on the configured local address. If so, it receives the message for message processing. This node can retain the offset of the last polled message read, and the next poll reads from after this offset. First, it checks if the message header after the offset contains a valid data length. If not, it means no new data has arrived in this poll. If so, it reads a message of the specified length for the processing flow and clears this section of memory.

[0098] 5. Each node in the cluster maintains a data structure that is used to maintain the (raddr, rkey) where messages are written to the node in the IBV_WR_RDMA_WRITE manner. The node that receives the message can obtain the message length from the content parsed from RDMA, and then the physical address for the next message reception increases by the corresponding offset. It should be noted that this memory is recycled and can be used for the message flow control mechanism.

[0099] So far, a round of the RDMA-based message processing mechanism is completed. It should be particularly noted that after using the ibv_post_send function of RDMA, after the requesting end receives the network card ACK, it generates a completion queue entry and places it in the completion queue, and the task is completed.

[0100] Embodiment 3

[0101] Figure 3 The following is a schematic structural diagram of a distributed consistency device provided in Embodiment 3 of the present invention. This device is applicable to the situation of achieving data consistency among majority replicas in a distributed environment, and this device can be implemented by software and / or hardware and is generally integrated on an electronic device.

[0102] As Figure 3 shown, the device includes: a switching module 110, a first writing module 120, a second writing module 130, and a processing module 140.

[0103] The switching module 110 is used to exchange the remote addresses of their own remote direct memory access between the master node and the node to be added during the process of the node to be added joining the highly available cluster, and the remote addresses are used to constitute the security mechanism of the remote direct memory access;

[0104] The first writing module 120 is used to initiate a two-phase commit protocol. The master node encapsulates the data sent by the client and writes it to the remote address of the remote direct content access of the newly added node, where the node to be added becomes the newly added node after joining the highly available cluster;

[0105] A second writing module 130, configured to, after a newly added node polls and checks that there is a new distributed consistency transmission message at a local address through a coroutine that receives a remote direct memory access request, write a processed reply message to a remote address of the remote direct memory access of the primary node;

[0106] A processing module 140, configured to, after the primary node polls and detects that there is a new distributed consistency reply message at a local address through a coroutine that replies to a remote direct memory access request, process the new distributed consistency reply message.

[0107] In this embodiment, the device first exchanges the remote addresses of its own remote direct memory access between the primary node and the node to be added during the process of the node to be added joining the highly available cluster through the exchange module 110, and the remote addresses are used to form a security mechanism for remote direct memory access; then initiates a two-phase commit protocol through the first writing module 120, and the primary node encapsulates the data sent by the client and writes it to the remote address of the remote direct content access of the newly added node, where the node to be added becomes the newly added node after joining the highly available cluster; then in the second writing module 130, the newly added node polls and checks that there is a new distributed consistency transmission message at a local address through a coroutine that receives a remote direct memory access request, and writes the processed reply message to the remote address of the remote direct memory access of the primary node; finally, in the processing module 140, the primary node polls and detects that there is a new distributed consistency reply message at a local address through a coroutine that replies to a remote direct memory access request, and processes the new distributed consistency reply message.

[0108] This embodiment provides a distributed consistency device, which can effectively reduce consistency latency.

[0109] Further, the exchange module 110 includes:

[0110] A first execution sub-module, configured to enable the node to be added to send a join request to the primary node using the core communication protocol of the Internet and perform initialization operations for remote direct memory access;

[0111] A second execution sub-module, configured to, after the primary node monitors the join request through a coroutine that receives the core communication protocol request, execute a distributed consistency protocol on the join request, and after the protocol is reached, each node in the highly available cluster executes the distributed consistency protocol;

[0112] A sending sub-module, configured to, after the primary node detects that the status information of the highly available cluster has changed, send the remote address of the remote direct memory access of the primary node to the node to be added;

[0113] A transfer sub-module, configured to enable the node to be added to save the remote address of the remote direct memory access of the master node into its own memory, transfer the remote address of the remote direct memory access of the node to be added to the master node, and request to obtain the status information of the highly available cluster;

[0114] A saving sub-module, configured to enable the master node to save the remote address of the remote direct memory access of the node to be added into its own memory, and send a snapshot of the status information of the highly available cluster to the node to be added;

[0115] An interaction sub-module, configured to enable the node to be added to exchange the remote addresses of the remote direct memory access of the two nodes in each node pair for each node pair after receiving the snapshot through the coroutine that replies to the core communication protocol request, where the node pair is composed of the node to be added and each node in the highly available cluster.

[0116] Based on the above optimization, the sending sub-module is specifically configured to: after the master node detects a change in the status information of the highly available cluster through the liveness detection coroutine, send a liveness detection message to the node to be added through the coroutine that sends the core communication protocol, where the liveness detection message includes the remote address of the remote direct memory access of the master node.

[0117] Among them, sending a liveness detection message to the node to be added through the coroutine that sends the core communication protocol includes: the coroutine that sends the core communication protocol establishes a communication connection with the node to be added, creates a reliable connection protection domain, creates a completion queue for the reliable connection, creates a work completion array, registers remote access to local memory, adds the remote address of the remote direct memory access of the master node to the liveness detection message, and sends the liveness detection message to the node to be added.

[0118] Based on the above technical solution, the transfer sub-module specifically includes:

[0119] A saving unit, configured to enable the node to be added to respond to the liveness detection message after receiving the liveness detection message through the coroutine that receives the core communication protocol request, extract the remote address of the remote direct memory access of the master node from the liveness detection message, and save the remote address of the remote direct memory access of the master node into its own memory;

[0120] An adding unit, configured to enable the node to be added to create a reliable connection protection domain, a completion queue for the reliable connection, and a work completion array, register remote access to local memory, add the remote address of the remote direct memory access of the master node to the response to the liveness detection message, and send the liveness detection message to the master node;

[0121] An obtaining unit, configured to enable the node to be added to request to obtain the status information of the highly available cluster.

[0122] Further, the device further includes a first creation module and a second creation module;

[0123] The first creation module is configured to, each time the master node monitors a join request, create a coroutine for receiving a core communication protocol request and a coroutine for receiving a direct memory access request, and the coroutine for receiving a remote direct memory access request is configured to receive a remote direct memory access request of a consistency protocol;

[0124] The second creation module is configured to, each time the master node monitors a join request, create a coroutine for replying to a core communication protocol request and a coroutine for replying to a direct memory access request, and the coroutine for replying to a direct memory access request is configured to process a remote direct memory access request of a consistency protocol.

[0125] Further, each node in the high-availability cluster maintains a data structure for maintaining a remote address of a remote direct memory access in which a message is written in a preset writing manner; wherein, the node that receives the message obtains the message length from the remote address of the received remote direct memory access, and dynamically adjusts the receive buffer offset according to the message length.

[0126] The above-mentioned distributed consistency device can execute the distributed consistency method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0127] Embodiment 4

[0128] Figure 4 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described herein and / or claimed.

[0129] As Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0130] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0131] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a distributed consistency method.

[0132] In some embodiments, a distributed consistency method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the distributed consistency method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute a distributed consistency method by any other appropriate means (e.g., by means of firmware).

[0133] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0134] In some embodiments, a distributed consistency method can be implemented as a computer program that is tangibly embodied in a computer program product, the computer program implementing, when executed by a processor, a distributed consistency method of the present invention. A computer program product can be understood as a software product that mainly realizes its solution through a computer program. The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0137] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0138] The computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0139] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0140] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A distributed consistency method, characterized in that, The method includes: During the process of a node to be added joining a highly available cluster, the master node and the node to be added exchange their remote direct memory access (RDMA) remote addresses, and the remote addresses are used to form a security mechanism for remote direct memory access; Initiate a two-phase commit protocol. The master node encapsulates the data sent by the client and writes it to the remote address of the remote direct content access of the newly added node. After the node to be added joins the highly available cluster, it serves as the newly added node; The newly added node, through the coroutine for receiving RDMA requests, polls to check if there is a new distributed consistency send message on the local address, and then writes the processed reply message to the remote address of the master node's remote direct memory access; The master node, through the coroutine for replying to RDMA requests, polls to detect if there is a new distributed consistency reply message on the local address, and then processes the new distributed consistency reply message.

2. The method according to claim 1, wherein During the process of a node to be added joining a highly available cluster, the master node and the node to be added exchange the remote addresses of remote direct memory access, including the following steps: Step 1: The node to be added sends a join request to the master node using the core communication protocol of the Internet and performs the initialization operation of remote direct memory access; Step 2: After the master node listens to the join request through the coroutine for receiving core communication protocol requests, it executes the distributed consistency protocol for the join request. After the protocol is reached, each node in the highly available cluster executes the distributed consistency protocol; Step 3: After the master node detects that the status information of the highly available cluster has changed, it sends the remote address of the master node's remote direct memory access to the node to be added; Step 4: The node to be added saves the remote address of the master node's remote direct memory access in its own memory, passes the remote address of its own remote direct memory access to the master node, and requests to obtain the status information of the highly available cluster; Step 5: The master node saves the remote address of the node to be added's remote direct memory access in its own memory and sends a snapshot of the status information of the highly available cluster to the node to be added; Step 6: After the node to be added receives the snapshot through the coroutine for replying to core communication protocol requests, for each pair of nodes, it executes Step 3 and Step 4 to implement the exchange of the remote addresses of remote direct memory access between the two nodes in each pair of nodes. The pair of nodes consists of the node to be added and each node in the highly available cluster.

3. The method according to claim 2, wherein After the master node detects that the status information of the highly available cluster has changed and sends the remote address of the master node's remote direct memory access to the node to be added, it includes: After the master node detects that the status information of the highly available cluster has changed through the liveness detection coroutine, it sends a liveness detection message to the node to be added through the coroutine for sending the core communication protocol. The liveness detection message includes the remote address of the master node's remote direct memory access.

4. The method according to claim 3, characterized in that, Sending a liveness detection message to the node to be added through the coroutine for sending the core communication protocol includes: The coroutine that sends the core communication protocol establishes a communication connection with the to-be-joined node, creates a reliable connection protection domain, creates a completion queue for the reliable connection, creates a work completion array, registers remote access to local memory, adds the remote address of the remote direct memory access of the master node to the liveness probe message, and sends the liveness probe message to the to-be-joined node.

5. The method according to claim 2, wherein The to-be-joined node saves the remote address of the remote direct memory access of the master node into its own memory, and transfers the remote address of the remote direct memory access of the to-be-joined node and the status information of the highly available cluster to the master node, and requests to obtain the status information of the highly available cluster, including: After the to-be-joined node receives the liveness probe message through the coroutine that receives the core communication protocol request, it responds to the liveness probe message, extracts the remote address of the remote direct memory access of the master node from the liveness probe message, and saves the remote address of the remote direct memory access of the master node into its own memory; Meanwhile, the to-be-joined node creates a reliable connection protection domain, a completion queue for the reliable connection, and a work completion array, and registers remote access to local memory, adds the remote address of the remote direct memory access of the master node to the response to the liveness probe message, and sends the liveness probe message to the master node; Meanwhile, the to-be-joined node requests to obtain the status information of the highly available cluster.

6. The method according to claim 2, wherein The method further includes: Each time the master node monitors an addition request, it creates a coroutine that receives the core communication protocol request and a coroutine that receives the direct memory access request. The coroutine that receives the remote direct memory access request is used to receive the remote direct memory access request of the consistency protocol; Each time the master node monitors an addition request, it creates a coroutine that replies to the core communication protocol request and a coroutine that replies to the direct memory access request. The coroutine that replies to the direct memory access request is used to process the remote direct memory access request of the consistency protocol.

7. The method according to claim 1, characterized in that, Each node in the highly available cluster maintains a data structure respectively, and the data structure is used to maintain the message written to the remote address of the remote direct memory access of the node in a preset writing manner; Among them, the node that receives the message obtains the message length from the received remote address of the remote direct memory access, and dynamically adjusts the receiving buffer offset according to the message length.

8. A distributed consistency device, characterized in that, The device includes: A switching module, which is used to exchange the remote addresses of their own remote direct memory access between the master node and the to-be-joined node during the process of the to-be-joined node joining the highly available cluster. The remote addresses are used to form a security mechanism for the remote direct memory access; A first writing module, which is used to initiate a two-phase commit protocol. The master node encapsulates the data sent by the client and writes it to the remote address of the remote direct content access of the newly joined node. Among them, after the to-be-joined node joins the highly available cluster, it serves as the newly joined node; A second writing module, which is used for the newly joined node to poll and check whether there is a new distributed consistency send message on the local address through the coroutine that receives the remote direct memory access request, and then writes the processed reply message to the remote address of the remote direct memory access of the master node; A processing module is used for the master node to poll and detect that there is a new distributed consistency reply message on the local address through a coroutine that replies to a remote direct memory access request, and then process the new distributed consistency reply message.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed consistency method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the distributed consistency method according to any one of claims 1-7 when executed by a processor.

11. A computer program product, characterized in that, The computer program product includes a computer program, and the computer program implements the distributed consistency method according to any one of claims 1-7 when executed by a processor.

Images