RDMA-based consensus method and system supporting fine-grained fault tolerance

Abstract

The present invention provides a consensus method based on RDMA that supports fine-grained fault tolerance. The slave node will perform the following steps in sequence during the master node election phase: the slave node checks the terms of all nodes stored locally; for each other node, the candidate node writes its own term into the election-related metadata of the node through RDMA; for each RDMA read request in step 3, after the read request is returned from the node, the candidate node analyzes the election-related metadata; the candidate node counts all votes, and if the number of votes it obtains exceeds half of the total number of all nodes, then step 6 is executed, otherwise step 7 is executed; the candidate node selects the latest log copy from multiple log copies stored locally; and the master node election phase ends. The present invention ensures that log backup can be completed with one RDMA round trip by continuously sending two RDMA write requests and one RDMA read request for log items and log indexes without waiting, further reducing the network communication delay of log backup.

Description

Technical Field

[0001] The present invention relates to the technical field of consensus algorithms for distributed systems, and in particular to a consensus method and system supporting fine-grained fault tolerance based on RDMA. Background Art

[0002] With the continuous development of information technology, distributed systems have gradually been widely used. In distributed systems, servers cannot remain available for a long time, their networks may experience delays or failures, and may even cause downtime due to processor failures and other reasons.

[0003] In order to make the system fault-tolerant in such an environment, a commonly used method is state machine replication, which uses multiple servers to form a cluster, where each server is a node in the cluster. Through consensus algorithms such as Paxos and Raft, all nodes in the server cluster communicate with each other, and can reach a consensus on the order of a series of operations and execute all operations in the same order to ensure that the status of all nodes is the same. The consensus algorithm is usually divided into two stages. The first stage is the master node election stage, and the second stage is the log backup stage.

[0004] Because there are multiple nodes in the same state, the server cluster can still provide services normally through the remaining nodes even if some of the nodes have network failures or processor failures. It has a certain fault tolerance capability. Compared with a single server, the server cluster has higher availability.

[0005] RDMA (Remote Direct Memory Access) is a technology that bypasses the remote node processor to access its memory. RDMA not only saves a lot of processor resources, but also improves system throughput and reduces the system's network communication latency. It is suitable for building high-performance, low-latency storage networks. In recent years, there have been some works combining RDMA with consensus algorithms, which have enabled state machine replication to have higher performance and lower latency.

[0006] Since RDMA can bypass the remote node processor, if the consensus algorithm uses RDMA for network communication, a fine-grained fault-tolerant model can be used. Node network failure and processor failure can be regarded as two independent parts that do not affect each other. If the processor of a node fails, its RDMA part can still work normally, and other nodes can maintain communication with the node and operate the memory of the node with the processor failure through RDMA.

[0007] Currently, most consensus algorithms use a coarse-grained fault-tolerance model, treating nodes as faulty units. If any part of the node's network or processor fails, the node is considered to have failed.

[0008] Under the coarse-grained fault-tolerance model, since more than half of the nodes need to communicate normally to reach a consensus, for a cluster containing 2F+1 nodes, F node failures can be tolerated.

[0009] The fine-grained fault-tolerance model still requires a cluster of 2F+1 nodes to tolerate RDMA failures of F nodes. However, since processor failure does not affect normal communication of nodes, as long as one node processor is kept working normally, the entire system can reach consensus normally, that is, a cluster of 2F+1 nodes can tolerate processor failures of 2F nodes.

[0010] Most RDMA-based consensus algorithm designs can only support coarse-grained fault tolerance models, or only the log backup phase of the consensus algorithm supports fine-grained fault tolerance models, while the master node election phase can still only support coarse-grained fault tolerance models.

[0011] At present, there is no RDMA-based consensus algorithm design that can achieve low network latency and support fine-grained fault tolerance models in both stages of the consensus algorithm. Therefore, there is still a certain gap in the previous work on the research of consensus algorithm design. Summary of the invention

[0012] In view of the defects in the prior art, the object of the present invention is to provide a consensus method and system based on RDMA that supports fine-grained fault tolerance.

[0013] According to a consensus method based on RDMA supporting fine-grained fault tolerance provided by the present invention, when the master node is not faulty, the master node executes the log backup phase, responsible for backing up the log to the slave node log; when the slave node detects the master node failure, the slave node executes the master node election phase, electing a new master node from the candidate nodes; the master node, slave node, and candidate node are three roles played by the node, and the master node is unique; the slave node will sequentially perform the following steps in the master node election phase:

[0014] Step 1: Become a candidate node. The slave node checks the terms of all nodes saved locally, sets its own term to the latest term plus one, becomes a candidate node, and executes step 2 of requesting a vote.

[0015] Step 2: Request voting step. For each other node, the candidate node writes its own term into the election-related metadata of the node through RDMA; and executes the collection vote step 3;

[0016] Step 3: Collect voting steps. For each other node, the candidate node reads all election-related metadata stored on the node through RDMA; and executes step 4 to check the voting results;

[0017] Step 4: Check voting results. For each RDMA read request in step 3 of collecting votes, after the read request is returned from the node, the candidate node analyzes the election-related metadata; if the candidate node meets the conditions for being voted by the node, the number of votes obtained by the candidate node increases by one; and step 5 of counting voting results is executed;

[0018] Step 5: Counting the voting results. The candidate node counts all the votes. If the number of votes it receives exceeds half of the total number of nodes, it will execute Step 6 to become the master node. Otherwise, it will execute Step 7 to end the election.

[0019] Step 6: Becoming the master node: the candidate node selects the latest log copy from the multiple log copies saved locally, and copies the latest log copy to the log copy belonging to the candidate node, updates the election-related metadata, and becomes the new master node; executes the election end step 7;

[0020] Step 7: The election ends, ending the master node election phase.

[0021] Preferably, the term refers to a time period during which there is at most one master node; a new term begins when a slave node initiates the election of a master node; and the term is represented by a monotonically increasing integer.

[0022] Preferably, the log and election-related metadata have copies on each node equal to the total number of nodes, wherein each copy belongs to one node; a node can only update one copy belonging to the node.

[0023] Preferably, the election-related metadata includes the term of office of the node and log metadata; the log metadata includes the term and index of the latest log item in the log.

[0024] Preferably, in step 2, the candidate node only writes the term to a copy of the election-related metadata belonging to the candidate node, and in step 3 of collecting votes, the candidate node reads all copies of the election-related metadata.

[0025] Preferably, the RDMA requests in step 3 and step 2 are sent continuously without waiting, requiring one RDMA round trip.

[0026] Preferably, the candidate node meets the conditions for being voted by the node, which means that the term of the candidate node is greater than the maximum value of all terms in the node, and the latest log of the candidate node is not older than the latest log of the node; the latest log of the node refers to the latest log of all node logs stored by the node.

[0027] Preferably, the node also performs the following steps in sequence during the log backup phase:

[0028] Step S1: backup step. For each slave node, the master node writes the new log items to be backed up into the slave node log via RDMA, and writes the term and index of the latest log item into the slave node election-related metadata via RDMA; executes the term collection step S2;

[0029] Step S2: collecting term steps, for each slave node, the master node reads all node terms saved by the slave node through RDMA; executing step S3 of checking backup results;

[0030] Step S3: Check the backup result step. For each RDMA read request in the term collection step S2, after the read request is returned, the master node analyzes the term obtained by reading; if the master node meets the backup success condition, the backup success number is increased by one; and the backup result statistics step S4 is executed;

[0031] Step S4: backup result statistics step, the master node counts all backup results, if the number of successful backups exceeds half of the total number of all nodes, the log item is successfully backed up, and the log submission step S5 is executed; otherwise, the log item backup fails, and the backup end step S6 is executed;

[0032] Step S5: Log submission step, the master node increases the submitted index of its own log by one; and performs backup and ends step S6;

[0033] Step S6: Backup end step, ending the log backup phase.

[0034] Preferably, the master node will only update a log copy and election-related metadata copy belonging to the master node on each node during the log backup phase; the implementation of the log is a circular memory buffer that occupies a fixed memory segment; the log contains many log items, and new log items are written at the end of the circular memory buffer; in the steps S1 and S2, RDMA writes the new log items that need to be backed up, RDMA writes the term and index of the latest log item in the slave node log, and RDMA reads the terms of all nodes saved by the slave node, and these three RDMA operations only require one RDMA round trip to complete; the committed index of the log refers to the index of the latest committed log item recorded in the log; log items with indexes less than the committed index are all committed log items; the backup success condition refers to the master node checking the terms of all nodes in the slave node; if the master node term is greater than or equal to the maximum value of all terms in the slave node, the backup success condition is met, otherwise the backup success condition is not met.

[0035] The present invention also provides a consensus system supporting fine-grained fault tolerance based on RDMA. When the master node is not faulty, the master node executes the log backup phase, responsible for backing up the log to the slave node log; when the slave node detects the master node failure, the slave node executes the master node election phase, electing a new master node from the candidate nodes; the master node, slave node, and candidate node are three roles played by the node, and the master node is unique; the slave node includes the following modules in the master node election phase:

[0036] Module M1: Becoming a candidate node step, the slave node checks the terms of all nodes saved locally, sets its own term to the latest term plus one, becomes a candidate node, and executes the request voting module M2;

[0037] Module M2: Request voting step. For each other node, the candidate node writes its own term into the election-related metadata of the node through RDMA; executes the collection vote module M3;

[0038] Module M3: Collect voting steps. For each other node, the candidate node reads all election-related metadata stored by the node through RDMA; executes the voting result check module M4;

[0039] Module M4: Check voting results step. For each RDMA read request in the voting collection module M3, after the read request is returned from the node, the candidate node analyzes the election-related metadata; if the candidate node meets the conditions for being voted by the node, the number of votes obtained by the candidate node is increased by one; execute the voting result statistics module M5;

[0040] Module M5: voting result counting step. The candidate node counts all votes. If the number of votes it receives exceeds half of the total number of nodes, it will execute the master node module M6. Otherwise, it will execute the election end module M7.

[0041] Module M6: Becoming the master node step, the candidate node selects the latest log copy from the multiple log copies saved locally, and copies the copy to the log copy belonging to the candidate node, updates the election-related metadata, and becomes the new master node; executes the election end module M7;

[0042] Module M7: Election end step, ending the master node election stage.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Compared with the existing consensus algorithm design that does not use RDMA, the present invention uses RDMA for network communication, so that the present invention has high throughput and low network communication latency.

[0045] 2. Compared with the existing consensus algorithm design using RDMA, the nodes in the present invention can bypass the remote processors of other nodes in both stages of the consensus algorithm and support a fine-grained fault-tolerant model.

[0046] 3. Compared with the existing consensus algorithm design that uses RDMA and requires the participation of slave node processors for log backup, the log backup stage of the present invention does not require the participation of slave node processors, thereby eliminating the delay caused by the response of slave node processors and reducing the network communication delay of log backup.

[0047] 4. Compared with the existing consensus algorithm design that uses RDMA and requires multiple RDMA round trips to complete log backup, the present invention only requires one RDMA round trip to complete the log backup operation, further reducing the network communication delay of the log backup. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0049] Figure 1 This is a flowchart for implementing the master node election phase of the present invention;

[0050] Figure 2 The present invention is a flow chart for implementing the log backup phase. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0052] Reference Figure 1 and Figure 2 The present invention provides an efficient consensus algorithm design based on RDMA that supports fine-grained fault tolerance. When the master node is not faulty, the master node executes the log backup phase and is responsible for backing up the log to the slave node log. If the slave node detects that the master node has failed, the slave node executes the master node election phase and elects a new master node from the candidate nodes. The master node, slave node, and candidate node are the three roles played by the node, and the master node is unique. The slave node will perform the following steps in sequence during the master node election phase:

[0053] Step 1: Become a candidate node: The slave node checks the terms of all nodes stored locally, sets the term to the latest term plus one, becomes a candidate node, and executes step 2 of requesting a vote.

[0054] Step 2: Request vote step: For each other node, the candidate node writes the term to the election-related metadata of the node via RDMA. Execute step 3 to collect votes.

[0055] Step 3: Collect voting steps: For each other node, the candidate node reads all election-related metadata of the node through RDMA. Execute step 4 to check the voting results.

[0056] Step 4: Check voting results: For each RDMA read request in step 3 that collects votes, after the read request is returned from the node, the candidate node analyzes the election-related metadata obtained. If the candidate node meets the conditions for being voted by the node, the number of votes obtained by the candidate node increases by one. Execute step 5 to count the voting results.

[0057] Step 5: Counting voting results: The candidate node counts all votes. If the number of votes it receives exceeds half of the total number of nodes, it executes step 6 to become the master node. Otherwise, it executes step 7 to end the election.

[0058] Step 6: Becoming the master node: The candidate node selects the latest log copy from the multiple local log copies, and copies the copy to the log copy belonging to the candidate node, updates the election-related metadata, and becomes the new master node. Execute the election and end step 7.

[0059] Step 7: End of election step: End of the master node election phase.

[0060] A term is a period of time during which there can be at most one master node. A new term begins when a slave node initiates a master election. A term is represented by a monotonically increasing integer; logs and election-related metadata are copied on each node equal to the total number of nodes, with each copy belonging to one node. A node can only update one copy belonging to the node, but can read all copies.

[0061] The election-related metadata includes the node's term and log metadata. The log metadata includes the term and index of the latest log entry in the log; in the request vote step 2, the candidate node only writes the term to the election-related metadata copy belonging to the candidate node through RDMA, and in the collect vote step 3, the candidate node reads all copies of the election-related metadata.

[0062] The two RDMA requests in step 3 of collecting votes and step 2 of requesting votes can be completed with only one RDMA round trip; the candidate node meets the conditions for being voted by the node if: the term of the candidate node is greater than the maximum value of all terms in the node, and the latest log of the candidate node is not older than the latest log of the node. The latest log of a node refers to the latest log among all node logs stored by the node.

[0063] The condition for judging the oldness of logs is that log A is not older than log B, which is equivalent to the latest log item of log A being not older than the latest log item of log B. The latest log item is the last log item of the log. The condition for judging the oldness of log items is that if the term of log item A is greater than the term of log item B, or the terms of log item A and log item B are the same and the index of log item A is not less than the index of log item B, then log item A is not older than log item B.

[0064] When the master node is not faulty, the master node executes the log backup phase and is responsible for backing up the logs to the slave node logs. If the slave node detects that the master node has failed, the slave node executes the master node election phase and elects a new master node from the candidate nodes. The master node, slave node, and candidate node are the three roles played by the node, and the master node is unique. The slave node will perform the following steps in sequence during the log backup phase:

[0065] Step S1: Backup step: For each slave node, the master node writes the new log item to the slave node log via RDMA, and writes the term and index of the latest log item to the election-related metadata stored by the slave node via RDMA. Execute the term collection step S2.

[0066] Step S2: Collecting terms: For each slave node, the master node reads all node terms of the slave node through RDMA. Execute step S3 to check the backup result.

[0067] Step S3: Checking the backup result: For each RDMA read request in the term collection step S2, after the read request is returned, the master node analyzes the term returned by the read request. If the master node term meets the backup success condition, the backup success number is increased by one. Execute the backup result statistics step S4.

[0068] Step S4: Counting the backup results: The master node counts the backup results. If the number of successful backups is greater than half of the total number of nodes, the log item is backed up successfully and the log submission step S5 is executed. Otherwise, the log item backup fails and the backup end step S6 is executed.

[0069] Step S5: Log submission step: The master node increases the submitted index of the log by 1. The backup is performed and the end step S6 is performed.

[0070] Step S6: Backup end step: end the log backup phase.

[0071] During the log backup phase, the master node only updates a log copy and election-related metadata copy belonging to the master node on each node; the log is implemented as a circular memory buffer that occupies a fixed amount of memory and stores many log items. New log items are written at the end of the circular memory buffer.

[0072] In the backup step S1 and the term collection step S2, RDMA writes the new log item that needs to be backed up, RDMA writes the term and index of the latest log item in the slave node log, and RDMA reads all node terms saved by the slave node. These three RDMA operations can be sent continuously without waiting, so only one RDMA round trip is required to complete.

[0073] The committed index of a log refers to the index of the latest committed log item in the log. Log items with an index less than the committed index can be considered committed log items. The backup success condition is: the master node checks the term of all nodes in the slave nodes. If the master node term is not less than the maximum value of all terms in the slave nodes, the backup success condition is met, otherwise it does not meet the backup success condition.

[0074] Compared with the existing consensus algorithm design that does not use RDMA, the present invention uses RDMA for network communication, which has high throughput and low network communication latency. Compared with the existing consensus algorithm design that uses RDMA, the present invention can bypass the remote processor in both stages of the consensus algorithm by providing multiple copies of logs and election-related metadata, as well as a write-first-then-read log backup verification method, and correctly supports a fine-grained fault-tolerant model. Compared with the existing consensus algorithm design that uses RDMA and requires the participation of the slave node processor in log backup, the present invention bypasses the remote processor and directly writes to the remote memory to perform log backup, eliminating the delay caused by the response of the slave node processor and reducing the delay of log backup. Compared with the existing consensus algorithm design that requires multiple RDMA round trips to complete log backup, the present invention ensures that only one RDMA round trip is required to complete the log backup by sending two RDMA write requests and one RDMA read request for log items and log indexes without waiting, further reducing the delay of log backup.

[0075] The present invention also provides a consensus system based on RDMA that supports fine-grained fault tolerance. When the master node is not faulty, the master node executes the log backup phase and is responsible for backing up the log to the slave node log; when the slave node detects that the master node has failed, the slave node executes the master node election phase and elects a new master node from the candidate nodes; the master node, slave node, and candidate node are three roles played by the node, and the master node is unique; the slave node includes the following modules in the master node election phase: Module M1: Becoming a candidate node step, the slave node checks the terms of all nodes stored locally, sets its own term to the latest term plus one, becomes a candidate node, and executes the request voting module M2; Module M2: Request voting step, for each other node, the candidate node writes its own term into the election-related metadata of the node through RDMA; executes the collection voting module M3; Module M3: Collection voting step, for each other node, the candidate node reads the All election-related metadata stored in the node; execute the voting result checking module M4; module M4: the voting result checking step, for each RDMA read request in the voting collection module M3, after the read request is returned from the node, the candidate node analyzes the election-related metadata; if the candidate node meets the conditions for being voted by the node, the number of votes obtained by the candidate node is increased by one; execute the voting result counting module M5; module M5: the voting result counting step, the candidate node counts all votes, and the number of votes obtained by itself exceeds half of the total number of all nodes, then execute the becoming the master node module M6, otherwise execute the election ending module M7; module M6: the becoming the master node step, the candidate node selects the latest log copy from the multiple log copies stored locally, and copies the copy to the log copy belonging to the candidate node, updates the election-related metadata, and becomes the new master node; execute the election ending module M7; module M7: the election ending step, ends the master node election stage.

[0076] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0077] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A consensus method based on RDMA supporting fine-grained fault tolerance, characterized in that: When the master node is not faulty, the master node executes the log backup phase, responsible for backing up the log to the slave node log; if the slave node detects that the master node is faulty, the slave node executes the master node election phase, electing a new master node from the candidate nodes; the master node, slave node, and candidate node are the three roles played by the node, and the master node is the only one; the slave node will perform the following steps in sequence during the master node election phase: Step 1: Become a candidate node. The slave node checks the terms of all nodes saved locally, sets its own term to the latest term plus one, becomes a candidate node, and executes step 2 of requesting a vote. Step 2: Request voting step. For each other node, the candidate node writes its term into the election-related metadata of the node through RDMA. Execute step 3 of collecting votes; Step 3: Collect votes. For each other node, the candidate node reads all election-related metadata stored on the node through RDMA. Execute step 4 to check voting results; Step 4: Check voting results. For each RDMA read request in step 3 that collects votes, after the read request is returned from the node, the candidate node analyzes the election-related metadata. If the candidate node meets the conditions for being voted by the node, the number of votes obtained by the candidate node increases by one; and step 5 of counting the voting results is executed; Step 5: Counting the voting results. The candidate node counts all the votes. If the number of votes it receives exceeds half of the total number of nodes, it will execute Step 6 to become the master node. Otherwise, it will execute Step 7 to end the election. Step 6: Becoming the master node: the candidate node selects the latest log copy from the multiple log copies saved locally, and copies the latest log copy to the log copy belonging to the candidate node, updates the election-related metadata, and becomes the new master node; executes the election end step 7; Step 7: The election ends, ending the master node election phase.

2. According to claim 1, a consensus method based on RDMA supporting fine-grained fault tolerance is characterized in that: The term refers to a period of time, during which there is at most one master node; a new term begins when a slave node initiates the election of a master node; the term is represented by a monotonically increasing integer.

3. According to claim 1, a consensus method based on RDMA supporting fine-grained fault tolerance is characterized in that: The log and election-related metadata have copies on each node equal to the total number of nodes, where each copy belongs to one node; a node can only update one copy belonging to the node.

4. According to claim 1, a consensus method based on RDMA supporting fine-grained fault tolerance is characterized in that: The election-related metadata includes the term of office of the node and log metadata; the log metadata includes the term and index of the latest log item in the log.

5. According to claim 1, a consensus method based on RDMA supporting fine-grained fault tolerance is characterized in that: In step 2, the candidate node only writes the term to a copy of the election-related metadata belonging to the candidate node, and in step 3 of collecting votes, the candidate node reads all copies of the election-related metadata.

6. According to claim 1, a consensus method based on RDMA supporting fine-grained fault tolerance is characterized in that: The RDMA requests in step 3 and step 2 are sent continuously without waiting, requiring one RDMA round trip.

7. According to claim 1, a consensus method based on RDMA supporting fine-grained fault tolerance is characterized in that: The candidate node meets the conditions for being voted by the node if the candidate node's term is greater than the maximum value of all terms in the node, and the candidate node's latest log is not older than the node's latest log; the node's latest log refers to the latest log among all node logs stored by the node.

8. According to claim 1, a consensus method based on RDMA supporting fine-grained fault tolerance is characterized in that: The node will perform the following steps in sequence during the log backup phase: Step S1: backup step. For each slave node, the master node writes the new log items to be backed up into the slave node log via RDMA, and writes the term and index of the latest log item into the slave node election-related metadata via RDMA. Execute the collection term step S2; Step S2: collecting term steps, for each slave node, the master node reads all node terms saved by the slave node through RDMA; executing step S3 of checking backup results; Step S3: Check the backup result step. For each RDMA read request in the term collection step S2, after the read request is returned, the master node analyzes the term obtained by reading; if the master node meets the backup success condition, the backup success number is increased by one; and the backup result statistics step S4 is executed; Step S4: backup result statistics step, the master node counts all backup results, if the number of successful backups exceeds half of the total number of all nodes, the log item is successfully backed up, and the log submission step S5 is executed; otherwise, the log item backup fails, and the backup end step S6 is executed; Step S5: Log submission step, the master node increases the submitted index of its own log by one; and performs backup and ends step S6; Step S6: Backup end step, ending the log backup phase.

9. The RDMA-based consensus method supporting fine-grained fault tolerance according to claim 8, characterized in that: The master node will only update a log copy and election-related metadata copy belonging to the master node on each node during the log backup phase; The implementation of the log is a circular memory buffer, which occupies a fixed memory; the log contains many log items, and the new log item is written at the end of the circular memory buffer; in the steps S1 and S2, RDMA writes the new log item that needs to be backed up, RDMA writes the term and index of the latest log item in the slave node log, and RDMA reads the terms of all nodes saved by the slave node, and these three RDMA operations only require one RDMA round trip to complete; the committed index of the log refers to the index of the latest committed log item recorded in the log; log items with indexes less than the committed index are all committed log items; the backup success condition refers to the master node checking the terms of all nodes in the slave node; If the term of the master node is greater than or equal to the maximum value of all terms of the slave nodes, the backup success condition is met; otherwise, the backup success condition is not met.

10. A consensus system based on RDMA supporting fine-grained fault tolerance, characterized in that: When the master node is not faulty, the master node executes the log backup phase, responsible for backing up the log to the slave node log; when the slave node detects the master node failure, the slave node executes the master node election phase, electing a new master node from the candidate nodes; the master node, slave node, and candidate node are the three roles played by the node, and the master node is the only one; the slave node includes the following modules in the master node election phase: Module M1: Steps to become a candidate node. The slave node checks the terms of all nodes saved locally, sets its own term to the latest term plus one, becomes a candidate node, and executes the voting request module M2; Module M2: Request voting step. For each other node, the candidate node writes its term into the election-related metadata of the node through RDMA. Execute the voting collection module M3; Module M3: Collect voting steps. For each other node, the candidate node reads all election-related metadata stored in the node through RDMA; Execute the voting result checking module M4; Module M4: Check voting results step. For each RDMA read request collected in the voting module M3, after the read request is returned from the node, the candidate node analyzes the election-related metadata; If the candidate node meets the conditions for being voted by the node, the number of votes obtained by the candidate node is increased by one; the voting result counting module M5 is executed; Module M5: voting result counting step. The candidate node counts all votes. If the number of votes it receives exceeds half of the total number of nodes, it will execute the master node module M6. Otherwise, it will execute the election end module M7. Module M6: Becoming the master node step, the candidate node selects the latest log copy from the multiple log copies saved locally, and copies the copy to the log copy belonging to the candidate node, updates the election-related metadata, and becomes the new master node; executes the election end module M7; Module M7: Election end step, ending the master node election stage.

Images