A high-performance randomized consensus method for uncertain environments

Abstract

The application relates to the technical field of computers and discloses a high-performance randomization consistency method for uncertain environments, which comprises the following steps: step one, implanting a unified random seed for each process, which is used for ensuring that the random numbers generated by the processes in the same round are the same; step two, adding a process ID to each process for a proposal, and generating a local proposal value P; step three, generating a full permutation array id_seq of all process IDs in a unified sequence; step four, sending the local proposal value P to other processes through reliable broadcasting; step five, adding the proposal process ID to the set values together with the proposal value P; step six, entering stage 1 of the first round; and step seven, entering stage 2 of the first round. The high-performance randomization consistency method for uncertain environments can still be active in an asynchronous environment, so that the system can still work normally when facing sudden asynchronous conditions such as clock frequency drift, network instability, task blocking and the like, and the availability of the system is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a high-performance randomized consistency method for uncertain environments. Background Art

[0002] Maintaining data consistency is particularly important when processing massive amounts of concurrent reads and writes in a distributed execution environment. For example, multiple nodes need to store the same data or provide the same read results. To achieve data consistency across nodes, a consensus method must be implemented within each node: nodes negotiate their respective proposals over several rounds, ultimately reaching a consensus. Currently, mainstream Paxos-like consensus methods rely on synchronous or partially synchronous systems. This means that differences in clock frequencies between nodes, as well as communication and processing delays, have known or unknown upper limits. This places high demands on hardware and the execution environment. Variations in hardware resources such as CPUs, memory, and timers, caused by manufacturing processes, make achieving high-performance consistency in uncertain environments extremely difficult. Currently, the common approach, both domestically and internationally, is to use high-cost physical components to ensure consistency. This traditional approach is not only expensive but also highly sensitive to environmental conditions across the entire system, failing to operate properly under abnormal temperature, pressure, and humidity conditions. Consequently, it requires a large amount of additional auxiliary equipment, such as fans, air conditioners, and liquid / water cooling systems, resulting in significant energy consumption.

[0003] According to FLP theory, in asynchronous environments, where there are no upper bounds on clock frequency differences or communication and processing delays between nodes, Paxos-like consensus methods lack liveness and may engage in endless negotiations without ultimately reaching consensus. To address this "livelock" problem, Paxos-like consensus methods introduce a leader node, which is responsible for writing data and synchronizing it with other nodes. This approach places excessive load on the leader node, resulting in load imbalance. We can leverage randomness to address this issue. Common randomized consensus methods include "binary coin" and "multi-value coin." When no proposal is accepted by a majority of nodes, a random selection from the received proposals is used as the proposal for the next round. This process is called "coin tossing." If the "coin toss" values ​​of all nodes are exactly the same, it is considered a perfect match, and consensus can be reached quickly. Perfect matching in "binary coin" can be achieved by using a pseudorandom algorithm to generate random numbers in each round. As long as the random seed is the same, the random numbers generated by different nodes in the same round are guaranteed to be the same. However, the value range of "binary coin" proposals can only be {0, 1}, limiting its applicability. "Multi-value Co-currency" uses the set of received proposals as its value domain and supports any proposed value. To achieve perfect matching, "Multi-value Co-currency" requires not only the same random seed and round number, but also the same set of received proposals, and a strict total order between any two proposals (allowing comparison of their sizes). These stringent conditions lead to low performance. Existing technologies lack a high-performance randomized consensus method for uncertain environments. Summary of the Invention

[0004] Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a high-performance randomized consistency method for uncertain environments, which has the advantages of being able to perform high-performance consistency calculations under abnormal conditions such as clock frequency drift, network instability, and task blocking, solving the problem that the existing technology lacks a high-performance randomized consistency method for uncertain environments.

[0006] (2) Technical solution

[0007] To achieve the above-mentioned goal of being able to perform high-performance consistency calculations under abnormal conditions such as clock frequency drift, network instability, and task blocking, the present invention provides the following technical solution: a high-performance randomized consistency method for uncertain environments, comprising the following steps:

[0008] Step 1: A unified random seed is implanted for each process to ensure that the random numbers generated by each process in the same round are the same;

[0009] Step 2: Each process adds its process ID to the proposal and generates a local proposal value P.

[0010] Step 3: Generate a fully arrayed array id_seq of all process IDs in a uniform order;

[0011] Step 4: Send the local proposal value P to other processes through reliable broadcast;

[0012] Step 5: Add the proposing process ID and the proposed value P to the set values;

[0013] Step 6: Enter Phase 1 of the first round;

[0014] Step 7: Enter Phase 2 of the first round;

[0015] Step 8. In phase 2, when the number of times a value v appears in the array Val used to store the local proposal value P in phase 1 is strictly greater than f, a consensus is reached and the step ends. Otherwise, return to phase 1 until the number of times the value v appears is strictly greater than f, where f is the number of faulty processes that can be tolerated.

[0016] Preferably, the first round of stage 1 is carried out according to the following steps:

[0017] A1. Send the local proposal value P to all processes through best-effort broadcast;

[0018] A2. Wait for all processes to send their local proposal values ​​P and save them in the array Val.

[0019] A3. Determine whether the number of proposals received in array Val is strictly more than half. If yes, proceed to step A4. If not, return to the previous step.

[0020] A3. Determine whether the number of times a proposed value v appears in the array Val is strictly more than half. If so, change the local proposed value P to v. Otherwise, change it to ⊥ and enter stage 2.

[0021] Preferably, the first round of stage 2 is carried out according to the following steps:

[0022] B1. Clear the array Val to receive the proposed value in phase 2.

[0023] B2. Each process broadcasts its local proposal value P to all processes again through best-effort broadcasting;

[0024] B3. Wait for all processes to send their proposed values ​​and save them in the array Val.

[0025] B4. Determine whether the number of proposed values ​​received by array Val is not less than Nf. If yes, proceed to step B5; otherwise, return to the previous step.

[0026] B5. Call the coin tossing step to generate a new random number and take the modulus of the size of the array id_seq, denoted as a.

[0027] B6. Determine whether the number of proposed values ​​v in the array Val is greater than f. If yes, proceed to step B7. If not, proceed to step B7.

[0028] B7. When the number of proposed values ​​v in the array Val is greater than f, consensus is reached. The process assigns the local decision value to v and calls reliable broadcast to broadcast a message to all processes to promote the decision value.

[0029] B8. After receiving the local decision value v, if the local decision has not yet been made, decide v and submit;

[0030] B9. Generate a new random seed based on the consensus result to ensure that the random numbers of each process in the next consensus are consistent and end the step;

[0031] B10. When the number of proposed values ​​v in the array Val is less than f, determine whether all elements in the array Val are ⊥. If so, select the process ID sorted by id_seq[a], select the maximum value b from values ​​in the order of id_seq[a], and change the local proposed value P to the currency value b, and start the next round. Otherwise, select any non-⊥ proposed value w from Val, then assign the local proposed value P to w, and start the next round.

[0032] Preferably, the coin tossing step in B5 is divided into the following two steps:

[0033] C1. Generate a new random number using a pseudo-random algorithm and take the remainder of the full permutation array id_seq, denoted as a, where a represents the permutation order of the process ID. The uniform random number seed ensures that each process in the same round selects the same permutation. This permutation is denoted as id_seq[a].

[0034] C2. According to the order of the proposed processes in id_seq[a], the largest proposed value is selected from the set values ​​as the coin value b, and the process executes the coin tossing step C1 to obtain a.

[0035] Preferably, the A3 step indicates that the process has not found a proposal recognized by the majority of processes. At this time, the process is represented by pi and the proposal value is represented by vi. When process pi sees that the proposal value vi appears strictly more than half of the times, and another process pj sees that the proposal value vj appears strictly more than half of the times, then vi is equal to vj, let vi=vj=v*, and then enter the first round of stage 2.

[0036] Preferably, in step three, initially, the local proposal value P, local decision value, proposal array Val, and proposal set values ​​of each process are all set to ⊥, where ⊥ represents empty.

[0037] Preferably, in step five, when each process receives a proposal request from an upper-layer application, it uses it as a local proposal value P, and adds its own process ID together with P to the set values, and then sends P to other processes through reliable broadcast. Once a process receives a proposal value from another process through reliable broadcast, it adds the proposed process ID together with the proposed value to the set values. Reliable broadcast is used to ensure that the set values ​​of each process are eventually the same.

[0038] Preferably, in step C2, when a process obtains a, if the number of times a value v in the array Val appears is strictly greater than f, it means that consensus has been reached and the decision value is v. At this time, v is v* in stage 1, and there is no need to calculate the currency value.

[0039] Preferably, in step B8, any process receiving the message will check whether the local decision value is empty. If it is empty, it will be assigned a value of v and delivered to ensure that the decision value is delivered only once.

[0040] Preferably, in step B9, a HASH value is calculated based on the decision value of this consensus as a random seed for the next consensus, and the consistency of the decision value is used to ensure the consistency of the random seed, wherein the HASH value is a unique identifier in the software.

[0041] Compared with the existing technology, this invention provides a high-performance randomized consistency method for uncertain environments, which has the following beneficial effects:

[0042] 1. The present invention can ensure that it is still active in an asynchronous environment, so that the system can still work normally when facing sudden asynchronous situations such as clock frequency drift, network instability, task blocking, etc., thereby improving the availability of the system.

[0043] 2. The present invention reduces the system's requirements for hardware and execution environment (the system can be constructed with inexpensive hardware and does not require additional refrigeration equipment), reduces hardware and operation and maintenance costs, saves energy, and reduces carbon emissions.

[0044] 3. The present invention can quickly reach consensus without relying on strict preset conditions. It is simple to implement and can be easily transplanted into various consensus systems. It does not require a leader and can avoid the overhead of re-election when a leader fails. It can replace the widely used Paxos-type consensus method and make the consensus system adapt to asynchronous environments. For example, it can be used to replace the leader election part of Zookeeper, the Raft method of etcd and Consul, etc.

[0045] 4. The present invention allows any node to make fair proposals, and there is no situation where a node's proposal has a higher priority, which can ensure load balancing of the entire cluster.

[0046] 5. The present invention can avoid excessive loss of certain equipment that is frequently elected as a leader node, reduce the cost of replacing equipment, and improve the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flowchart for realizing the present invention;

[0048] Figure 2 A schematic diagram illustrating the use and execution of the present invention;

[0049] Figure 3 Schematic diagram of an example of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] In an uncertain execution environment, especially an asynchronous environment, the difference in clock frequency and the communication and processing delays have no upper limit, so there is no reliable failure detector (FailureDetector), that is, the process cannot determine whether the other process has an error by timeout. The present invention can reach a consensus without the help of a failure detector, which is different from mainstream consensus methods, such as the ZAB method in zookeeper and the Raft method in etcd. The communication components in the method of the present invention are similar to the mainstream consensus methods, including best-effort broadcast and reliable broadcast. Both broadcasts can send messages from a certain process to all correct processes. The difference is that reliable broadcast guarantees that even if the sending process crashes, the message can be transmitted to all correct processes, but best-effort broadcast cannot guarantee that.

[0052] See also Figure 1 , a high-performance randomized consistency method for uncertain environments, including the following steps:

[0053] Step 1: A unified random seed is implanted for each process to ensure that the random numbers generated by each process in the same round are the same;

[0054] Step 2: Each process adds its process ID to the proposal and generates a local proposal value P.

[0055] Step 3: Generate a fully arrayed array id_seq of all process IDs in a uniform order;

[0056] Step 4: Send the local proposal value P to other processes through reliable broadcast;

[0057] Step 5: Add the proposing process ID and the proposed value P to the set values;

[0058] Step 6: Enter Phase 1 of the first round;

[0059] Step 7: Enter Phase 2 of the first round;

[0060] Step 8. In phase 2, when the number of times a value v appears in the array Val used to store the local proposal value P in phase 1 is strictly greater than f, a consensus is reached and the step ends. Otherwise, return to phase 1 until the number of times the value v appears is strictly greater than f, where f is the number of faulty processes that can be tolerated.

[0061] The present invention can ensure activity in an asynchronous environment, so that the system can still work normally when facing sudden asynchronous situations such as clock frequency drift, network instability, task blocking, etc., thereby improving the availability of the system and reducing the system's requirements for hardware and execution environment (the system can be constructed with inexpensive hardware and does not require additional refrigeration equipment), reducing hardware and operation and maintenance costs, saving energy, and reducing carbon emissions.

[0062] The specific implementation method is as follows:

[0063] At the beginning of a consensus, we first implant a unified random seed for each process to ensure that the random numbers generated by each process in the same round are the same; secondly, each process generates a fully permuted array id_seq of all process IDs in a unified order. Finally, each process's local proposal value P, local decision value decision, proposal array Val, and proposal set values ​​are all set to ⊥, where ⊥ represents empty.

[0064] After receiving a proposal request from an upper-layer application, each process uses it as its local proposal value P and adds its own process ID and P to the set values. It then sends P to other processes via reliable broadcast. Once a process receives a proposal value from another process via reliable broadcast, it adds the proposed process ID and the proposed value to the set values. Reliable broadcast ensures that the set values ​​of each process are ultimately the same. This then enters Phase 1 of the first round.

[0065] Each round is divided into two stages. In stage 1, each process sends a local proposal value P to all processes through best-effort broadcasting, and then waits for the proposal values ​​sent by all processes and saves them in the array Val. When receiving the proposal value sent by strictly more than half of the processes, if the number of times a proposal value v in the array Val appears is strictly more than half, the local proposal value P is changed to v, otherwise it is changed to ⊥, indicating that the process has not found a proposal recognized by the majority of processes. From this process, it is not difficult to see that if process pi sees that the proposal value vi appears strictly more than half, and process pj sees that the proposal value vj appears strictly more than half, then vi must be equal to vj, let vi=vj=v*, and then enter stage 2 of the first round.

[0066] In phase 2 of the first round, the array Val is first cleared to receive the proposed values ​​of phase 2. Then each process broadcasts its local proposed value P to all processes again through best-effort broadcasting. At this time, P is either equal to v* or equal to ⊥. Then, it waits for the proposed values ​​sent by all processes and saves them in the array Val. When the number of proposed values ​​received is not less than Nf (f is the number of faulty processes that can be tolerated), the coin tossing step is called. The coin tossing step is divided into two steps:

[0067] Generate a new random number using a pseudo-random algorithm and take the remainder of the full permutation array id_seq, denoted as a, where a represents the permutation order of the process ID. The uniform random number seed ensures that each process in the same round selects the same permutation, denoted as id_seq[a].

[0068] According to the order of the proposed processes in id_seq[a], the largest proposal is selected from the set values ​​as the coin value b. Even if the set values ​​of each process are not exactly the same, as long as they all contain the proposal value with the largest order, they can be perfectly matched. In this case, the process only executes step 1 of the coin toss to obtain a.

[0069] When a process obtains a, if the number of times a value v appears in the array Val is strictly greater than f, it means that consensus has been reached and the decision value is v. At this time, v is v* in stage 1. There is no need to calculate the currency value. The process assigns the local decision value decision to v, and calls reliable broadcast to broadcast the [DECIDED, decision] message to all processes to promote all processes to deliver the decision value. Any process that receives the [DECIDED, v] message will check whether the local decision value decision is empty. If it is empty, it will be assigned to v and delivered. This process ensures that the decision value is delivered only once. Finally, since the number of rounds experienced by each process in each consensus may be different, the random numbers need to be aligned before the next consensus. The present invention calculates the HASH value based on the decision value of this consensus as the random seed for the next consensus. The consistency of the decision value ensures the consistency of the random seed, where the HASH value is the unique identifier in the software.

[0070] On the contrary, if the number of occurrences of a certain value v in the array Val is less than or equal to f, then any non-⊥ value w is picked out, and the local proposal value P is assigned to w. At this time, w is v* in stage 1, and there is no need to calculate the currency value. If the values ​​in the array Val are all ⊥, then step 2 is executed. The currency value b is calculated based on the set values ​​and id_seq[a] in step 1, and the local proposal value is changed to currency value b. Then, the next round of stage 1 is entered. The logic of each round is completely consistent. This is executed round by round. Since the process IDs of each process in the same round are arranged in the same order, and the set values ​​of each process are ultimately the same, the currency values ​​will eventually match perfectly. Therefore, in the final stage 2 of a certain round, a consensus will be reached that the number of occurrences of a certain value v in the array Val is strictly greater than f. Since the present invention selects the largest proposal from the received proposals as the currency value according to the order of the proposed processes in the arrangement, even if the set values ​​of each process are not exactly the same, as long as they all contain the largest proposed value in the arrangement order, a perfect match can be achieved. Compared with traditional consensus methods, it can greatly improve the matching probability, thereby reducing the number of rounds to reach consensus and improving consensus efficiency.

[0071] like Figure 2As shown in the figure, the usage and execution of the present invention are briefly described. First, we implant a unified random seed into each node 1, 2, and 3, and generate a fully permuted array of all process IDs in a unified order (e.g., [123, 132, 213, 231, 312, 321]). Then, we send proposal requests v1, v2, and v3 to each node through the upper-layer application. Each node first combines its own process ID and the proposed value to obtain (1, v1), (2, v2), and (3, v3), respectively. These are then reliably broadcast, and the received sequence pairs are added to the set values. Then, in phase 1, each node tries its best to broadcast its local proposed value. Each node modifies its local proposed value based on the received proposed value. Then, in phase 2, each node tries its best to broadcast its local proposed value again. If consensus is not reached, the received proposed value or currency value is used to modify the local proposed value, and the next round of phase 1 is entered. After n rounds, consensus is reached and the decision value is delivered to the upper-layer application. As can be seen, each node can make proposals and has equal status, achieving load balancing.

[0072] The present invention has described the principle and usage of a high-performance randomized consistency method for uncertain environments. In order to further illustrate the practicability of the proposed method, the present invention is described from the perspective of engineering practice with a specific example. Figure 3 As shown, taking the consensus system in a distributed key-value storage cluster as an example, the entire system consists of an odd number of nodes, taking 3 nodes as an example, each node maintains the following information:

[0073] round, indicating the current calculation round;

[0074] Phase indicates the calculation phase in each round;

[0075] P, represents the local proposal value;

[0076] Val represents the array of received proposal values. When a specified number of proposal values ​​are received, the corresponding algorithm steps are executed.

[0077] values, which represents the set of all proposed values ​​from which the coin is tossed;

[0078] decision, which indicates the decision value obtained by the final consensus;

[0079] id_seq, which represents a fully permuted array of all process IDs generated in a uniform order.

[0080] Nodes communicate using best-effort broadcast (BEB) and reliable broadcast (RB) to form a distributed consensus system. Multiple upper-layer clients can connect to any node in the consensus system sequentially or simultaneously and send corresponding proposals, such as assigning x to 1 or 2. Nodes within the consensus system use our proposed method to reach consensus, ensuring the consistency of the x value in the three nodes. Upper-layer clients can connect to any node to read the value of x and ensure that it is the same.

[0081] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A high-performance randomized consistency method for uncertain environments, characterized by: The following steps are involved: Step 1: A unified random seed is implanted for each process to ensure that the random numbers generated by each process in the same round are the same; Step 2: Each process adds its process ID to the proposal and generates a local proposal value P. Step 3: Generate a fully arrayed array id_seq of all process IDs in a uniform order; Step 4: Send the local proposal value P to other processes through reliable broadcast; Step 5: Add the proposing process ID and the proposed value P to the set values; Step 6: Enter Phase 1 of the first round, including: A1. Send the local proposal value P to all processes through best-effort broadcast; A2. Wait for all processes to send their local proposal values ​​P and save them in the array Val. A3. Determine whether the number of proposals received in array Val is strictly more than half. If yes, proceed to step A4. If not, return to the previous step. A4. Determine whether the number of times a proposed value v appears in the array Val is strictly more than half. If yes, change the local proposed value P to v. Otherwise, change it to ⊥ and enter stage 2; where ⊥ represents null. Step 7: Enter Phase 2 of the first round, including: B1. Clear the array Val to receive the proposed value in phase 2. B2. Each process broadcasts its local proposal value P to all processes again through best-effort broadcasting; B3. Wait for all processes to send their proposed values ​​and save them in the array Val. B4. Determine whether the number of proposed values ​​received by array Val is not less than Nf. If yes, proceed to step B5. If not, return to the previous step. B5. Call the coin tossing step to generate a new random number and take the modulus of the size of the array id_seq, denoted as a. B6. Determine whether the number of proposed values ​​v in the array Val is greater than f. If yes, proceed to step B7. If not, proceed to step B7. B7. When the number of proposed values ​​v in the array Val is greater than f, consensus is reached. The process assigns the local decision value to v and calls reliable broadcast to broadcast a message to all processes to promote the decision value. B8. After receiving the local decision value v, if the local decision has not yet been made, decide v and submit; B9. Generate a new random seed based on the consensus result to ensure that the random numbers of each process in the next consensus are consistent and end the step; B10. When the number of proposed values ​​v in the array Val is less than f, determine whether all elements in the array Val are ⊥. If so, select the process IDs in the order id_seq[a], select the maximum value b from values ​​in the order of id_seq[a], and change the local proposed value P to the currency value b, and start the next round. If not, select any non-⊥ proposed value w from Val, then assign the local proposed value P to w, and start the next round. Step 8. In phase 2, when the number of times a value v appears in the array Val used to store the local proposal value P in phase 1 is strictly greater than f, a consensus is reached and the step ends. Otherwise, return to phase 1 until the number of times the value v appears is strictly greater than f, where f is the number of faulty processes that can be tolerated.

2. A high-performance randomized consistency method for uncertain environments according to claim 1, characterized in that: The coin tossing step in B5 is divided into the following two steps: C1. Generate a new random number using a pseudo-random algorithm and take the remainder of the full permutation array id_seq, denoted as a, where a represents the permutation order of the process ID. The uniform random number seed ensures that each process in the same round selects the same permutation. This permutation is denoted as id_seq[a]. C2. According to the order of the proposed processes in id_seq[a], the largest proposed value is selected from the set values ​​as the coin value b, and the process executes the coin tossing step C1 to obtain a.

3. A high-performance randomized consistency method for uncertain environments according to claim 2, characterized in that: The A3 step indicates that the process has not found a proposal recognized by the majority of processes. At this time, the process is represented by pi and the proposal value is represented by vi. When process pi sees that the proposal value vi appears strictly more than half of the times, and another process pj sees that the proposal value vj appears strictly more than half of the times, then vi and vj are equal, let vi=vj=v*, and then enter the first round of stage 2.

4. A high-performance randomized consistency method for uncertain environments according to claim 1, characterized in that: In the step 3, initially, the local proposal value P, local decision value, proposal array Val, and proposal set values ​​of each process are all set to ⊥, where ⊥ represents empty.

5. The high-performance randomized consistency method for uncertain environments according to claim 1, characterized in that: In step 5, when each process receives a proposal request from the upper-layer application, it uses it as the local proposal value P, and adds its own process ID together with P to the set values, and then sends P to other processes through reliable broadcast. Once a process receives a proposal value from another process through reliable broadcast, it adds the proposed process ID together with the proposed value to the set values. Reliable broadcast is used to ensure that the set values ​​of each process are eventually the same.

6. A high-performance randomized consistency method for uncertain environments according to claim 3, characterized in that: In step C2, when a process obtains a, if the number of times a value v in the array Val appears is strictly greater than f, it means that consensus has been reached and the decision value is v. At this time, v is v* in stage 1, and there is no need to calculate the currency value.

7. The high-performance randomized consistency method for uncertain environments according to claim 1, characterized in that: In step B8, any process receiving the message will check whether the local decision value is empty. If it is empty, it will be assigned a value of v and delivered to ensure that the decision value is delivered only once.

8. A high-performance randomized consistency method for uncertain environments according to claim 7, characterized in that: In step B9, a HASH value is calculated based on the decision value of this consensus as the random seed for the next consensus. The consistency of the decision value is used to ensure the consistency of the random seed, wherein the HASH value is a unique identifier in the software.

Images