High-scalability consensus method based on Paxos protocol in cloud edge-end environment

Abstract

The invention provides a high-scalability consensus method based on a Paxos protocol in a cloud side end environment, and belongs to the technical field of distributed computing, edge computing and cloud computing. The method comprises the following steps of: hierarchically dividing nodes of the distributed consensus system into an upper layer and a lower layer; calculating the weight of an upper layer node; an arbitration set calculation method of an EPaxos protocol is modified, so that the distributed consensus system keeps high availability and high performance; adopting an optimization strategy based on delayed batch submission to optimize a submission strategy of the EPaxos protocol; the proposal replication and leader switching mechanism based on the EPaxos protocol ensures high availability and stability of the distributed consensus system; the weight distribution of the upper layer nodes is dynamically adjusted according to the real-time state and historical performance of the nodes, so that the distributed consensus system can adapt to the change of a network environment and node faults; the throughput and the consensus success rate of the distributed consensus system are improved by optimizing a proposal submission strategy.

Description

Technical Field

[0001] The present invention relates to the technical fields of distributed computing, edge computing and cloud computing, and in particular to a highly scalable consensus method based on the Paxos protocol in a cloud-edge environment. Background Art

[0002] In the traditional cross-region distributed cloud storage scenario, the traditional state machine replication protocol requires multiple rounds of wide area network communication to establish the global transaction execution order, which leads to increased latency in cluster request responses. To solve this problem, the Weave (Eischer M, Straβner B, Distler T. Low-latency geo-replicated state machines with guaranteed writes [C] / / Proceedings of the 7th Workshop on Principles and Practice of Consistency for Distributed Data. 2020: 1-9.) protocol adopts a multi-leader model, which builds replica groups in different geographic regions and designates a local leader in each group to be responsible for assigning sequence numbers to requests in this region, effectively reducing the need for cross-region communication and significantly reducing the latency of request processing. The Weave protocol also supports multiple transaction consistency levels, enabling clients to make a trade-off between response time and data consistency. Nomad (HaoZ, YiS, LiQ. Nomad: An efficient consensus approach for latency-sensitive edge-cloud applications [C] / / IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019: 2539-2547.) is optimized for latency-sensitive applications in edge computing. It uses a lease-based master-slave protocol and a dynamic leadership allocation strategy to achieve fast transaction sorting. The protocol dynamically adjusts the leadership of edge nodes based on the historical logs of system operation using an adaptive Paxos protocol. Giving cloud nodes the role of arbitrators enables cloud data centers to flexibly switch between master-slave modes based on real-time monitoring of edge network workloads to further improve the overall performance and reliability of the system.

[0003] In terms of fault tolerance optimization in hybrid cloud environments, the SeeMoRe (Amiri MJ, Maiyya S, Agrawal D, et al. Seemore: A fault-tolerant protocol for hybrid cloud environments [C] / / 2020 IEEE 36th International Conference on Data Engineering (ICDE). IEEE, 2020: 1345-1356.) protocol takes into account the characteristics that private clouds may experience node crashes and downtime and that public clouds may contain malicious nodes. It dynamically adjusts the operating mode (Trusted Master Node Centralized Coordination (TPCC) mode / Trusted Master Node Decentralized Coordination (TPDC) mode / Untrusted Master Node Decentralized Coordination (UPDC) mode) according to the workload status of the private cloud and the communication delay between clouds, thereby effectively dealing with the risk of malicious failures in the cluster while maintaining system stability.

[0004] Existing consensus protocols usually assume that all nodes are equal and the network is fully connected. However, when the replication state machine encounters network partitions or partial node connectivity, the service may be interrupted, hindering the consensus process. The Omni-Paxos (Ng H, Haridi S, Carbone P. Omni-Paxos: Breaking the barriers of partial connectivity [C] / / Proceedings of the Eighteenth European Conference on Computer Systems. 2023: 314-330.) protocol improves the robustness of the consensus protocol in the face of network partitions and unstable connections by decoupling and optimizing the log replication, leader election, and system reconfiguration modules respectively. Its core innovation lies in the use of the BLE (Ballot Leader Election) mechanism to ensure that a leader with good connections to the majority of nodes is elected during network changes, while maintaining data consistency and improving the system's adaptability to dynamic network environments.

[0005] The Paxos protocol performs well in distributed systems, especially in ensuring system availability. However, it still faces several key challenges. First, to ensure the correctness of the protocol, Paxos needs to build a redundant arbitration set. As the scale of the system expands, this redundancy will lead to resource waste. Second, the protocol relies on the leader node to coordinate consensus, but the leader node may become a single point of failure or performance bottleneck, directly affecting the stability and efficiency of the system. Finally, in a complex heterogeneous network environment, the scalability of Paxos is limited, and network delays and uneven bandwidth may reduce the efficiency of message delivery, further affecting the performance of the protocol. These problems limit the widespread application of the Paxos protocol in practical applications. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a highly scalable consensus method based on the Paxos protocol in a cloud-edge environment. This method targets the diversity of network environments, heterogeneity of node performance, and concurrency of multi-client requests. By introducing a multi-Paxos cluster hierarchical strategy, weights are assigned to EPaxos protocol nodes, and a proposal batch submission method is adopted. This enables the distributed consensus system based on the Paxos protocol to more flexibly respond to different network environments and load requirements, effectively improves its adaptability to high-concurrency scenarios, and overcomes the problem of insufficient ability to cope with complex network environments and high-concurrency requests in existing distributed consensus systems. At the same time, the method introduces dynamic load balancing and asynchronous processing mechanisms on the server side, so that the server side can efficiently handle concurrent client requests, and further improves the performance and stability of the system by optimizing the proposal batch submission strategy and multi-cluster coordination mechanism.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a highly scalable consensus method based on the Paxos protocol in a cloud-edge environment, comprising the following steps:

[0008] Step 1: Under the cloud-edge-device architecture, divide the node hierarchy of the distributed consensus system into upper and lower layers;

[0009] The upper layer is the decision-making layer in the cloud, which adopts the EPaxos protocol and is responsible for voting and confirming the proposals of the lower-layer nodes;

[0010] The lower layer is the edge replication layer, which uses the Multi-Paxos protocol and is responsible for replicating and executing proposals put forward by upper-layer nodes.

[0011] Step 2: Define the weight distribution mechanism of the upper-level nodes, which is adapted to the heterogeneous network environment. A dynamic weight distribution algorithm is used to calculate the weight of the upper-level nodes from three aspects: signal-to-noise ratio, the number of sub-nodes under the node, and the average communication delay of multiple clients.

[0012] The signal-to-noise ratio (SNR) is used to measure the communication quality of the upper node in a wireless network environment. The specific formula is:

[0013]

[0014] The network quality evaluation index based on the signal-to-noise ratio SNR in a wireless network environment, that is, the node importance weight based on the signal-to-noise ratio SNR in a wireless network environment, is as shown in the following formula:

[0015]

[0016] Where K is the number of other nodes related to the current node, excluding the current node, and P s is the communication capability of the upper node s, P n is the communication capability of the upper node n, G k is the signal transmission capability of node k, G n is the ability of the upper node n to receive signals, PL is the path loss from node k to the upper node n, calculated by the empirical model, X is a random variable with mean 0 and variance σ 2 The normal distribution of is the additive Gaussian white noise between node communications, dB is the decibel of the random variable X, N is the total number of nodes, and covers all nodes, SNR n is the signal-to-noise ratio of the upper node n, SNR i is the signal-to-noise ratio of node i, Weight iSNR is the network quality evaluation index of node i in the wireless network environment;

[0017] The number of sub-nodes of the node is used to reflect the influence of the node as a proposer in the lower layer. The importance weight of the node in the network is calculated based on the number of sub-nodes of the node. The specific formula is:

[0018]

[0019] Among them, Weight iNS is the importance weight of node i in the network; N n is the number of child nodes under node n, N i is the number of child nodes under node i;

[0020] The average communication delay of multiple clients is used to measure the speed at which a node responds to a client request, as shown in the following formula:

[0021]

[0022] Among them, d t is the average communication delay of multiple clients, C is the number of clients, d c,nis the communication delay between the cth client and node n, d i is the communication delay of node i;

[0023] The weight of the node based on communication delay during the consensus process is shown in the following formula;

[0024]

[0025] Among them, Weight iMsg is the weight of node i in the consensus process based on communication delay;

[0026] Step 3: Modify the quorum set calculation method of the EPaxos protocol and assign a reasonable weight to each node, so that the distributed consensus system based on the Paxos protocol in the cloud-edge environment can maintain high availability and high performance in both wired and wireless network environments and load conditions;

[0027] In the face of both wireless network environment and wired network environment, the final node weight formula that comprehensively considers the signal-to-noise ratio, the number of sub-nodes under the node, and the average communication delay of multiple clients is:

[0028]

[0029] Among them, Weight i is the final comprehensive weight of the i-th node, and α is the node weight coefficient, which is used to weigh the relationship between high availability and request latency of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment;

[0030] Under the new weight distribution mechanism, the arbitration set of the EPaxos protocol no longer depends solely on the number of nodes, but is calculated based on the sum of the weights of the nodes that are comprehensively considered at the end;

[0031] The arbitration set formula of the EPaxos protocol is:

[0032]

[0033] Among them, Q is the arbitration set of the EPaxos protocol;

[0034] Step 4: Use the optimization strategy based on delayed batch submission to optimize the submission strategy of the EPaxos protocol. The specific process is as follows:

[0035] S1: Set the receiver node’s buffer and delayed reply parameters: After receiving a proposal, the receiver node does not process it immediately, but stores it in the local buffer and waits for the preset delay time;

[0036] S2: Batch processing of proposals: After the delay time is over, the recipient node groups the proposals in the buffer according to the client number and data object, and submits them in batches after sorting;

[0037] S3: Prioritize conflicting proposals: In conflict scenarios, the sorting priority strategy ensures that high-priority proposals can be submitted through the fast path of the EPaxos protocol, reducing communication overhead and latency;

[0038] Step 5: Adopt an efficient proposal replication and leader switching mechanism based on the EPaxos protocol to ensure the high availability and stability of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment;

[0039] In the lower-layer Multi-Paxos protocol, heartbeat packets are used to detect node activity to ensure the activity of the current leader. If the leader is found to be invalid, the leader switching mechanism is triggered, thereby ensuring the high availability and stability of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment.

[0040] Step 6: Dynamically adjust the weight distribution of upper-level nodes based on the real-time status and historical performance of the nodes, so that the distributed consensus system based on the Paxos algorithm in the cloud-edge environment can adapt to changes in the network environment and node failures, and maintain an efficient distributed consensus process;

[0041] Step 7: Apply the batch-submitted EPaxos protocol in conflict-free and conflict-free scenarios, and improve the throughput and consensus success rate of the distributed consensus system based on the Paxos protocol in the cloud-edge environment by optimizing the proposal submission strategy;

[0042] In a conflict-free scenario, multiple conflict-free proposals are merged into a joint operation through a delayed batch submission strategy; dynamically determine whether to merge proposals based on the current state of the node to ensure that the merged proposals come from nodes with good network quality and low latency;

[0043] In conflict scenarios, when dealing with conflicting proposals, the order of proposal processing is optimized through a sorting priority strategy to reduce the interdependence between proposals, thereby improving the consensus success rate. In this scenario, the weight of the node will affect the priority of proposal sorting. Proposals from high-weight nodes will be processed first to ensure that the most reliable nodes in conflict scenarios reach consensus first, thereby improving the consensus efficiency and stability of the system and reducing communication overhead and latency.

[0044] The beneficial effect of adopting the above technical solution is that: the present invention provides a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment. The method designs a hybrid Paxos protocol and proposes the following key innovations to solve the data consistency problem in a cloud-edge-device heterogeneous network environment:

[0045] (1) Hierarchical consensus mechanism. Based on the multi-Paxos protocol, the present invention designs a hierarchical consensus mechanism. The mechanism divides the cloud edge nodes into a decision layer and a replication layer, where the decision layer is responsible for voting and confirming proposals, and the replication layer is responsible for storing and executing proposals. Through this hierarchical design, the heterogeneity of different nodes and the complexity of the network environment are fully considered, and the consensus efficiency of the protocol is optimized.

[0046] (2) Improved EPaxos strategy with weights. To solve the problem of neglecting node heterogeneity in traditional consensus protocols, the present invention introduces an improved EPaxos strategy with weights. This strategy dynamically adjusts the weight distribution of each node in the consensus process based on factors such as the signal-to-noise ratio, the number of sub-nodes of the node, and the average communication delay of multiple clients. Through this weight mechanism, the high availability and performance of the system are improved, so that it can still maintain stable and efficient operation in the face of complex network environments.

[0047] (3) EPaxos optimization based on batch submission. Aiming at the performance bottleneck of the EPaxos protocol in high-conflict scenarios, the present invention proposes an optimization strategy based on delayed batch submission. By setting a buffer on the recipient node and introducing a delayed submission mechanism, the system can process proposals in batches, reduce conflicts between proposals, and reduce communication overhead and delay, thereby improving the overall performance of the EPaxos protocol.

[0048] Compared with the prior art, the hybrid Paxos protocol proposed in the present invention has significant performance improvement in the cloud-edge-end heterogeneous network environment; through the layered consensus mechanism and batch submission optimization strategy, the cloud-edge-end heterogeneous network delay is effectively reduced and the throughput of the distributed consensus system based on the Paxos algorithm in the cloud-edge-end environment is improved; in the simulation experiment, the complex network environment is simulated by Docker, combined with the HTB queue rules and TC traffic control tools, the network characteristics in the real scenario are successfully simulated; in the high-conflict scenario, the distributed consensus system based on the Paxos algorithm in the cloud-edge-end environment with a batch submission strategy significantly reduces the transaction submission delay, and the optimization effect is more obvious with the increase of the conflict probability; in addition, the stability and reliability of the EPaxos protocol and the Multi-Paxos protocol in the wide area network communication environment have been verified, which significantly reduces the delay fluctuation compared with the traditional Paxos protocol, while ensuring the security and activity of the system; by optimizing the scalability of the protocol, the present invention provides an efficient and reliable data consistency solution for distributed systems in a cloud-edge-end collaborative environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flow chart of a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a hybrid Paxos protocol in a cloud-edge-end heterogeneous network provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a submission strategy for optimizing the EPaxos protocol using an optimization strategy based on delayed batch submission provided in an embodiment of the present invention, wherein (a) - a fast path in the EPaxos protocol, (b) - a slow path in the EPaxos protocol;

[0052] Figure 4 A schematic diagram of an application of the batch-submission-based EPaxos protocol in a conflict-free scenario provided by an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of the application of the batch-commit-based EPaxos protocol in a conflict scenario provided by an embodiment of the present invention;

[0054] Figure 6 The embodiment of the present invention provides an impact of the batch size of the TPC-C data set on the cluster performance, wherein: (a) - the relationship between the number of node threads and the batch size, (b) - the relationship between the workload and the batch size, (c) - the relationship between the new order workload and the batch size, and (d) - the relationship between the payment order workload and the batch size. DETAILED DESCRIPTION

[0055] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0056] In this embodiment, a highly scalable consensus method based on the Paxos protocol in a cloud-edge environment is applied to abnormal vehicle detection, such as Figure 1 As shown, the following steps are included:

[0057] Step 1: Under the cloud-edge architecture, divide the node hierarchy of the distributed consensus system into upper and lower layers, such as Figure 2 As shown;

[0058] The upper layer is the decision-making layer in the cloud, which adopts the EPaxos protocol and is responsible for voting and confirming the proposals of the lower-layer nodes. Different from the traditional EPaxos protocol, the upper-layer nodes are given different weights to reflect their influence in the decision-making process.

[0059] The lower layer is the edge replication layer, which uses the Multi-Paxos protocol and is responsible for replicating and executing proposals put forward by upper-layer nodes. The lower-layer nodes are assumed to have similar performance and reliability as the upper-layer nodes to ensure the stability and consistency of the protocol.

[0060] In this embodiment, the EPaxos protocol is an improved consensus protocol. Different from the multi-Paxos cluster hierarchical strategy, the EPaxos protocol improves performance through leaderless design and conflict minimization optimization, while the Multi-Paxos protocol is an extension of the classic Paxos protocol optimized for a single master node. The two have different goals and implementation methods.

[0061] In this embodiment, the node hierarchy division method is as follows: assuming that there are 12 nodes in the distributed consensus system, of which 3 nodes are in the cloud and 9 nodes are in the edge; the cloud nodes constitute the decision-making layer, responsible for voting and confirming the proposals of the lower nodes; the edge nodes are responsible for copying and executing the proposals put forward by the upper nodes; each node will be configured according to its role in the distributed consensus system, with the cloud nodes as decision makers and the edge nodes as executors;

[0062] In this embodiment, the method of assigning different weights is: assigning preliminary weights to each cloud node and edge node; for example, for a cloud node, its signal-to-noise ratio is 30dB, the number of child nodes is 4, and the delay is 15ms, and its weight is calculated to be a fixed value (such as 0.8); another node has a signal-to-noise ratio of 35dB, the number of child nodes is 2, and the delay is 25ms, and its weight is calculated to be 0.7;

[0063] Step 2: Define the weight distribution mechanism of the upper-level nodes, which is adapted to the heterogeneous network environment. A dynamic weight distribution algorithm is used to calculate the weight of the upper-level nodes from three aspects: signal-to-noise ratio, the number of sub-nodes under the node, and the average communication delay of multiple clients.

[0064] The signal-to-noise ratio (SNR) is used to measure the communication quality of the upper node in a wireless network environment. The specific formula is: the higher the signal-to-noise ratio (SNR), the stronger the communication capability of the upper node and the higher the weight;

[0065]

[0066] The network quality evaluation index based on the signal-to-noise ratio SNR in a wireless network environment, that is, the node importance weight based on the signal-to-noise ratio SNR in a wireless network environment, is as shown in the following formula:

[0067]

[0068] Where K is the number of other nodes related to the current node (such as node n), excluding the current node, and P s is the communication capability of the upper node s, P n is the communication capability of the upper node n, G k is the signal transmission capability of node k, G nis the ability of the upper node n to receive signals, PL is the path loss from node k to the upper node n, calculated by the empirical model, X is a random variable with mean 0 and variance σ 2 Normal distribution, is the additive Gaussian white noise between node communications, dB is the decibel of the random variable X, N is the total number of nodes, and covers all nodes (including the current node), SNR n is the signal-to-noise ratio of the upper node n, SNR i is the signal-to-noise ratio of node i, Weight iSNR is the network quality evaluation index of node i in the wireless network environment; the evaluation index Weight iSNR Consider the communication distance between upper-level nodes and the communication quality of nodes;

[0069] The empirical model is a logarithmic distance path loss model, which is used to estimate how the signal strength decays with distance during propagation.

[0070]

[0071] Where PL0 is the path loss at the reference distance d0; β is the path loss exponent, which depends on the propagation environment; d is the distance between the transmitter and the receiver;

[0072] The number of sub-nodes of the node is used to reflect the influence of the node as a proposer in the lower layer. The importance weight of the node in the network is calculated based on the number of sub-nodes of the node. The specific formula is: the more sub-nodes a node has, the greater the importance of the node and the higher the weight.

[0073]

[0074] Among them, Weight iNS is the importance weight of node i in the network; N n is the number of child nodes under node n, N i is the number of child nodes under node i;

[0075] The average communication delay of multiple clients is used to measure the speed at which a node responds to a client request; the lower the speed delay of the client request, the stronger the response capability of the node and the higher the weight, as shown in the following formula:

[0076]

[0077] Among them, d t is the average communication delay of multiple clients, C is the number of clients, d c,n is the communication delay between the cth client and node n, d i is the communication delay of node i;

[0078] The weight of the node based on communication delay during the consensus process is as shown in the following formula:

[0079]

[0080] Among them, Weight iMsg is the weight of node i in the consensus process based on communication delay;

[0081] In this embodiment, the influence of the signal-to-noise ratio, the number of sub-nodes under a node, and the average communication delay of multiple clients in the dynamic weight allocation algorithm are specifically as follows:

[0082] The impact of the signal-to-noise ratio: Assuming that the network environment is a wireless network environment, nodes with higher signal-to-noise ratios (e.g., 40 dB) will be given priority to obtain more weights; if the signal-to-noise ratio of a node is 50 dB, its weight allocation calculation method will be adjusted to be higher than that of the 40 dB node according to a preset algorithm;

[0083] The number of sub-nodes of the node: for example, one node has 6 sub-nodes and another node has 2 sub-nodes; according to the dynamic weight allocation algorithm, the more sub-nodes a node has, the greater the influence of the node, so the node with 6 sub-nodes will get a higher weight (for example, the weight is 0.9, while the weight of the node with 2 sub-nodes is 0.6);

[0084] The average communication delay of multiple clients: The delay of the node is also an important factor in calculating the weight; if the delay of a node is less than 10ms, its weight may be much higher than that of a node with a delay greater than 100ms. For example, a node with a delay of 5ms can be assigned a higher weight of 0.8, while a node with a delay of 50ms may be assigned a weight of 0.5;

[0085] Step 3: Modify the quorum set calculation method of the EPaxos protocol and assign a reasonable weight to each node, so that the distributed consensus system based on the Paxos protocol in the cloud-edge environment can maintain good high availability and high performance in both wired and wireless network environments and load conditions.

[0086] By comprehensively considering the signal-to-noise ratio, the number of sub-nodes under a node, and the average communication delay of multiple clients, the EPaxos protocol can assign a reasonable weight to each node, so that the distributed consensus system based on the Paxos algorithm in the entire cloud-edge environment can maintain good high availability and high performance when facing different network conditions and load conditions;

[0087] Due to Weight SNRThe main measure is the network quality in the wireless network environment. Therefore, in the face of both wireless network environment and wired network environment, the final node weight formula that comprehensively considers the signal-to-noise ratio, the number of sub-nodes under the node, and the average communication delay of multiple clients is:

[0088]

[0089] Among them, Weight i is the final comprehensive weight of the i-th node, and α is the node weight coefficient, which is used to weigh the relationship between high availability and request latency of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment;

[0090] High availability focuses on ensuring that the distributed consensus system based on the Paxos algorithm in the cloud-edge environment can still operate normally in the face of node failures or network fluctuations, while request latency is concerned with the response time of each client request. Improving the high availability of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment usually requires more message transmission and redundancy, which may increase latency. Reducing latency may sacrifice certain fault tolerance or availability. Therefore, α is used as a weight coefficient to adjust between the two to find the optimal balance.

[0091] When α=1, it means that the distributed consensus system based on the Paxos algorithm in the cloud-edge-end environment is more inclined to high availability of the system and has stronger fault tolerance. When α=0, it means that the distributed consensus system based on the Paxos algorithm in the cloud-edge-end environment is more inclined to system performance and hopes that the system can maintain lower request latency.

[0092] Under the new weight distribution mechanism, the arbitration set of the EPaxos protocol no longer depends solely on the number of nodes, but is calculated based on the sum of the weights of the nodes that are comprehensively considered. The weight of the node reflects the stability of the wired and wireless network environments, communication efficiency, and multi-client responsiveness, ensuring that the selection of the arbitration set is more adaptable to heterogeneous network environments.

[0093] The arbitration set formula of the EPaxos protocol is:

[0094]

[0095] Among them, Q is the arbitration set of the EPaxos protocol;

[0096] In this embodiment, the arbitration set calculation method of the EPaxos protocol is modified as follows:

[0097] G1: Selection of the quorum set: In the traditional EPaxos protocol, the quorum set is selected based on the number of nodes; by introducing a dynamic weight mechanism, the selection of the quorum set not only considers the number of nodes, but also the weight of each node; for example, when nodes with higher weights participate in the arbitration, they have greater influence, so their votes and confirmations will be given priority when making decisions;

[0098] G2: Weight adjustment: For example, in the decision-making process of cloud nodes, based on the weight of each node, the final arbitration set will include nodes with high signal-to-noise ratio, low latency, and many sub-nodes, thereby improving the fault tolerance and stability of the system; through this dynamic adjustment, it can ensure that the system can better adapt to changes in complex network environments;

[0099] Step 4: Use the delayed batch submission optimization strategy to optimize the submission strategy of the EPaxos protocol;

[0100] In order to improve the performance of the EPaxos protocol in high-conflict scenarios, an optimization strategy based on delayed batch submission is adopted, such as Figure 3 As shown, the specific process is as follows:

[0101] S1: Set the receiver node’s buffer and delayed reply parameters: After receiving the proposal, the receiver node does not process it immediately, but stores it in the local buffer and waits for the preset delay time (Reply Delay);

[0102] In this embodiment, when the acceptor node receives a proposal, it first stores the proposal in a local buffer. Assume that the buffer can hold up to 50 proposals and the delay time is 20ms. During this time, the node will not process the proposal, but wait for more proposals to arrive so that it can be processed in batches.

[0103] S2: Batch processing of proposals: After the delay time is over, the recipient node groups the proposals in the buffer according to the client number and data object, and submits them in batches after sorting;

[0104] In this embodiment, when the buffer reaches the set delay time, the recipient node will group the proposals by client number and data object; for example, the proposals from client A and client B are divided into two groups, and are sorted and submitted in batches according to the data object (such as product ID, order ID);

[0105] S3: Prioritize conflicting proposals: In conflict scenarios, the sorting priority strategy ensures that high-priority proposals can be submitted through the fastpath of the EPaxos protocol, reducing communication overhead and latency;

[0106] In this embodiment, it is assumed that the proposals proposed by nodes A and B conflict with each other; the nodes will prioritize proposals with fewer conflicts according to the preset sorting priority strategy; for example, if the proposal of node A involves a single data object, and the proposal of node B involves multiple data objects, the distributed consensus system may prioritize the proposal of node A to avoid unnecessary resource competition;

[0107] Step 5: Adopt an efficient proposal replication and leader switching mechanism based on the EPaxos protocol to ensure the high availability and stability of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment;

[0108] In the lower-layer Multi-Paxos protocol, heartbeat packets are used to detect node activity to ensure the activity of the current leader. If the leader is found to be invalid, the leader switching mechanism is triggered, thereby ensuring the high availability and stability of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment.

[0109] In this embodiment, the specific method of using heartbeat packets to detect node activity is as follows: each node will periodically send heartbeat packets to the decision-making layer to ensure that the leader node in the distributed consensus system is active; assuming that a heartbeat packet is sent every 10 seconds, if a node does not receive a heartbeat packet within 15 seconds, it is considered that the node is invalid and the leader switching mechanism is started;

[0110] The leader switching mechanism is as follows: after the leader fails, the distributed consensus system will select a new leader; for example, if the current leader node has a low weight and does not respond in time, the distributed consensus system will elect a node with a higher weight (for example, a node with a higher signal-to-noise ratio and lower latency) as the new leader to ensure that the system continues to operate normally;

[0111] Step 6: Dynamically adjust the weight distribution of upper-level nodes based on the real-time status and historical performance of the nodes, so that the distributed consensus system based on the Paxos algorithm in the cloud-edge environment can adapt to changes in the network environment and node failures, and maintain an efficient distributed consensus process;

[0112] In this embodiment, the specific method of dynamically adjusting the weight distribution of the upper-level nodes according to the real-time status of the nodes is as follows: the nodes dynamically adjust the weight distribution of the upper-level nodes according to their real-time status (such as network quality, delay, fault record); for example, if a node has experienced multiple timeouts or delays in the past 30 minutes, the distributed consensus system will automatically reduce its weight and adjust the distribution according to its current status;

[0113] The specific method of dynamically adjusting the weight distribution of upper-level nodes based on the historical performance of the nodes is as follows: if a node has not experienced any failures in the past two hours and has low communication latency with other nodes, then its weight may be increased and it may be given priority as a member of the arbitration set;

[0114] Step 7: Apply the batch-submitted EPaxos protocol in conflict-free and conflict-free scenarios, and improve the throughput and consensus success rate of the distributed consensus system based on the Paxos protocol in the cloud-edge environment by optimizing the proposal submission strategy;

[0115] In a conflict-free scenario, that is, there is no mutual dependence or conflict between multiple proposals (that is, these proposals can be submitted and processed independently), multiple conflict-free proposals are merged into a joint operation through a delayed batch submission strategy; in order to improve throughput and reduce communication overhead, whether to merge proposals is dynamically determined based on the current state of the node (such as latency and load), ensuring that the merged proposals come from nodes with good network quality and low latency, such as Figure 4 As shown;

[0116] In this embodiment, it is assumed that in the case of no conflict in the distributed consensus system, multiple requests will be merged into one batch for processing; for example, when there is no conflict in the requests of 10 clients, all proposals will be submitted together, thereby reducing the number of communications between nodes and improving throughput;

[0117] In conflict scenarios, when dealing with conflicting proposals (i.e. when there is resource contention or dependency between multiple proposals), the order of proposal processing is optimized through a sorting priority strategy to reduce the interdependence between proposals, thereby improving the consensus success rate. In this scenario, the weight of the node (based on its historical performance, network latency, load, etc.) will affect the priority of proposal sorting. Proposals from high-weight nodes will be processed first to ensure that the most reliable nodes in conflict scenarios reach consensus first, thereby improving the consensus efficiency and stability of the system and reducing communication overhead and latency. Figure 5 As shown;

[0118] In this embodiment, when processing conflicting proposals, the distributed consensus system will process them according to the sorting priority strategy; for example, when two proposals conflict, the distributed consensus system will select the proposal with less conflict to submit first, thereby reducing the possibility of falling back to the fast path and improving the consensus success rate;

[0119] In the experimental environment deployment, docker-compose was used to simulate the complex environment of cloud-edge-end multi-nodes. In order to simulate the network characteristics more realistically, the actual communication delay data in Table 1 was referred to during the simulation process, and the network traffic of the node container was artificially set using the network traffic control tool TC (Traffic Control);

[0120] Table 1 Communication delay between cloud server nodes deployed nationwide

[0121] Beijing Hohhot Shanghai Hangzhou Hongkong Guangzhou Beijing 0.204 17.548 26.941 30.944 44.818 45.769 Hohhot 17.548 0.145 43.115 37.118 63.501 63.287 Shanghai 26.941 43.115 0.275 8.719 38.049 32.764 Hangzhou 30.944 37.118 8.719 0.251 30.594 32.432 Hongkong 44.818 63.501 38.049 30.594 0.192 11.551 Guangzhou 45.769 63.287 32.764 32.432 11.551 0.185

[0122] Evenly distribute the nodes in the cluster to each data center (3 data centers are deployed for a 9-node cluster and 5 data centers are deployed for a 25-node cluster), ensuring that the nodes in the same data center are in the LAN, while the nodes across the same data center are in the WAN communication mode; during the network topology simulation process, the HTB (Hierarchical Token Bucket) queue rules are set in advance according to the source address and destination address of the data packet to achieve precise traffic control; in order to simulate the WAN communication during cloud-edge collaboration, the communication bandwidth between the cloud side and the edge side nodes is limited to 1OMBps; for LAN communication during edge-end collaboration, the communication bandwidth between the edge side and the end side is kept at the default 10GBps;

[0123] In order to enrich the experimental results of this embodiment, the TPC-C data set (Transaction Processing Performance Council Benchmark C) is additionally introduced here. This data set is a benchmark test widely recognized by the industry and is usually used to simulate the online transaction processing environment of the retail industry in the real world. This test benchmark covers a series of common transaction types, including new orders, payment orders, and mixed scenarios (mixed = new order + payment), and allows the proportion of cross-partition transactions to be adjusted to simulate different business scenarios.

[0124] For the TPC-C dataset, it can be concluded from the figure that the batch size is very sensitive to factors such as the number of node worker threads, workload type, and data distribution; Figure 6 (a) shows the relationship between batch size and the number of worker threads. In the default workload scenario, the graph shows that there is a turning point in the improvement of cluster performance by batch size. Linear fitting can give an optimal batch size in practice:

[0125] Batch size =125*Num threads +500 (10)

[0126] Figure 6 (b)- Figure 6 (d) discusses the transaction type, that is, the impact of transaction operation type and cross-partition transaction on batch size; Figure 6 The mixed transaction in (b) consists of 80% new order transactions and 20% payment order transactions. In the standard new order scenario, insertion operations account for the majority, while the payment scenario involves calculating and modifying multiple data (inventory, balance, etc.). Compared with new orders, the payment business logic is more complex. If the two share the same Batch size Parameters often do not result in optimal performance; Figure 6 (c) and (d) further analyze the impact of cross-partition transactions on performance. The concurrency characteristics brought by cross-partition transactions also have a significant impact on the selection of batch size. Combining the analysis results of these three subgraphs, when the default cluster can tolerate a delay of 10000us, the optimal batch size calculation formula can be obtained as follows:

[0127] Batch size =1250×x 2 -250×x+500 (x represents the proportion of new orders) (11)

[0128] For the scalability analysis of the present invention, the average delay performance of different algorithms under a 25-node configuration is shown; the following table represents the range of system delay (in ms), thus reflecting the stability and reliability of the protocol performance;

[0129] The layered approach of the multi-Paxos protocol used in the experimental results measurement decomposes the overall delay into two parts: decision time and proposal replication delay. The decision time is mainly responsible for ensuring the security and activity of the protocol, while the proposal replication delay is related to the availability of the system. Since the number of nodes in the decision layer is reduced and only the cloud and edge layers are involved, the delay fluctuation range is reduced. Considering the characteristics of the wide area network environment, the EPaxos protocol used in the decision layer has a parameter α set to 0.5, which makes the performance reach the same level as the 5-node EPaxos.

[0130] Table 2 Performance of hybrid Paxos protocol in cloud-edge environment

[0131] Number of nodes Paxos EPaxos Hybrid-Paxos-Hide Hybrid-Paxos 5 23-201 20-210 10-75 53-174 25 5-890 5-780 3-104 40-470

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A highly scalable consensus method based on the Paxos protocol in a cloud-edge environment, characterized by: The following steps are involved: Step 1: Under the cloud-edge-device architecture, divide the node hierarchy of the distributed consensus system into upper and lower layers; Step 2: Define the weight distribution mechanism of the upper-level nodes, which is adapted to the heterogeneous network environment. A dynamic weight distribution algorithm is used to calculate the weight of the upper-level nodes from three aspects: signal-to-noise ratio, the number of sub-nodes under the node, and the average communication delay of multiple clients. Step 3: Modify the quorum set calculation method of the EPaxos protocol and assign a reasonable weight to each node, so that the distributed consensus system based on the Paxos protocol in the cloud-edge environment can maintain high availability and high performance in both wired and wireless network environments and load conditions; Step 4: Use the delayed batch submission optimization strategy to optimize the submission strategy of the EPaxos protocol; Step 5: Adopt an efficient proposal replication and leader switching mechanism based on the EPaxos protocol to ensure the high availability and stability of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment; Step 6: Dynamically adjust the weight distribution of upper-level nodes based on the real-time status and historical performance of the nodes, so that the distributed consensus system based on the Paxos algorithm in the cloud-edge environment can adapt to changes in the network environment and node failures, and maintain an efficient distributed consensus process; Step 7: Apply the batch-submitted EPaxos protocol in conflict-free and conflict-free scenarios, and improve the throughput and consensus success rate of the distributed consensus system based on the Paxos protocol in the cloud-edge environment by optimizing the proposal submission strategy.

2. According to claim 1, a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment is characterized by: The upper layer is the decision-making layer in the cloud, which adopts the EPaxos protocol and is responsible for voting and confirming the proposals of the lower-layer nodes; The lower layer is the edge replication layer, which adopts the Multi-Paxos protocol and is responsible for replicating and executing the proposals put forward by the upper-layer nodes.

3. According to claim 1, a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment is characterized by: The signal-to-noise ratio (SNR) described in step 2 is used to measure the communication quality of the upper node in the wireless network environment. The specific formula is: The network quality evaluation index based on the signal-to-noise ratio SNR in a wireless network environment, that is, the node importance weight based on the signal-to-noise ratio SNR in a wireless network environment, is as shown in the following formula: Where K is the number of other nodes related to the current node, excluding the current node, and P s is the communication capability of the upper node s, P n is the communication capability of the upper node n, G k is the signal transmission capability of node k, G n is the ability of the upper node n to receive signals, PL is the path loss from node k to the upper node n, calculated by the empirical model, X is a random variable with mean 0 and variance σ 2 The normal distribution of is the additive Gaussian white noise between node communications, dB is the decibel of the random variable X, N is the total number of nodes, and covers all nodes, SNR n is the signal-to-noise ratio of the upper node n, SNR i is the signal-to-noise ratio of node i, Weight iSNR is the network quality evaluation index of node i in the wireless network environment; The number of sub-nodes of the node is used to reflect the influence of the node as a proposer in the lower layer. The importance weight of the node in the network is calculated based on the number of sub-nodes of the node. The specific formula is: Among them, Weight iNS is the importance weight of node i in the network; N n is the number of child nodes under node n, N i is the number of child nodes under node i; The average communication delay of multiple clients is used to measure the speed at which a node responds to a client request, as shown in the following formula: Among them, d t is the average communication delay of multiple clients, C is the number of clients, d c,n is the communication delay between the cth client and node n, d i is the communication delay of node i; The weight of the node based on communication delay during the consensus process is shown in the following formula; Among them, Weight iMsg is the weight of node i in the consensus process based on communication delay.

4. According to claim 3, a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment is characterized by: The specific method of step 3 is: In the face of both wireless network environment and wired network environment, the final node weight formula that comprehensively considers the signal-to-noise ratio, the number of sub-nodes under the node, and the average communication delay of multiple clients is: Among them, Weight i is the final comprehensive weight of the i-th node, and α is the node weight coefficient, which is used to weigh the relationship between high availability and request latency of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment; Under the new weight distribution mechanism, the arbitration set of the EPaxos protocol no longer depends solely on the number of nodes, but is calculated based on the sum of the weights of the nodes that are comprehensively considered at the end; The arbitration set formula of the EPaxos protocol is: Among them, Q is the arbitration set of the EPaxos protocol.

5. According to claim 4, a highly scalable consensus method based on the Paxos protocol in a cloud-edge environment is characterized by: The specific process of step 4 is: S1: Set the receiver node’s buffer and delayed reply parameters: After receiving a proposal, the receiver node does not process it immediately, but stores it in the local buffer and waits for the preset delay time; S2: Batch processing of proposals: After the delay time is over, the recipient node groups the proposals in the buffer according to the client number and data object, and submits them in batches after sorting; S3: Prioritize conflicting proposals: In conflict scenarios, the sorting priority strategy ensures that high-priority proposals can be submitted through the fast path of the EPaxos protocol, reducing communication overhead and latency.

6. According to claim 5, a highly scalable consensus method based on the Paxos protocol in a cloud-edge environment is characterized by: The specific method of step 5 is: In the lower-layer Multi-Paxos protocol, heartbeat packets are used to detect node activity to ensure the activity of the current leader; if the leader is found to be invalid, the leader switching mechanism is triggered, thereby ensuring the high availability and stability of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment.

7. According to claim 6, a highly scalable consensus method based on the Paxos protocol in a cloud-edge environment is characterized by: In the non-conflict scenario, step 7 merges multiple non-conflict proposals into one joint operation through a delayed batch submission strategy; dynamically determines whether to merge proposals based on the current state of the node to ensure that the merged proposals come from nodes with good network quality and low latency.

8. According to claim 6, a highly scalable consensus method based on the Paxos protocol in a cloud-edge environment is characterized by: In the above step 7, in the conflict scenario, when processing conflicting proposals, the processing order of proposals is optimized through the sorting priority strategy, the mutual dependence between proposals is reduced, and the consensus success rate is improved; in this scenario, the weight of the node will affect the priority of the proposal sorting; the proposals of high-weight nodes will be processed first to ensure that the most reliable nodes in the conflict scenario reach consensus first, thereby improving the consensus efficiency and stability of the system and reducing communication overhead and delay.

Images