A highly scalable consensus method based on paxos protocol in cloud edge environment

Abstract

The application provides a high-extensible consensus method based on a Paxos protocol in a cloud edge-end environment, and belongs to the technical field of distributed computing, edge computing and cloud computing. The method divides the node level of a distributed consensus system into an upper layer and a lower layer; the weight of the upper layer node is calculated; the arbitration set calculation method of the EPaxos protocol is modified to make the distributed consensus system maintain high availability and high performance; an optimization strategy based on delay batch submission is adopted to optimize the submission strategy of the EPaxos protocol; the proposal replication and leader switching mechanism based on the EPaxos protocol guarantee the high availability and stability of the distributed consensus system; the weight distribution of the upper layer node is dynamically adjusted according to the real-time state and historical performance of the node, so that the distributed consensus system can adapt to the change of the network environment and node failure; and the throughput and consensus success rate of the distributed consensus system are improved by optimizing the proposal submission strategy.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing, edge computing and cloud computing technologies, and in particular to a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment. Background Technology

[0002] In traditional cross-regional distributed cloud storage scenarios, conventional state machine replication protocols require multiple rounds of wide area network communication to establish a global transaction execution order, leading to increased latency in cluster request responses. To address this issue, the Weave protocol (Eischer M, Straßner B, Distler T. Low-latency geo-replicated state machines with guaranteed writes [C] / / Proceedings of the 7th Workshop on Principles and Practice of Consistency for Distributed Data. 2020: 1-9.) employs a multi-leader model. This protocol builds replica groups in different geographical regions and designates a local leader in each group, responsible for assigning sequence numbers to requests within that region. This effectively reduces the need for cross-regional communication and significantly lowers request processing latency. The Weave protocol also supports multiple transaction consistency levels, allowing clients to balance response time and data consistency. Nomad (Hao Z, Yi S, Li Q. Nomad: An efficient consensus approach for latency-sensitive edge-cloud applications[C] / / IEEE INFOCOM 2019-IEEE Conference on Computer Communications.IEEE, 2019:2539-2547.) is specifically optimized for latency-sensitive applications in edge computing. It employs a lease-based master-slave protocol and a dynamic leadership allocation strategy to achieve fast transaction ordering. This protocol dynamically adjusts the leadership of edge nodes using an adaptive Paxos protocol based on historical system logs. By assigning cloud nodes the role of arbitrators, cloud data centers can flexibly switch between master-slave modes based on real-time monitored edge network workloads, further improving overall system performance and reliability.

[0003] Regarding fault tolerance optimization in hybrid cloud environments, the SeeMoRe (Amiri MJ, Maiyya S, Agrawal D, et al. Seemore: A fault-tolerant protocol for hybrid cloud environments[C] / / 2020 IEEE 36th International Conference on Data Engineering (ICDE). IEEE, 2020: 1345-1356.) protocol considers the characteristics of private clouds where node crashes and downtime may occur, and public clouds where malicious nodes may be present. It dynamically adjusts the operating mode (Trusted Master Node Centralized Coordination (TPCC) mode / Trusted Master Node Distributed Coordination (TPDC) mode / Untrusted Master Node Distributed Coordination (UPDC) mode) according to the workload of the private cloud and the communication latency between clouds, thereby effectively dealing with the risk of malicious failures in the cluster while maintaining system stability.

[0004] Existing consensus protocols typically assume that all nodes are peers and that the network is fully connected. However, when the replicating state machine encounters network partitions or partial node connectivity, service may be interrupted, thus hindering the consensus process. The Omni-Paxos protocol (Ng H, Haridi S, Carbone P. Omni-Paxos: Breaking the barriers of partial connectivity [C] / / Proceedings of the Eighteenth European Conference on Computer Systems. 2023: 314-330.) improves the robustness of the consensus protocol in the face of network partitions and unstable connections by decoupling and optimizing the log replication, leader election, and system reconfiguration modules respectively. Its core innovation lies in the adoption of the BLE (Ballot Leader Election) mechanism, which ensures that a leader with good connectivity to a majority of nodes is elected during network changes, thereby improving the system's adaptability to dynamic network environments while maintaining data consistency.

[0005] The Paxos protocol performs well in distributed systems, especially in ensuring system availability. However, it still faces several key challenges. First, to guarantee the correctness of the protocol, Paxos requires building redundant arbitration sets, which can lead to resource waste as the system scales. Second, the protocol relies on a leader node to coordinate consensus, but the leader node can become a single point of failure or a performance bottleneck, directly affecting the stability and efficiency of the system. Finally, in complex heterogeneous network environments, Paxos's scalability is limited; network latency and uneven bandwidth can reduce message passing efficiency, further impacting the protocol's performance. These issues limit the widespread adoption of the Paxos protocol in practical applications. Summary of the Invention

[0006] The technical problem this invention aims to solve is to address the shortcomings of existing technologies by providing a highly scalable consensus method based on the Paxos protocol in cloud-edge-device environments. This method addresses the diversity of network environments, heterogeneous node performance, and the concurrency of multiple client requests by introducing a multi-Paxos cluster hierarchical strategy, assigning weights to EPaxos protocol nodes, and employing a proposal batch submission approach. This enables the Paxos-based distributed consensus system to more flexibly respond to different network environments and load demands, effectively improving its adaptability to high-concurrency scenarios and overcoming the insufficient capacity of existing distributed consensus systems to handle complex network environments and high-concurrency requests. Simultaneously, this method introduces dynamic load balancing and asynchronous processing mechanisms on the server side, enabling the server to efficiently handle concurrent client requests. Furthermore, by optimizing the proposal batch submission strategy and multi-cluster coordination mechanism, the system's performance and stability are further improved.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment, comprising the following steps:

[0008] Step 1: In the cloud-edge-device architecture, divide the node hierarchy of the distributed consensus system into upper and lower layers;

[0009] The upper layer is the cloud-based decision-making layer, which uses the EPaxos protocol and is responsible for voting on and confirming proposals from lower-level nodes.

[0010] The lower layer is the edge replication layer, which adopts the Multi-Paxos protocol and is responsible for replicating and executing the proposals put forward by the upper-layer nodes;

[0011] Step 2: Define the weight allocation mechanism for upper-layer nodes to adapt to heterogeneous network environments. Use a dynamic weight allocation algorithm to calculate the weight of upper-layer nodes from three aspects: signal-to-noise ratio, number of child nodes attached to a node, and average communication latency of multiple clients.

[0012] The signal-to-noise ratio (SNR) is used to measure the communication quality of upper-layer nodes in a wireless network environment. The specific formula is as follows:

[0013]

[0014] The network quality evaluation index based on signal-to-noise ratio (SNR) in a wireless network environment, i.e., the node importance weight based on SNR in a wireless network environment, is shown in the following formula:

[0015]

[0016] Where K is the number of other nodes related to the current node, excluding the current node, and P s For the communication capabilities of the upper-level node s, P n For the communication capability of the upper-level node n, G k G is the ability of node k to transmit signals. n PL represents the signal reception capability of the upper-level node n, PL represents the path loss from node k to the upper-level node n, calculated by an empirical model, and X is a random variable with a mean of 0 and a variance of σ. 2 The normal distribution is represented by additive white Gaussian noise between nodes, dB is the decibel of the random variable X, N is the total number of nodes, and it covers all nodes. SNR n Let SNR be the signal-to-noise ratio of the upper-layer node n. i Let Weight be the signal-to-noise ratio of node i. iSNR This is a network quality evaluation index for node i in a wireless network environment.

[0017] The number of child nodes attached to a node reflects its influence as a proposer at the lower level. The importance weight of a node in the network is calculated based on the number of its child nodes, using the following formula:

[0018]

[0019] Among them, Weight iNS N represents the importance weight of node i in the network. n N is the number of child nodes of node n. i Let i be the number of child nodes attached to node i.

[0020] The average communication latency across multiple clients is used to measure the speed at which a node responds to client requests, as shown in the following formula:

[0021]

[0022] Where, d t Let C be the average communication latency across multiple clients, and d be the number of clients. c,nLet d be the communication delay between the c-th client and node n. i Let be the communication delay of node i;

[0023] The weights of nodes based on communication latency during the consensus process are shown in the following formula;

[0024]

[0025] Among them, Weight iMsg Let i be the weight of node i in the consensus process based on communication latency;

[0026] Step 3: Modify the arbitration set calculation method of the EPaxos protocol to assign a reasonable weight to each node, so that the distributed consensus system based on the Paxos protocol in the cloud-edge-device environment can maintain high availability and high performance when facing wired network environment and wireless network environment and load conditions.

[0027] Considering both wireless and wired network environments, the final node weight formula, which comprehensively considers three aspects—signal-to-noise ratio, the number of child nodes connected to a node, and the average communication latency for multiple clients—is as follows:

[0028]

[0029] Among them, Weight i The final weight of the i-th node is determined by comprehensive consideration, and α is the node weight coefficient, which is used to balance the relationship between high availability and request latency in a distributed consensus system based on the Paxos algorithm in a cloud-edge-device environment.

[0030] Under the new weight allocation mechanism, the arbitration set of the EPaxos protocol no longer depends solely on the number of nodes, but is calculated based on the sum of the weights of the nodes after comprehensive consideration.

[0031] The arbitration set formula for the EPaxos protocol is:

[0032]

[0033] Where Q is the arbitration set of the EPaxos protocol;

[0034] Step 4: Optimize the EPaxos protocol's commit strategy using a delayed batch commit-based optimization strategy. The specific process is as follows:

[0035] S1: Set the buffer and delayed response parameters for the receiver node: After receiving a proposal, the receiver node does not process it immediately, but stores it in a local buffer and waits for a preset delay time.

[0036] S2: Batch processing of proposals: After the delay time ends, the recipient node will group the proposals in the buffer according to the client number and data object, sort them, and then submit them in batches;

[0037] S3: Prioritize conflicting proposals: In conflict scenarios, the priority strategy ensures that high-priority proposals can be submitted through the fast path of the EPaxos protocol, reducing communication overhead and latency;

[0038] Step 5: Employ an efficient proposal replication and leader switching mechanism based on the EPaxos protocol to ensure the high availability and stability of the Paxos-based distributed consensus system in a cloud-edge-device environment;

[0039] In the underlying Multi-Paxos protocol, heartbeat packets are used to detect node activity and ensure the current leader's activity. If the leader fails, a leader switching mechanism is triggered, thereby ensuring the high availability and stability of the Paxos-based distributed consensus system in cloud-edge-device environments.

[0040] Step 6: Based on the real-time status and historical performance of the nodes, dynamically adjust the weight distribution of the upper-layer nodes so that the distributed consensus system based on the Paxos algorithm in the cloud-edge-device environment can adapt to changes in the network environment and node failures, and maintain an efficient distributed consensus process.

[0041] Step 7: Application of the EPaxos protocol based on batch submission in conflict-free and conflict scenarios. Improve the throughput and consensus success rate of the Paxos protocol-based distributed consensus system in cloud-edge-device environments by optimizing the proposal submission strategy.

[0042] In conflict-free scenarios, multiple conflict-free proposals are merged into a single joint operation through a delayed batch commit strategy. The current state of the nodes is dynamically used to determine whether to merge proposals, ensuring that the merged proposals come from nodes with good network quality and low latency.

[0043] In conflict scenarios, when processing conflicting proposals, a priority-based sorting strategy is used to optimize the processing order of proposals, reduce interdependencies between proposals, and thus improve the consensus success rate. In this scenario, the weight of a node will affect the priority of proposal sorting; proposals from high-weight nodes will be processed first to ensure that the most reliable nodes in the conflict scenario reach consensus first, thereby improving the system's consensus efficiency and stability, and reducing communication overhead and latency.

[0044] The beneficial effects of adopting the above technical solution are as follows: This invention provides a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment. This method, by designing a hybrid Paxos protocol, addresses the data consistency problem in heterogeneous network environments of cloud, edge, and device, and proposes the following key innovations:

[0045] (1) Layered consensus mechanism: Based on the multi-Paxos protocol, this invention designs a layered consensus mechanism. This mechanism divides cloud-edge-end nodes into a decision layer and a replication layer. The decision layer is responsible for voting and confirming proposals, while the replication layer is responsible for storing and executing proposals. Through this layered design, the heterogeneity of different nodes and the complexity of the network environment are fully considered, and the consensus efficiency of the protocol is optimized.

[0046] (2) Weighted EPaxos Improvement Strategy: To address the issue of neglecting node heterogeneity in traditional consensus protocols, this invention introduces a weighted EPaxos improvement strategy. This strategy dynamically adjusts the weight allocation of each node in the consensus process based on factors such as signal-to-noise ratio, the number of child nodes attached to a node, and the average communication latency of multiple clients. Through this weighting mechanism, the high availability and performance of the system are improved, enabling it to maintain stable and efficient operation even in complex network environments.

[0047] (3) EPaxos optimization based on batch submission: In response to the performance bottleneck of the EPaxos protocol in high-conflict scenarios, this invention proposes an optimization strategy based on delayed batch submission. By setting a buffer on the receiver node and introducing a delayed submission mechanism, the system can process proposals in batches, reduce conflicts between proposals, reduce communication overhead and latency, thereby improving the overall performance of the EPaxos protocol.

[0048] Compared with existing technologies, the hybrid Paxos protocol proposed in this invention demonstrates significant performance improvements in heterogeneous cloud-edge-device network environments. Through a layered consensus mechanism and batch commit optimization strategy, it effectively reduces latency in heterogeneous cloud-edge-device networks and increases the throughput of Paxos-based distributed consensus systems in such environments. Simulation experiments successfully simulated real-world network characteristics using Docker to simulate complex network environments, combined with HTB queue rules and TC flow control tools. In high-conflict scenarios, the Paxos-based distributed consensus system in cloud-edge-device environments employing the batch commit strategy significantly reduces transaction commit latency, with the optimization effect becoming more pronounced as the probability of conflict increases. Furthermore, the stability and reliability of the EPaxos and Multi-Paxos protocols in wide area network communication environments have been verified, significantly reducing latency fluctuations compared to traditional Paxos protocols while ensuring system security and activity. This invention provides an efficient and reliable data consistency solution for distributed systems in cloud-edge-device collaborative environments by optimizing protocol scalability. Attached Figure Description

[0049] Figure 1 A flowchart of a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment is provided in this embodiment of the invention;

[0050] Figure 2 A schematic diagram of the hybrid Paxos protocol under a cloud-edge-device heterogeneous network provided in this embodiment of the invention;

[0051] Figure 3 The present invention provides a schematic diagram of optimizing the EPaxos protocol submission strategy using an optimization strategy based on delayed batch submission, wherein (a) - the fast path in the EPaxos protocol, and (b) - the slow path in the EPaxos protocol;

[0052] Figure 4 A schematic diagram illustrating the application of the batch-submit-based EPaxos protocol in a conflict-free scenario provided in this embodiment of the invention;

[0053] Figure 5 A schematic diagram illustrating the application of the batch-submit-based EPaxos protocol in a conflict scenario, as provided in this embodiment of the invention.

[0054] Figure 6 The impact of TPC-C dataset batch size on cluster performance provided in this embodiment of the invention includes: (a) the relationship between the number of node threads and batch size; (b) the relationship between workload and batch size; (c) the relationship between new order workload and batch size; and (d) the relationship between payment order workload and batch size. Detailed Implementation

[0055] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0056] In this embodiment, a highly scalable consensus method based on the Paxos protocol in a cloud-edge-device environment is applied to abnormal vehicle detection, such as... Figure 1 As shown, it includes the following steps:

[0057] Step 1: In a cloud-edge-device architecture, divide the node hierarchy of the distributed consensus system into upper and lower layers, such as... Figure 2 As shown;

[0058] The upper layer is the cloud-based decision-making layer, which adopts the EPaxos protocol and is responsible for voting on and confirming proposals from lower-level nodes. Unlike the traditional EPaxos protocol, upper-level nodes are assigned different weights to reflect their influence in the decision-making process.

[0059] The lower layer is the edge replication layer, which adopts the Multi-Paxos protocol and is responsible for replicating and executing the proposals proposed by the upper-layer nodes. The lower-layer nodes are assumed to have similar performance and reliability as the upper-layer nodes to ensure the stability and consistency of the protocol.

[0060] In this embodiment, the EPaxos protocol is an improved consensus protocol. Unlike the multi-Paxos cluster hierarchical strategy, the EPaxos protocol improves performance through leaderless design and conflict minimization optimization, while the Multi-Paxos protocol is an extension of the classic Paxos protocol optimized for a single master node. The two have different goals and implementation methods.

[0061] In this embodiment, the node hierarchy is divided as follows: Assuming there are 12 nodes in the distributed consensus system, 3 of which are in the cloud and 9 are in the edge; the cloud nodes constitute the decision layer, responsible for voting on and confirming the proposals of the lower-level nodes; the edge nodes are responsible for replicating and executing the proposals proposed by the upper-level nodes; each node is configured according to its role in the distributed consensus system, with the cloud nodes acting as decision-makers and the edge nodes acting as executors.

[0062] In this embodiment, the method for assigning different weights is as follows: assign an initial weight to each cloud node and edge node; for example, for a cloud node with a signal-to-noise ratio of 30dB, 4 child nodes, and a latency of 15ms, its weight is calculated to be a certain fixed value (such as 0.8); for another node with a signal-to-noise ratio of 35dB, 2 child nodes, and a latency of 25ms, its weight is calculated to be 0.7.

[0063] Step 2: Define the weight allocation mechanism for upper-layer nodes to adapt to heterogeneous network environments. Use a dynamic weight allocation algorithm to calculate the weight of upper-layer nodes from three aspects: signal-to-noise ratio, number of child nodes attached to a node, and average communication latency of multiple clients.

[0064] The signal-to-noise ratio (SNR) is used to measure the communication quality of upper-layer nodes in a wireless network environment. The specific formula is: the higher the SNR, the stronger the communication capability of the upper-layer node, and the higher its weight.

[0065]

[0066] The network quality evaluation index based on signal-to-noise ratio (SNR) in a wireless network environment, i.e., the node importance weight based on SNR in a wireless network environment, is shown in the following formula:

[0067]

[0068] Where K is the number of other nodes related to the current node (e.g., node n), excluding the current node, and P s For the communication capabilities of the upper-level node s, P n For the communication capability of the upper-level node n, G k G is the ability of node k to transmit signals. nPL represents the signal reception capability of the upper-level node n, PL represents the path loss from node k to the upper-level node n, calculated by an empirical model, and X is a random variable with a mean of 0 and a variance of σ. 2 The normal distribution is represented by additive white Gaussian noise in node communication, dB is the decibel value of the random variable X, N is the total number of nodes, and it covers all nodes (including the current node). SNR n Let SNR be the signal-to-noise ratio of the upper-layer node n. i Let Weight be the signal-to-noise ratio of node i. iSNR Weight is a network quality evaluation index for node i in a wireless network environment. iSNR Consider the communication distance between upper-layer nodes and the communication quality between nodes;

[0069] The empirical model is a logarithmic distance path loss model, used to estimate how signal strength attenuates with distance during propagation.

[0070]

[0071] Where PL0 is the path loss at the reference distance d0; β is the path loss exponent, which depends on the propagation environment; and d is the distance between the transmitter and the receiver.

[0072] The number of child nodes attached to a node is used to reflect the node's influence as a proposer at the lower level. The importance weight of a node in the network is calculated based on the number of child nodes attached to it. The specific formula is: the more child nodes a node has, the greater its importance and the higher its weight.

[0073]

[0074] Among them, Weight iNS N represents the importance weight of node i in the network. n N is the number of child nodes of node n. i Let i be the number of child nodes attached to node i.

[0075] The average communication latency across multiple clients is used to measure the speed at which a node responds to client requests; the lower the latency of client requests, the stronger the node's response capability, and the higher its weight, as shown in the following formula:

[0076]

[0077] Where, d t Let C be the average communication latency across multiple clients, and d be the number of clients. c,n Let d be the communication delay between the c-th client and node n. i Let be the communication delay of node i;

[0078] The weights of nodes in the consensus process based on communication latency are shown in the following formula:

[0079]

[0080] Among them, Weight iMsg Let i be the weight of node i in the consensus process based on communication latency;

[0081] In this embodiment, the three aspects of the dynamic weight allocation algorithm—the influence of signal-to-noise ratio, the number of child nodes attached to a node, and the average communication latency of multiple clients—are specifically as follows:

[0082] The impact of signal-to-noise ratio (SNR): Assuming the network environment is a wireless network, nodes with a higher SNR (e.g., 40dB) will preferentially receive more weight; if a node has an SNR of 50dB, its weight allocation calculation method will be adjusted according to the preset algorithm to be higher than that of a node with an SNR of 40dB.

[0083] The number of child nodes of a node: For example, one node has 6 child nodes and another node has 2 child nodes; According to the dynamic weight allocation algorithm, the more child nodes a node has, the greater its influence. Therefore, a node with 6 child nodes will get a higher weight (for example, a weight of 0.9, while a node with 2 child nodes has a weight of 0.6).

[0084] The average communication latency of multiple clients: Node latency is also an important factor in calculating weights; if a node's latency is less than 10ms, its weight may be much higher than a node with a latency greater than 100ms. For example, a node with a latency of 5ms can be assigned a higher weight of 0.8, while a node with a latency of 50ms may be assigned a weight reduced to 0.5.

[0085] Step 3: Modify the arbitration set calculation method of the EPaxos protocol to assign a reasonable weight to each node, so that the Paxos-based distributed consensus system in cloud-edge-device environments can maintain good high availability and high performance in wired and wireless network environments and under different load conditions.

[0086] By comprehensively considering the signal-to-noise ratio, the number of child nodes attached to a node, and the average communication latency of multiple clients, the EPaxos protocol can assign a reasonable weight to each node, enabling the distributed consensus system based on the Paxos algorithm in the entire cloud-edge-device environment to maintain good high availability and high performance when facing different network conditions and load situations.

[0087] Due to Weight SNRThe primary focus is on network quality in wireless network environments. Therefore, considering both wireless and wired network environments, the final node weight formula, which comprehensively considers three aspects—signal-to-noise ratio, the number of child nodes attached to a node, and the average communication latency across multiple clients—is as follows:

[0088]

[0089] Among them, Weight i The final weight of the i-th node is determined by comprehensive consideration, and α is the node weight coefficient, which is used to balance the relationship between high availability and request latency in a distributed consensus system based on the Paxos algorithm in a cloud-edge-device environment.

[0090] High availability focuses on ensuring that a Paxos-based distributed consensus system in a cloud-edge-device environment can still operate normally in the face of node failures or network fluctuations, while request latency concerns the response time of each client request. Improving the high availability of a Paxos-based distributed consensus system in a cloud-edge-device environment usually requires more message passing and redundancy, which may increase latency; while reducing latency may sacrifice some fault tolerance or availability. Therefore, α is used as a weighting coefficient to adjust between the two to find the optimal balance.

[0091] When α = 1, it means that the distributed consensus system based on the Paxos algorithm in the cloud-edge-device environment is more inclined to high availability and has stronger fault tolerance; when α = 0, it means that the distributed consensus system based on the Paxos algorithm in the cloud-edge-device environment is more inclined to system performance and hopes that the system can maintain lower request latency.

[0092] Under the new weight allocation mechanism, the arbitration set of the EPaxos protocol no longer depends solely on the number of nodes, but is calculated based on the sum of the weights of the nodes after comprehensive consideration. The weights of the nodes reflect the stability, communication efficiency, and multi-client responsiveness of wired and wireless network environments, ensuring that the selection of the arbitration set is more adaptable to heterogeneous network environments.

[0093] The arbitration set formula for the EPaxos protocol is:

[0094]

[0095] Where Q is the arbitration set of the EPaxos protocol;

[0096] In this embodiment, the arbitration set calculation method of the EPaxos protocol is modified as follows:

[0097] G1: Selection of the Arbitration Set: In the traditional EPaxos protocol, the arbitration set is selected based on the number of nodes; by introducing a dynamic weighting mechanism, the selection of the arbitration set not only considers the number of nodes, but also the weight of each node; for example, when a node with a higher weight participates in arbitration, its influence is greater, so its votes and confirmations will be given priority when making decisions.

[0098] G2: Weight Adjustment: For example, in the decision-making process of cloud nodes, based on the weight of each node, the final arbitration assembly includes nodes with high signal-to-noise ratio, low latency, and many subordinate child nodes, thereby improving the fault tolerance and stability of the system; through this dynamic adjustment, it can be ensured that the system can better adapt to changes when facing complex network environments.

[0099] Step 4: Optimize the EPaxos protocol's commit strategy using a delayed batch commit-based optimization strategy;

[0100] To improve the performance of the EPaxos protocol in high-collision scenarios, an optimization strategy based on delayed batch commit is adopted, such as... Figure 3 As shown, the specific process is as follows:

[0101] S1: Set the buffer and delayed reply parameters for the receiver node: After receiving a proposal, the receiver node does not process it immediately, but stores it in the local buffer and waits for the preset delay time (Reply Delay);

[0102] In this embodiment, when the receiver node receives a proposal, it first stores the proposal in its local buffer. Assuming the buffer can hold a maximum of 50 proposals and the delay time is 20ms, the node will not process the proposals during this time, but will wait for more proposals to arrive so that it can process them in batches.

[0103] S2: Batch processing of proposals: After the delay time ends, the recipient node will group the proposals in the buffer according to the client number and data object, sort them, and then submit them in batches;

[0104] In this embodiment, when the buffer reaches the set delay time, the receiver node will group the proposals according to the client number and data object; for example, the proposals from client A and client B are divided into two groups, and sorted and submitted in batches according to the data object (e.g., product ID, order ID);

[0105] S3: Prioritize conflicting proposals: In conflict scenarios, the sorting priority strategy ensures that high-priority proposals can be submitted through the fast path of the EPaxos protocol, reducing communication overhead and latency;

[0106] In this embodiment, it is assumed that the proposals put forward by node A and node B conflict. The nodes will prioritize the proposals with fewer conflicts according to the preset sorting priority strategy. For example, if node A's proposal involves a single data object, while node B's proposal involves multiple data objects, the distributed consensus system may prioritize node A's proposal to avoid unnecessary resource competition.

[0107] Step 5: Employ an efficient proposal replication and leader switching mechanism based on the EPaxos protocol to ensure the high availability and stability of the Paxos-based distributed consensus system in a cloud-edge-device environment;

[0108] In the underlying Multi-Paxos protocol, heartbeat packets are used to detect node activity and ensure the current leader's activity. If the leader fails, a leader switching mechanism is triggered, thereby ensuring the high availability and stability of the Paxos-based distributed consensus system in cloud-edge-device environments.

[0109] In this embodiment, the specific method for using heartbeat packets to detect node activity is as follows: each node will periodically send heartbeat packets to the decision layer to ensure that the leader node in the distributed consensus system is active; assuming that a heartbeat packet is sent every 10 seconds, if a node does not receive a heartbeat packet within 15 seconds, it is considered that the node has failed and the leader switching mechanism is initiated.

[0110] The leader switching mechanism is as follows: after the leader fails, the distributed consensus system will select a new leader; for example, assuming that the current leader node has a low weight and does not respond in time, the distributed consensus system will elect a node with a higher weight (e.g., a node with a higher signal-to-noise ratio and lower latency) as the new leader to ensure that the system continues to operate normally.

[0111] Step 6: Based on the real-time status and historical performance of the nodes, dynamically adjust the weight distribution of the upper-layer nodes so that the distributed consensus system based on the Paxos algorithm in the cloud-edge-device environment can adapt to changes in the network environment and node failures, and maintain an efficient distributed consensus process.

[0112] In this embodiment, the specific method for dynamically adjusting the weight allocation of upper-layer nodes based on the real-time status of nodes is as follows: the weight allocation of upper-layer nodes is dynamically adjusted based on their real-time status (such as network quality, latency, and fault records); for example, if a node experiences multiple timeouts or delays in the past 30 minutes, the distributed consensus system will automatically reduce its weight and adjust the allocation based on its current status.

[0113] The specific method for dynamically adjusting the weight allocation of upper-layer nodes based on the historical performance of nodes is as follows: if a node has not experienced a failure in the past two hours and has low communication latency with other nodes, then its weight may be increased, and it will be given priority as a member of the arbitration set.

[0114] Step 7: Application of the EPaxos protocol based on batch submission in conflict-free and conflict scenarios. Improve the throughput and consensus success rate of the Paxos protocol-based distributed consensus system in cloud-edge-device environments by optimizing the proposal submission strategy.

[0115] In conflict-free scenarios, where multiple proposals are independent and do not conflict with each other (i.e., these proposals can be submitted and processed independently), a delayed batch submission strategy is used to merge multiple conflict-free proposals into a single joint operation. To improve throughput and reduce communication overhead, the decision to merge proposals is dynamically made based on the current state of the nodes (such as latency and load), ensuring that the merged proposals come from nodes with good network quality and low latency. Figure 4 As shown;

[0116] In this embodiment, it is assumed that in the absence of conflicts, multiple requests in the distributed consensus system will be merged into a batch for processing; for example, when the requests from 10 clients are not conflicting, all proposals will be submitted together, thereby reducing the number of communications between nodes and improving throughput.

[0117] In conflict scenarios, when handling conflicting proposals (i.e., when multiple proposals contend for resources or have dependencies), a prioritization strategy optimizes the processing order of proposals, reducing inter-property dependencies and thus improving consensus success rates. In this scenario, the weight of a node (based on its historical performance, network latency, load, etc.) influences the priority of proposal sorting; proposals from high-weight nodes are processed first to ensure that the most reliable node reaches consensus first in conflict scenarios, thereby improving the system's consensus efficiency and stability, and reducing communication overhead and latency. Figure 5 As shown;

[0118] In this embodiment, when processing conflicting proposals, the distributed consensus system will process them according to a priority strategy; for example, when two proposals conflict, the distributed consensus system will select the proposal with less conflict to submit first, thereby reducing the possibility of falling back to the fast path and improving the consensus success rate.

[0119] In the experimental environment deployment, docker-compose was used to simulate the complex environment of multiple nodes in the cloud-edge-device. In order to simulate network characteristics more realistically, the actual communication delay data in Table 1 was referenced during the simulation process, and the network traffic of the node containers was manually set using the network traffic control tool TC (Traffic Control).

[0120] Table 1. Communication latency between cloud server nodes deployed nationwide.

[0121] Beijing Hohhot Shanghai Hangzhou Hongkong Guangzhou Beijing 0.204 17.548 26.941 30.944 44.818 45.769 Hohhot 17.548 0.145 43.115 37.118 63.501 63.287 Shanghai 26.941 43.115 0.275 8.719 38.049 32.764 Hangzhou 30.944 37.118 8.719 0.251 30.594 32.432 Hongkong 44.818 63.501 38.049 30.594 0.192 11.551 Guangzhou 45.769 63.287 32.764 32.432 11.551 0.185

[0122] The nodes within the cluster are evenly distributed across various data centers (3 data centers for a 9-node cluster; 5 data centers for a 25-node cluster), ensuring that nodes within the same data center are on a local area network (LAN), while nodes across data centers communicate in a wide area network (WAN) mode. During network topology simulation, HTB (Hierarchical Token Bucket) queue rules are pre-set based on the source and destination addresses of data packets to achieve precise flow control. To simulate WAN communication during cloud-edge collaboration, the communication bandwidth between cloud-side and edge-side nodes is limited to 100MBps. To facilitate LAN communication during edge-end collaboration, the communication bandwidth between edge-side and end-side nodes is maintained at the default 10GBps.

[0123] To enrich the experimental results of this embodiment, the TPC-C dataset (Transaction Processing Performance Council Benchmark C) is additionally introduced here. This dataset is a widely recognized benchmark in the industry and is usually used to simulate the online transaction processing environment of the retail industry in the real world. The benchmark covers a series of common transaction types, including new orders, payment orders, and mixed scenarios (mixed = new order + payment), and allows adjustment of the proportion of cross-partition transactions to simulate different business scenarios.

[0124] For the TPC-C dataset, the graph shows that batch size is highly sensitive to factors such as the number of worker threads on a node, workload type, and data distribution. Figure 6 (a) illustrates the relationship between batch size and the number of worker threads. Under the default workload type, the graph shows a turning point in the improvement of cluster performance by batch size. Linear fitting can provide an optimal batch size in practice:

[0125] Batch size =125*Num threads +500 (10)

[0126] Figure 6 (b)- Figure 6 (d) discusses the transaction type, namely the impact of transaction operation type and cross-partition transactions on the batch size value; Figure 6 (b) The mixed transaction consists of 80% new order transactions and 20% payment order transactions. In the standard new order scenario, insert operations account for the majority; while the payment scenario involves calculating and modifying multiple data (inventory, balance, etc.). Compared to new orders, the business logic of payments is more complex. If both share the same batch... size Parameters often do not yield optimal performance; Figure 6 (c) and (d) further analyze the impact of cross-partition transactions on performance. The concurrency characteristics brought about by cross-partition transactions also have a significant impact on the choice of batch size. Combining the analysis results of these three subgraphs, when the default cluster can tolerate a latency of 10,000us, the optimal batch size calculation formula can be derived as follows:

[0127] Batch size =1250×x 2 -250×x+500 (x represents the percentage of new orders) (11)

[0128] The scalability analysis of this invention demonstrates the average latency performance of different algorithms in a 25-node configuration; the table below represents the range of system latency (in milliseconds), thus reflecting the stability and reliability of the protocol performance;

[0129] The experimental results measured using a multi-Paxos protocol layering method, which decomposes the overall latency into two parts: decision time and proposal replication latency. Decision time is mainly responsible for ensuring the security and liveness of the protocol, while proposal replication latency is related to the availability of the system. Due to the reduced number of nodes in the decision layer and the fact that it only involves the cloud and edge layers, the latency fluctuation range is reduced. Considering the characteristics of the wide area network environment, the decision layer adopts the EPaxos protocol with parameter α set to 0.5, which makes the performance comparable to that of 5-node EPaxos.

[0130] Table 2 Performance of the hybrid Paxos protocol in a cloud-edge-device environment

[0131] Number of nodes Paxos EPaxos Hybrid-Paxos-Hide Hybrid-Paxos 5 23-201 20-210 10-75 53-174 25 5-890 5-780 3-104 40-470

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A high-scalable consensus method based on Paxos protocol on a cloud edge environment, characterized in that: The method comprises the following steps: Step 1: under the cloud edge architecture, the node hierarchy of the distributed consensus system is divided into an upper layer and a lower layer; the upper layer is a decision layer in the cloud, adopts an EPaxos protocol, and is responsible for voting and confirming proposals of the lower layer nodes; the lower layer is a replication layer in the edge, adopts a Multi-Paxos protocol, and is responsible for replicating and executing the proposals of the upper layer nodes; Step 2: a weight distribution mechanism of the upper layer nodes is defined, is adaptive to a heterogeneous network environment, and adopts a dynamic weight distribution algorithm to calculate the weight of the upper layer nodes from three aspects of a signal-to-noise ratio, a number of lower-hanging child nodes of the nodes and an average communication delay of multiple clients; Step 3: a method for calculating an arbitration set of the EPaxos protocol is modified, a reasonable weight is distributed to each node, and therefore the distributed consensus system based on the Paxos protocol under the cloud edge environment can maintain high availability and high performance when facing a wired network environment and a wireless network environment and a load condition; Step 4: an optimization strategy based on delay batch submission is adopted to optimize a submission strategy of the EPaxos protocol; Step 5: an efficient proposal replication and leader switching mechanism based on the EPaxos protocol is adopted to guarantee the high availability and stability of the distributed consensus system based on the Paxos algorithm under the cloud edge environment; Step 6: according to real-time states and historical performances of the nodes, the weight distribution of the upper layer nodes is dynamically adjusted, and the distributed consensus system based on the Paxos algorithm under the cloud edge environment can adapt to changes of the network environment and node failures and maintain an efficient distributed consensus process; Step 7: application of the EPaxos protocol based on batch submission in a conflict-free and conflict scenario, and the throughput and consensus success rate of the distributed consensus system based on the Paxos protocol under the cloud edge environment are improved through optimization of a proposal submission strategy. 2.The highly scalable consensus method based on Paxos protocol in cloud edge-end environment according to claim 1, characterized in that: The signal-to-noise ratio described in step 2 For measuring the communication quality of the upper layer node in the wireless network environment, the specific formula is: （1）； （2）； The network quality evaluation index based on the signal-to-noise ratio in a wireless network environment, that is, the node importance weight based on the signal-to-noise ratio in a wireless network environment is shown in the following formula: （3）； wherein K is the number of other nodes related to the current node, excluding the current node, the communication capability of the upper layer node , the communication capability of the upper layer node , the signal transmitting capability of node k, the signal receiving capability of the upper layer node , the path loss from node k to the upper layer node , calculated by an empirical model, is a random variable and obeys a normal distribution with mean 0 and variance is an additive white Gaussian noise between node communications, dB is the decibel of the random variable , is the total number of nodes and encompasses all nodes, the signal-to-noise ratio of the upper layer node , the signal-to-noise ratio of node , is a network quality evaluation index of node under a wireless network environment; The number of lower-hanging child nodes of the nodes is used for reflecting the influence of the nodes as proposers in the lower layer, the importance weight of the nodes in the network is calculated based on the number of lower-hanging child nodes of the nodes, and a specific formula is as follows: （4）； wherein, is a reaction node importance weight in the network; is a node number of hanging child nodes of the node is a node number of hanging child nodes of the node The average communication delay of the multiple clients is used for measuring the speed of the nodes in responding to client requests, and a formula is as follows: （5）； wherein, is the average communication delay for the plurality of clients, C is the number of clients, is the communication delay between the cth client and the node , is the communication delay of the node . Then the weight of the nodes based on the communication delay in the consensus process is as follows: （6）； wherein, is a node weight based on communication delay in consensus process.

3. The highly scalable consensus method based on Paxos protocol in cloud edge-end environment according to claim 2, characterized in that: The specific method of step 3 is as follows: In the face of the wireless network environment and the wired network environment, finally, a node weight formula considering the signal-to-noise ratio, the number of lower-hanging child nodes of the nodes and the average communication delay of the multiple clients is as follows: （7）； wherein, is the final obtained comprehensive consideration of the ith node weight, and a is a node weight coefficient, used to weigh the relationship between high availability and request delay of the distributed consensus system based on the Paxos algorithm in the cloud-edge-end environment. Under the new weight distribution mechanism, the arbitration set of the EPaxos protocol no longer depends on the number of nodes, but is calculated based on the total weight of the nodes finally considered; The arbitration set formula of the EPaxos protocol is as follows: （8）； Wherein, Q is the arbitration set of the EPaxos protocol.

4. The highly scalable consensus method based on Paxos protocol in cloud edge-end environment according to claim 3, characterized in that: The specific process of step 4 is as follows: S1: setting a buffer of an acceptor node and a delay reply parameter: after receiving a proposal, the acceptor node does not process immediately, but stores in a local buffer and waits for a preset delay time; S2: Batch processing proposal: After the end of the delay time, the receiver node groups the proposals in the buffer according to the client number and data object, sorts them and then submits them in batches; S3: Priority processing of conflicting proposals: In the conflict scenario, the sorting priority strategy ensures that high-priority proposals can be submitted through the fast path of the EPaxos protocol, reducing communication overhead and delay.

5. The highly scalable consensus method based on Paxos protocol in cloud edge-end environment according to claim 4, characterized in that: The specific method of step 5 is: In the underlying Multi-Paxos protocol, heartbeat packets are used to keep the node alive, ensuring the activity of the current leader; if the leader is found to be invalid, the leader switching mechanism is triggered, thereby ensuring the high availability and stability of the distributed consensus system based on the Paxos algorithm in the cloud-edge environment.

6. The highly scalable consensus method based on Paxos protocol in cloud edge-end environment according to claim 5, characterized in that: The step 7 in the non-conflict scenario, by using the delay batch submission strategy, combines multiple non-conflict proposals into one joint operation; according to the current state of the node, it dynamically judges whether to merge the proposal, ensuring that the merged proposal comes from a node with good network quality and low delay.

7. The highly scalable consensus method based on Paxos protocol in cloud edge-end environment according to claim 6, characterized in that: In the conflict scenario, when processing conflicting proposals, the sorting priority strategy optimizes the processing order of the proposals, reduces the mutual dependence between the proposals, and thereby improves the consensus success rate; in this scenario, the weight of the node will affect the priority of the proposal sorting; the proposal of the high-weight node will be processed first to ensure that the most reliable node in the conflict scenario reaches consensus first, thereby improving the consensus efficiency and stability of the system, reducing communication overhead and delay.

Images