Distributed consensus protocol implementation method based on device priority

Abstract

The invention discloses a distributed consensus protocol implementation method based on equipment priority, which is characterized in that a deterministic main selection algorithm and a log copying method are adopted, a client request is processed by a main leader and then is distributed to nodes in a group through a grouping leader, and the nodes in the group feed back a result to the main leader after execution so as to judge log submission. The deterministic master selection algorithm ensures that a high-performance leader is selected through broadcasting node priority information; according to the log copying, after a main leader and a group leader are elected, the main leader sends a log to the group leader, then the group leader forwards the log to an intra-group node, and the intra-group node executes the log and feeds back a result to the main leader for log submission. Compared with the prior art, the method has the advantages that the method adapts to different network topologies and equipment performance, system performance, stability and data consistency are improved, high-performance equipment is reasonably utilized to accelerate data synchronization, and normal cooperative work of the equipment in a network can be maintained when the equipment is frequently online and offline.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed system data consistency, and in particular to a method for implementing a distributed consensus protocol based on device priority that is applicable to edge devices of a ubiquitous operating system. Background Art

[0002] With the all-round extension of the Internet to human society and the physical world, as well as the rise of new technologies such as cloud computing, big data, and artificial intelligence, various new application models and new application scenarios have emerged. The operating system-related technologies are facing many challenges and changes. The concept of operating system has begun to show a generalization trend. With the goal of shielding and managing heterogeneous resources downward and refining and precipitating application commonality upward, a series of new operating systems have been born. With the advent of the era of ubiquitous computing that integrates man, machine, and things, various types of ubiquitous IoT terminals and new ubiquitous computing models have emerged, and ubiquitous operating systems have come into being. Storage management is a core function of ubiquitous operating systems. How to design lightweight, high-performance, and dynamically adaptable storage systems for ubiquitous operating systems is a necessary measure to promote the development of ubiquitous operating systems. In the ubiquitous operating system environment, there are three levels of storage requirements: "end (terminal device) - edge (edge ​​end) - cloud (cloud end)". There is a large amount of storage demand for structured data at all three levels, and the end-side devices show characteristics such as resource extremes and heterogeneity. In ubiquitous operating systems, end-side devices usually have lower computing and communication capabilities. Therefore, they are unable to undertake heavy consensus tasks, so how to perform distributed data management on end-side devices is a major difficulty.

[0003] In operating systems that support device interconnection, distributed database services are the basic services at the bottom of the system and are also an indispensable part. Such operating systems have the natural characteristics of shielding communication protocols and conveniently and quickly completing data interaction, transmission and collaboration. Based on this characteristic, such systems can be widely used in scenarios with harsh working environments and difficult communication networks, such as mines, ocean-going ships, and power tower construction. By utilizing distributed data capabilities, it is possible to achieve interconnection and data collaboration between devices without relying on specific network communication methods. Operating systems that support device interconnection in these specific workplaces play a huge advantage, but with the unstable fluctuations in network conditions, devices are frequently online and offline. How to make the devices in the network work stably and efficiently has become an important problem to solve. The Raft algorithm is simple and has low requirements for computing power and communication capabilities, making it suitable for edge devices. However, the Raft protocol is random in selecting devices to become leaders, and it is impossible to select a more suitable leader for complex and changing states. Due to the diversity and changing states of edge devices, the randomly selected leader cannot fully exert the performance of the device. Therefore, it is necessary to design a new leader election mechanism.

[0004] In summary, in ubiquitous operating systems, especially edge devices, it is urgent to ensure stable and efficient collaboration of devices within the network. A lightweight distributed consensus protocol based on device optimization level is designed for edge device operating systems. It supports the device operating system on the edge and ensures the consistency of device data while introducing as little communication overhead as possible. It also makes reasonable use of higher-performance devices to accelerate data synchronization based on the performance and status of different devices. In addition, in scenarios where devices frequently go online and offline due to unstable network conditions, the device automatically reselects the group to maintain the normal collaboration of devices within the network. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed consensus protocol implementation method based on device priority in view of the deficiencies of the prior art. The method adopts a deterministic master election algorithm and a log replication method based on group leaders. By broadcasting node priority information, a high-performance leader is selected, and a group strategy is adopted to reduce complexity. After the main leader and the group leader are elected, the main leader sends the log to the group leader, and then the group leader forwards it to the nodes in the group. After the nodes in the group execute, the results are fed back to the main leader for log submission. At the same time, the number of groups and the priority broadcast frequency are optimized and adjusted according to the actual situation to adapt to different network topologies and device performance, improve system performance, stability and data consistency, and better solve the data consistency and collaborative work problems of edge devices in complex network environments. The lightweight and deterministic device priority distributed consensus protocol of this method is used for the edge device operating system. While reducing communication overhead, it ensures device data consistency, reasonably uses high-performance devices to accelerate data synchronization, and can maintain normal collaborative work of devices in the network when devices are frequently online and offline. It has good application prospects and commercial development value.

[0006] The specific technical scheme for implementing the present invention is: a method for implementing a distributed consensus protocol based on device priority, which is characterized by adopting a deterministic leader election algorithm, including broadcasting node priority information to all nodes and then voting to elect a leader, as well as a group leader election strategy, in which the group leader election first determines the leader within the group, and then the final master leader is determined by inter-group election; using a log replication scheme based on group leaders, client requests are processed by the master leader and then distributed to the nodes within the group through the group leader, and the nodes within the group feed back the results to the master leader after execution to determine the log submission.

[0007] The group leader election strategy of the deterministic leader election algorithm is to group the nodes relatively evenly during grouping. The elected nodes in the group are initially in a specific state. After a timeout, they change and broadcast information. After other nodes receive and broadcast their own information, they vote to elect the leader in the group according to specific rules. In the inter-group election, the leader in the group broadcasts information, integrates the information, sorts it according to logs and priorities, divides it, and traverses it. The final primary leader is determined by voting, and the specific log integrity conditions of the primary leader are guaranteed.

[0008] The log replication adopts a hierarchical synchronization scheme, that is, the log is first synchronized from the main leader to the group leader, and then synchronized from the group leader to the nodes within the group. The nodes within the group receive the log information from the group leader and perform log legitimacy checks and update operations. The execution process must follow strict data integrity and consistency rules to ensure that the operation is accurate. After the execution is completed, the nodes within the group will promptly feedback the execution results to the main leader. The main leader, based on the preset judgment logic and comprehensive consideration of the feedback results from multiple nodes in the group, accurately judges whether the log has reached a submittable state. If the conditions are met, the log will be correctly applied to the local state machine to ensure data accuracy and system consistency of the entire log replication process.

[0009] The group leader election strategy also includes optimizing the number of groups and the priority broadcast frequency: the number of groups is adjusted based on the total number of nodes and actual factors, and in theory there is a specific calculation method to achieve a better performance balance; the priority broadcast frequency is dynamically adjusted based on network and device conditions, and can be changed as needed after the initial setting; and consistent hashing can be used for dynamic changes in groups.

[0010] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress: 1) The deterministic leader election algorithm effectively solves the problem of random leader election in the traditional Raft algorithm. Through deterministic leader election based on device priority, the accuracy of leader selection is improved, communication overhead is reduced, and the system can better adapt to the complex environment of edge devices.

[0011] 2) The log replication scheme based on group leaders improves the parallelism of log replication, fully utilizes the resources of groupLeader, reduces the pressure on the main leader, thereby improving the read and write throughput of the system and accelerating the data synchronization process.

[0012] 3) The overall protocol is designed to be lightweight and suitable for resource-constrained embedded devices. It can ensure the system operation of heterogeneous devices under unstable networks, guarantee data consistency, improve the stability and reliability of the system, and enhance the collaborative working ability of edge devices in the ubiquitous operating system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1It is the node state transition diagram of Example 1; Figure 2 This is a flow chart of the log copy program of Example 1. DETAILED DESCRIPTION

[0014] The present invention specifically comprises the following steps: Step 1: Deterministic master election algorithm The priority information of each node is broadcast to other nodes, so that all nodes can vote after obtaining all the priority information, thereby ensuring that the node with the highest performance is selected as the leader and enhancing the certainty of the election. In order to reduce the complexity brought by information exchange, a grouping strategy is adopted. First, all nodes are grouped relatively evenly. During the election within the group, the initial state of the node is follower. After the timeout, it is transformed into groupCandidate, and the priority and log index information are broadcast to other nodes in the group. After receiving it, other nodes first broadcast their own information, and then select voting objects based on priority and log index information, giving priority to nodes with the latest logs and the highest priority. The node that obtains the majority of votes becomes the groupLeader. During the inter-group election, the groupLeader broadcasts the information again. After obtaining all node information from each other, it selects the node with the highest priority from the nodes with newer logs than most nodes according to log index and priority sorting, and finally determines the main leader.

[0015] Through theoretical analysis and simulation experiments, it is proved that this grouping optimization method can effectively reduce the amount of communication. For example, in the case of a large number of nodes, the amount of communication after grouping optimization is significantly reduced compared to the unoptimized state, and as the node scale changes, the advantage of communication volume is still obvious. At the same time, the theoretical analysis of the optimal solution for grouping is highly consistent with the simulation results, which further verifies the effectiveness of the grouping strategy.

[0016] Step 2: Log replication scheme based on group leaders After the main leader and multiple groupLeaders are elected, during the log replication phase, the client request reaches the main leader first. The main leader sends the log add rpc to the groupLeader, instead of sending it to each follower node as in the traditional way. After receiving it, the groupLeader sends the log to the followers in the group. This design reduces the network bandwidth pressure of the main leader and makes full use of the bandwidth resources of other nodes. After the groupFollower and followers have completed the log validity check and update operations, they send the execution results to the main leader to ensure the consistency and correctness of the data.

[0017] The present invention is further described in detail below in conjunction with specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art, and the present invention has no special restrictions. Example

[0018] See also Figure 1 , the group leader election strategy specifically includes: 1. In-group election 1-1: Group operation Divide all nodes relatively evenly into x logical groups. For example, for a cluster of n nodes, it can be divided into x groups, with approximately n / x nodes in each group, based on the actual network topology and device distribution.

[0019] 1-2: State transition and information broadcast: The initial state of the node is follower. After the timeout, it changes to groupCandidate and broadcasts an election request containing its own priority information and log index information to other nodes in the group. After receiving the request, other nodes immediately broadcast their own priority information and log index information to ensure fast synchronization of information within the group.

[0020] 1-3: Voting decision After receiving information from all nodes in the group, the node traverses the list of <log index information, priority information, node id>, selects the node with the highest priority, and casts the approval vote to the node with the highest priority. When a groupCandidate node receives y / 2+1 approval votes (y is the number of nodes in the group, y is approximately n / x), it becomes the groupLeader and completes the group election.

[0021] 2. Inter-group election 2-1: Information integration and broadcasting The groupLeader elected in the group has mastered the log index information and priority information of the nodes in the group. At this time, the groupLeader broadcasts this information to the groupLeaders of other groups, so that each groupLeader can obtain information about all nodes in the entire cluster.

[0022] 2-2: Sorting and selection After receiving information from other groupLeaders, the groupLeader sorts the list of <log index information, priority information, node id>, with log index as the first keyword (ensuring that the node with the latest log is ranked first) and priority information as the second keyword (in the case of the same log, the node with higher priority is given priority). Then, using the median information of the list as the boundary, traverse from the front to the middle of the list, select the node with the highest priority as the voting object, and cast the consent vote for the node. The leader finally elected must meet the conditions of being newer and having the highest priority than the logs of most nodes to ensure Leader Completeness (the leader has all logs owned by the majority of nodes). The elected leader is not necessarily the original groupLeader.

[0023] See also Figure 2 , log replication process optimization: 2-1: Traditional processes and problems In the traditional log replication process, after the client sends a request, the leader appends the request as a new log entry to its own log and sends a log add RPC to each server in parallel. When most followers accept the request, the entry is considered committed, and the leader applies the committed entry to the local state machine. During this process, the leader needs to send logs to each follower node. As the number of followers increases, the network bandwidth pressure of the leader increases dramatically, becoming a system performance bottleneck.

[0024] 2-2: Log replication optimization solution based on the leader within the group In the optimized log replication process, when a client request arrives, the leader only sends the log add rpc to the groupleader of each group. After receiving the log, the groupleader forwards the log to the followers in the group. For example, in a cluster divided into groups, the leader only needs to send logs to the groupleader instead of all follower nodes, which significantly reduces the network bandwidth pressure of the leader.

[0025] After the follower completes the log validity check and update operation, it sends the execution result to the leader instead of the groupLeader. The leader determines whether the log can be submitted based on the received execution result. If the leader receives the successful execution result from the majority of followers, the log is submitted and applied to the local state machine to complete the log replication process. This solution makes full use of the resources of the groupLeader, improves the efficiency of log replication, and enhances the overall performance of the system.

[0026] (III) Grouping parameter adjustment and optimization 3-1: Optimization of the number of groups According to theoretical analysis and actual tests, the selection of the number of groups has an important impact on system performance. Through simulation experiments on clusters of different sizes, it is found that when ∛(n^2 / 2) (n is the total number of nodes), the system communication overhead and time delay can achieve a relatively good balance. Therefore, the number of nodes in each group is adjusted to ∛(n^2 / 2) to obtain the best performance. However, in actual applications, it is also necessary to comprehensively consider factors such as network topology and device performance differences to appropriately adjust the number of groups.

[0027] 3-2: Priority broadcast frequency adjustment Considering that node priorities are relatively stable in the short term, the frequency of priority broadcasts can be reduced to reduce communication overhead. For example, the initial setting is to broadcast a priority once per election, and then dynamically adjust the broadcast frequency based on changes in the network environment and fluctuations in device performance. If it is found that node performance changes frequently, the broadcast frequency can be appropriately increased; if the network and device status are stable, the broadcast frequency can be further reduced to optimize system performance.

[0028] 3-3: Dynamic grouping based on consistent hashing In order to minimize the change in the mapping relationship between existing service requests and request processing servers when removing or adding a server, consistent hashing is used and the consistent hashing algorithm is transplanted into device grouping to solve the problem of uneven grouping after devices frequently join and leave for a long time.

[0029] Through the above specific implementation scheme, the present invention can effectively adapt to the complex environment of edge devices in ubiquitous operating systems, improve system performance, stability and data consistency, and ensure efficient collaboration between devices. In practical applications, relevant parameters can be flexibly adjusted according to specific scenario requirements and device characteristics to achieve the best operating effect.

[0030] The above specific implementations are only for further explanation of the present invention, and are not intended to limit the patent of the present invention. Any equivalent implementation of the present invention should be included in the scope of the claims of the patent of the present invention.

Claims

1. A method for implementing a distributed consensus protocol based on device priority, characterized in that: A deterministic leader election algorithm and a group leader-based log replication method are adopted. After the client request is processed by the main leader, it is distributed to the nodes in the group through the group leader. After the execution, the nodes in the group feed back the results to the main leader to determine the log submission. The deterministic leader election algorithm includes: broadcasting node priority information to all nodes and then voting to elect the leader and group leader election; the group leader election is to evenly group the nodes, first elect the group leader within the group, and then elect the final main leader between groups; the log replication adopts a hierarchical synchronization method, the log is first synchronized from the main leader to the group leader, and then synchronized from the group leader to the nodes in the group. The nodes in the group receive the log information from the group leader and perform log legitimacy checks and update operations. After the execution is completed, the nodes in the group feed back the execution results to the main leader. The main leader judges whether the log has reached a submittable state based on the preset judgment logic and comprehensively considers the feedback results from multiple nodes in the group. If the conditions are met, the log is applied to the local state machine.

2. According to the method for implementing a distributed consensus protocol based on device priority in claim 1, it is characterized in that: The election node in the group is initially in a specific state, and after a timeout, it changes and broadcasts information. After other nodes receive and broadcast their own information, they vote to elect a leader in the group according to specific rules; The inter-group election is carried out by the leader in the group broadcasting information, integrating the information, sorting it according to logs and priorities, dividing and traversing it, voting to determine the final primary leader, and ensuring the specific log integrity conditions of the primary leader.

3. According to the method for implementing a distributed consensus protocol based on device priority in claim 1, it is characterized in that: The voting and grouping of node priority information broadcast to all nodes to elect a leader includes: optimizing the number of groups and the frequency of priority broadcasts, and using consistent hashing for dynamic changes in groups. The number of groups is adjusted based on the total number of nodes and actual factors.

Images