Log synchronization methods, devices, electronic equipment and storage media

Abstract

Embodiments of the present application provide a log synchronization method and device, electronic equipment and storage medium, belonging to the technical field of data processing, the method is applied to a distributed system, the distributed system includes a plurality of nodes, the method comprises: determining a prophet node from the plurality of nodes, controlling the prophet node to send a query request to the nodes other than the prophet node, thereby controlling the prophet node to retain uncommitted logs meeting the effective condition according to the log information, determining the latest log item, and determining a candidate node from each node according to the latest log item, selecting a leader node from the candidate node according to a preset voting rule, and then controlling the leader node to perform a log synchronization operation on each node other than the leader node according to the latest log item, thereby ensuring that only uncommitted logs meeting the effective condition are retained in the uncommitted logs when the log synchronization is performed, so as to reduce the influence of the uncommitted logs and improve the data consistency effect.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a log synchronization method, apparatus, electronic device, and storage medium. Background Technology

[0002] The Raft algorithm is a consensus protocol for distributed systems. It maintains consistency of the replicated logs among nodes by electing a leader with supreme authority and assigning that leader the responsibility of managing the replicated logs. The leader receives log entries from clients, replicates them to other servers (followers), and, while ensuring safety, instructs other servers to apply the log entries to their state machines.

[0003] As long as the number of failed nodes in a distributed system does not exceed half of the total number of nodes, the Raft algorithm can guarantee the normal operation of the system and ensure that the log data of all nodes remains consistent. Current Raft optimization algorithms primarily focus on how to select a leader. While these methods can select a leader and include all committed logs, they are affected by uncommitted logs on each node, making it difficult to guarantee consistency with the actual operations of the client, resulting in poor data consistency (synchronization). Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a log synchronization method, apparatus, electronic device and storage medium that can improve the current Raft algorithm and Raft optimization algorithm, which suffer from poor data consistency due to the influence of uncommitted logs in each node, making it difficult to ensure that the actual operation of each node device in the distributed system is consistent with that of the client.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows.

[0006] In a first aspect, embodiments of the present invention provide a log synchronization method, which adopts the following technical solution.

[0007] A log synchronization method is applied to a distributed system, the distributed system comprising multiple nodes, the method comprising:

[0008] The prophet node is determined from the plurality of nodes;

[0009] The system controls the prophet node to send query requests to all nodes other than the prophet node, and receives log information returned by each node.

[0010] controlling the prophet node to retain uncommitted logs satisfying an effective condition according to the log information, to determine a latest log entry, and to determine a plurality of candidate nodes from the nodes according to the latest log entry;

[0011] electing a leader node from the candidate nodes according to a preset voting rule;

[0012] controlling the leader node to perform log synchronization operation on the nodes other than the leader node according to the latest log entry.

[0013] Further, the log information includes term, log entry, and whether the log is committed;

[0014] The step of controlling the prophet node to retain uncommitted logs satisfying an effective condition according to the log information, to determine a latest log entry, includes:

[0015] analyzing each log according to the log information in order from the latest to the earliest term;

[0016] if the current analyzed log is a committed log, obtaining a latest log entry according to the term and the log entry of the current analyzed log;

[0017] if the current analyzed log is an uncommitted log and satisfies the effective condition, retaining the current analyzed log and obtaining a latest log entry according to the term and the log entry of the current analyzed log;

[0018] if the current analyzed log is an uncommitted log and does not satisfy the effective condition, controlling the nodes to delete the current analyzed log, and continuing to analyze logs after the current analyzed log until a latest log entry is obtained.

[0019] Further, the effective condition includes that the number of nodes possessing the current analyzed log exceeds a preset number.

[0020] Further, the step of determining a prophet node from the plurality of nodes includes:

[0021] controlling each node to broadcast heartbeat information to nodes other than itself, the heartbeat information including a unique ID of the node;

[0022] controlling the nodes to compare their own IDs with the IDs in the heartbeat information, and returning a first response to the node sending the heartbeat information when the ID in the heartbeat information is greater than the own ID, and returning a second response to the node sending the heartbeat information when the ID in the heartbeat information is less than the own ID;

[0023] determining a node receiving only the second response as the prophet node.

[0024] Further, the step of determining the candidate nodes from the nodes according to the latest log entry includes:

[0025] controlling the prophet node to send the latest log entry to the node that owns the latest log entry, and the node that receives the latest log entry becomes a candidate node.

[0026] Further, the step of controlling the leader node to perform log synchronization operation on each of the nodes other than the leader node according to the latest log entry includes:

[0027] controlling the leader node to delete the log information after the latest log entry, and generate the latest log information;

[0028] controlling the leader node to send the latest log information to each of the nodes other than the leader node, so that each of the nodes other than the leader node synchronizes its own log information to the latest log information.

[0029] Further, the step of electing the leader node from the candidate nodes according to a preset voting rule includes:

[0030] controlling each of the candidate nodes to send a voting request to each of the nodes other than the candidate node;

[0031] controlling each of the nodes other than the candidate node to vote for the candidate node that meets its own preset condition after receiving the voting request;

[0032] determining the leader node from the candidate nodes according to the number of votes obtained by each candidate node.

[0033] In a second aspect, an embodiment of the present application provides a log synchronization device, which adopts the following technical solution.

[0034] A log synchronization device applied to a distributed system, the distributed system including a plurality of nodes, the log synchronization device including a query module, an election module and a synchronization module;

[0035] The query module is configured to determine a prophet node from the plurality of nodes, and control the prophet node to send a query request to each of the nodes other than the prophet node, and receive log information returned by each node;

[0036] The election module is configured to control the prophet node to retain an uncommitted log that meets a valid condition according to the log information, determine a latest log entry, and determine a plurality of candidate nodes from the nodes according to the latest log entry.

[0037] The election module is further configured to elect a leader node from the candidate nodes according to a preset voting rule.

[0038] The synchronization module is configured to control the leader node to perform a log synchronization operation on the nodes other than the leader node according to the latest log entry.

[0039] In a third aspect, an electronic device is provided, which employs the technical solutions as follows.

[0040] An electronic device includes a processor and a memory, the memory storing machine executable instructions capable of being executed by the processor, and the processor being capable of executing the machine executable instructions to implement the log synchronization method as described in the first aspect.

[0041] In a fourth aspect, an electronic device is provided, which employs the technical solutions as follows.

[0042] A storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the log synchronization method as described in the first aspect.

[0043] The log synchronization method, device, electronic device and storage medium provided by the embodiments of the present application can select a prophet node from a plurality of nodes of a distributed system, the prophet node obtains log information of each node other than the prophet node, controls the prophet node to retain an uncommitted log satisfying an effective condition according to all the log information, determines a latest log entry, and determines a candidate node according to the latest log entry, and then selects a leader node from the candidate node, so that the leader node performs a log synchronization operation on each node other than the leader node, thereby retaining only the uncommitted log satisfying the effective condition in the uncommitted log when performing the log synchronization, so as to eliminate the influence of the uncommitted log to a certain extent, and thereby improve the data consistency effect.

[0044] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following will describe preferred embodiments in detail with reference to the attached drawings. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1A block diagram of a distributed system is shown.

[0047] Figure 2 A flowchart of a log synchronization method is shown.

[0048] Figure 3 A block diagram of an electronic device is shown. Figure 2 A flowchart of part of the sub-steps of step S101 is shown.

[0049] Figure 4 A flowchart of part of the sub-steps of step S105 is shown. Figure 2 A flowchart of part of the sub-steps of step S107 is shown.

[0050] Figure 5 A flowchart of part of the sub-steps of step S109 is shown. Figure 2 A flowchart of part of the sub-steps of step S109 is shown.

[0051] Figure 6 A block diagram of a log synchronization device is shown. Figure 2 A block diagram of a log synchronization device is shown.

[0052] Figure 7 A block diagram of a distributed system is shown.

[0053] Figure 8 A block diagram of a log synchronization device is shown.

[0054] Figure: 100-distributed system; 110-client; 120-node; 130-electronic device; 140-memory; 150-processor; 160-communication module; 170-log synchronization device; 180-query module; 190-election module; 200-synchronization module. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0056] Therefore, the detailed description of the embodiments of the present application provided in the accompanying drawings below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] It should be noted that the relational terms herein, such as first and second and the like, are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0058] The Raft algorithm is a consensus protocol for distributed systems proposed by Diego Ongaro and John Ousterhout of Stanford University in the paper "In Search of an Understandable Consensus Algorithm" published in 2013. The algorithm maintains the consistency of the replication log among nodes by electing a powerful leader from the nodes of the distributed system and assigning the leader the task of managing the replication log.

[0059] The leader receives log entries from the client, and the log entries are replicated to other servers (the distributed system includes multiple servers, and the server is a node). Under the premise of ensuring security, the leader tells other servers to apply the log entries to their state machines.

[0060] The Raft algorithm can ensure the normal operation of the distributed system and keep the log data of all nodes consistent when the number of down nodes does not exceed half of the total number of nodes. In the current optimization methods of the Raft algorithm, some methods use load strategy to select the node with the lowest resource load as the leader, and some methods use auxiliary nodes to increase the number of nodes participating in the election to improve the reliability of the election process.

[0061] These Raft algorithms and Raft algorithm optimization methods can ensure that the leader node contains all committed logs, but for uncommitted logs, they can only be discarded or retained probabilistically. Therefore, these algorithms are affected by uncommitted logs in each node, making it difficult to ensure consistency with the actual operation of the client, resulting in poor data consistency (synchronization) effect.

[0062] Based on the above considerations, the embodiment of the present application provides a log synchronization scheme which can improve the current Raft algorithm and Raft optimization algorithm. Due to the influence of uncommitted logs in each node, it is difficult to ensure that each node device of the distributed system and the actual operation of the client are consistent, resulting in poor data consistency effect. In the following, the log synchronization method will be introduced from the perspective of the log synchronization method.

[0063] Please refer to Figure 1 , a block diagram of a distributed system 100, the distributed system 100 includes a plurality of nodes 120 and a plurality of clients 110, the client 110 and the node 120 are in communication connection, and the node 120 can be a server in essence. Each node 120 of the distributed system 100 can be deployed with a transaction system to realize the financial payment type business.

[0064] The financial payment type business has strict requirements for data strong consistency and high reliability. For example, if user A pays X yuan to user B, the account balance of user A should be displayed as reduced by X yuan, and the account balance of user B should be displayed as increased by X yuan. At this time, each node 120 of the distributed system 100 deployed with the transaction system about the accounts of user A and user B should synchronize the data (the account balance of user A is displayed as reduced by X yuan, and the account balance of user B is displayed as increased by X yuan).

[0065] When the client 110 performs a payment transaction, it will generate a log about the transaction and send it to the leader node 120 of the distributed system 100 (i.e. the leader node 120 to be elected in the log synchronization method provided in the embodiment). The leader node 120 will apply the log to the state machine and control other nodes 120 to copy and apply the log.

[0066] For a single node 120, the log becomes a committed log after the node 120 applies the log to the state machine (commit log), and the log is an uncommitted log before the log is applied to the state machine (uncommit log). Therefore, the node 120 includes committed logs and uncommitted logs.

[0067] In the embodiment, for the entire distributed system 100, the committed log indicates that all nodes 120 that own the log have committed the log, i.e. applied to the state machine, and the uncommitted log indicates that at least one node 120 among all nodes 120 that own the log has not committed the log.

[0068] The log synchronization method provided in the embodiment of the present application is applied to Figure 1The log synchronization method is applied to the distributed system 100 in the application, the prophet node 120 is determined, the prophet node 120 is controlled to obtain log information of each node 120 except the prophet node 120, and then the prophet node 120 is controlled to retain uncommitted logs meeting an effective condition in each node 120 according to the log information, determine a latest log entry, determine a plurality of candidate nodes 120 according to the latest log entry, select the leader node 120 from the plurality of candidate nodes 120, and then control the leader node 120 to perform a log synchronization operation on the nodes 120 except the leader node 120 according to the latest log entry.

[0069] It should be noted that the log synchronization method provided by the embodiment of the application can also be understood as an optimization method of the raft algorithm.

[0070] In one embodiment, as shown in Figure 2 , a log synchronization method is provided. The embodiment mainly takes the method applied to the distributed system 100 in the application as an example for illustration. Figure 1

[0071] S101, a prophet node is determined from a plurality of nodes.

[0072] S103, a prophet node is controlled to send a query request to each node except the prophet node and receive log information returned by each node.

[0073] The prophet node 120 can send the query request in a broadcast manner, and each node 120 except the prophet node 120 returns the log information to the prophet node 120 after receiving the query request. The log information can be log information of each node in a latest preset time period, that is, log information of logs in a latest time period, or log information of all logs.

[0074] The latest preset time period refers to a time period from a certain time point to a current time point. For example, the current time point is 4 o'clock, the latest preset time period is 1 hour, and the log information is log information from 3 o'clock to 4 o'clock.

[0075] S105, the prophet node is controlled to retain uncommitted logs meeting an effective condition according to the log information, determine a latest log entry, and determine a plurality of candidate nodes from the nodes according to the latest log entry.

[0076] The plurality refers to two or more than two. However, in an extreme case, there can be only one candidate node.

[0077] ​It should be understood that the control prophet node 120 retains the uncommitted log satisfying the valid condition according to the log information of each node 120 other than the prophet node and itself, that is, the log information of all nodes 120 of the distributed system 100, determines the latest log item, and determines the candidate node 120 from each node 120 according to the latest log item.

[0078] It should be noted that retaining the uncommitted log satisfying the valid condition also represents deleting the uncommitted log not satisfying the valid condition. The prophet node 120 obtains the log information of each node 120 other than the prophet node and the log information of itself.

[0079] S107, according to the preset voting rule, the leader node is elected from each candidate node.

[0080] S109, the control leader node performs log synchronization operation on each node other than the leader node according to the latest log item.

[0081] Through the above log synchronization method, by selecting the prophet node 120 from the multiple nodes 120 of the distributed system 100, the prophet node 120 obtains the log information of each node 120 other than the prophet node, controls the prophet node 120 to retain the uncommitted log satisfying the valid condition according to all the log information, determines the latest log item, and determines the candidate node 120 according to the latest log item, and then selects the leader node 120 from the candidate node 120, so that the leader node 120 performs log synchronization operation on each node 120 other than the leader node, thereby realizing that only the uncommitted log satisfying the valid condition is retained in all the uncommitted logs when performing log synchronization, so as to be able to eliminate the influence of the uncommitted log to a certain extent, and then the data consistency effect can be improved.

[0082] It should be noted that when the distributed system 100 is just initialized, there is no log information, at this time, any raft algorithm can be used to determine the leader node 120. After the log is generated, the log synchronization method provided by the embodiment of the application is used to determine the leader node 120.

[0083] For example, when the distributed system 100 is just initialized, since the log of each node 120 is empty, each node 120 can become a candidate node 120 (candidate), the number of candidate nodes 120 is not less than two, and the node 120 other than the candidate node 120 is a follower node 120 (follower). The candidate node 120 initiates voting, and the candidate node 120 with the most votes becomes the initial leader node 120 (leader), at this time, each node 120 other than the candidate node is a follower node 120.

[0084] Each node 120 in the distributed system 100 is assigned a unique ID. In one implementation, the process can be as follows: a node 120 is randomly selected as the initialization node 120, and each node 120 other than the initialization node requests a node 120 from the initialization node 120; the initialization node 120 assigns a node 120 to each node 120 according to the order in which the nodes 120 request the node 120; each node 120 records its own ID.

[0085] For example, distributed system 100 has 7 nodes 120. The initial node 120 has its own ID of 1 by default. According to the order of ID application of the nodes 120 other than the initial node, IDs 2-7 are assigned respectively.

[0086] The method for determining the prophet node 120 from multiple nodes 120 can be flexibly chosen; for example, it can be directly specified, or it can be selected according to set rules. In one implementation, refer to... Figure 3 The above is a flowchart of some sub-steps of step S101. The following steps are used to determine the prophet node 120 from multiple nodes 120.

[0087] S101-1 controls each node to broadcast heartbeat information to nodes other than itself.

[0088] The heartbeat information includes the node's unique ID.

[0089] S101-2 controls each node to compare its own ID with the ID in the heartbeat information.

[0090] S101-3, when the ID in the heartbeat information is greater than its own ID, return the first response to the node that sent the heartbeat information; when the ID in the heartbeat information is less than its own ID, return the second response to the node that sent the heartbeat information.

[0091] The first response can be a false response, and the second response can be a true response.

[0092] S101-4, the node whose received responses are all second responses is identified as the prophet node.

[0093] By following the steps S101-1 to S101-4 above, the node with the smallest ID, 120, can be identified as the prophet node 120.

[0094] In other implementations, S101-1 to S101-4 can be adaptively adjusted to determine the node 120 with the largest ID as the prophet node 120.

[0095] In one implementation, the log synchronization method provided in this embodiment can be used to re-select the leader node 120 at regular intervals. The interval can be referred to as the term of office.

[0096] Alternatively, after leader node 120 fails, the log synchronization method provided in this embodiment can be used to re-select leader node 120. The time periods of different leader nodes 120 can also be referred to as terms of office.

[0097] In one implementation, each node 120 of the distributed system 100 broadcasts heartbeat information to other nodes 120 at regular intervals. If a node 120 does not receive the heartbeat information broadcast by the leader node 120 within a preset time, it is determined that the leader node 120 has crashed, triggering an election to reselect a leader node 120.

[0098] Log information can include the term of office, log entries, and whether the log has been committed. For example, the log information could be (1-2C), where "1" represents the term of office, "2" represents the entry, and "C" represents a commit. Similarly, (1-3U) indicates that the third log entry in the first term is an uncommitted log entry.

[0099] On each node 120, logs are committed in the order they are logged, meaning the logs are applied to the state machine one by one.

[0100] Based on the above, in one implementation method, refer to Figure 4 The following is a flowchart of some sub-steps of step S105 above. The following steps are used to control the prophet node 120 to retain uncommitted logs that meet the valid conditions based on log information and determine the latest log entry.

[0101] S105-1, Analyze each log based on the log information in the order from most recent to latest term.

[0102] S105-2 If the current analysis log is a committed log, then the latest log entry is obtained based on the term and log entry of the current analysis log.

[0103] The latest log entry can be: last_index = term + index. Here, last_index represents the latest log entry, term represents the term, and index represents the entry.

[0104] It should be noted that log analysis ends once the latest log entry is obtained. That is, if the first log entry analyzed is a committed log, the latest log entry is obtained based on the log information of that log, and the log analysis ends there; there is no need to analyze the log entries before that entry.

[0105] S105-3 If the current analysis log is an uncommitted log and meets the valid conditions, retain the current analysis log and obtain the latest log entry based on the term and log entries of the current analysis log.

[0106] S105-4 If the current analysis log is an uncommitted log and does not meet the valid conditions, control each node to delete the current analysis log and continue to analyze the logs after the current analysis log until the latest log entry is obtained.

[0107] It should be noted that in steps S105-1 to S105-4, "currently analyzed log is a committed log" means that all nodes 120 have this log information, and the log information of this log on each node 120 indicates that it has been committed. Furthermore, uncommitted log entries prior to the latest log entry are also retained.

[0108] The valid conditions in S105 and S105-1 to S105-2 can be flexibly set; for example, they can be a setting rule or a determination method. In one implementation, the valid condition may include the number of nodes 120 that have the current analysis log exceeding a preset number.

[0109] For example, in a distributed system 100, there are 7 nodes 120 with IDs 1 to 7. The node 120 with ID 1 is the oracle node 120. The oracle node 120 obtains the latest log information from each node 120 other than the oracle node within a preset time period. The final log information is shown below.

[0110] ID=1, log=...(3-4c)(4-1c)(4-2c)(5-1u)

[0111] ID=2, log=...(3-4c)(4-1c)(4-2c)(5-1c)(5-2u)

[0112] ID=3, system crash

[0113] ID=4, log=...(3-4c)(4-1c)(4-2c)(5-1u)

[0114] ID=5, log=...(3-4c)(4-1c)(4-2c)(5-1u)

[0115] ID=6, log=...(3-4c)(4-1c)(4-2u)

[0116] ID=7, log=...(3-4c)(4-1c)(4-2u)

[0117] Logs with the same term and entry are considered the same log entry. Furthermore, the valid condition can be set to have more than three nodes 120 possessing the current analysis log. Node 120 with ID 3 can be a former leader node 120 that has failed.

[0118] The control oracle node 120 performs log analysis based on the above log information. Since log (5-2) is an uncommitted log from node 120, and no more than three nodes 120 possess this log, log (5-2) is deleted. Next, log (5-1) is analyzed. Nodes 120 with IDs 1, 2, 4, and 5 possess log (5-1). Since log (5-1) is uncommitted in nodes 120 with IDs 1, 4, and 5, log (5-1) is considered an uncommitted log. However, since four nodes 120 possess log (5-1), the valid condition is met, and log (5-1) is retained. Therefore, the latest log entry is obtained based on the log information in log (5-1), and the log analysis ends.

[0119] Further, please continue to refer to Figure 4 In one embodiment, step S105 further includes S105-5, through which multiple candidate nodes 120 are determined from each node 120 based on the latest log entry.

[0120] S105-5, the control oracle node sends the latest log entry to the node that has the latest log entry, and the node that receives the latest log entry becomes a candidate node.

[0121] It may also include controlling the prophet node 120 to send a "No Candidate" message to nodes 120 that do not have the latest log entry. These nodes 120 that do not have the latest log entry become the crowd nodes 120.

[0122] For example, after determining that the latest log entry is (5-1), the prophet node 120 sends (5-1) to nodes 120 with IDs 1, 2, 4, and 5. Nodes 120 with IDs 1, 2, 4, and 5 respond to the latest log entry and become candidate nodes 120. Nodes 120 with IDs 3, 6, and 7 become crowd nodes 120.

[0123] Furthermore, in one implementation, refer to Figure 5 This is a flowchart illustrating the sub-steps of step S107 above. The following steps are used to elect a leader node 120 from candidate nodes 120 according to preset voting rules.

[0124] S107-1 controls each candidate node to send voting requests to nodes other than the candidate node.

[0125] S107-2 controls all nodes other than the candidate nodes to vote for candidate nodes that meet their preset conditions after receiving a voting request.

[0126] The preset conditions can be flexibly set; for example, they can be set to the earliest time to send a vote request or the lowest resource load.

[0127] S107-3, determine the leader node from the candidate nodes based on the number of votes each candidate node receives.

[0128] The candidate node 120 that receives the most votes becomes the leader node 120. Alternatively, the candidate node 120 that receives half or more of the votes becomes the leader node 120.

[0129] In one implementation, if there is only one candidate node 120, it can become the leader node 120 if it receives more than half of the votes; otherwise, it can directly become the leader node 120 in a re-election.

[0130] In one implementation, reference is made to Figure 6 The above is a flowchart of some sub-steps of step S109. The following steps can be used to control the leader node 120 to perform log synchronization operations on all nodes 120 other than the leader node 120 according to the latest log entries.

[0131] S109-1, Control the leader node to delete the latest log entry and generate the latest log information.

[0132] Generating the latest log information includes updating uncommitted log entries before the latest log entry to committed log entries.

[0133] S109-2 controls the leader node to send the latest log information to all nodes other than the leader node, so that all nodes other than the leader node can synchronize their own log information to the latest log information.

[0134] After receiving the latest log information, each node 120 other than the leader node processes the logs to make its own logs consistent with the latest log information, thereby achieving log synchronization.

[0135] For example, if the latest log entry is (5-1), then after log synchronization, the log information of each node 120 will be as follows.

[0136] ID=1, log=...(3-4c)(4-1c)(4-2c)(5-1c)

[0137] ID=2, log=...(3-4c)(4-1c)(4-2c)(5-1c)

[0138] ID=3, system crash

[0139] ID=4, log=...(3-4c)(4-1c)(4-2c)(5-1c)

[0140] ID=5, log=...(3-4c)(4-1c)(4-2c)(5-1c)

[0141] ID=6, log=...(3-4c)(4-1c)(4-2c)(5-1c)

[0142] ID=7, log=...(3-4c)(4-1c)(4-2c)(5-1c)

[0143] The log synchronization method provided in this embodiment of the invention determines a prophet node 120, controls the prophet node 120 to obtain log information from all nodes 120 other than the prophet node, analyzes its own log information and that of all nodes 120 other than the prophet node, deletes uncommitted logs that do not meet the valid conditions, and obtains the latest log entries. The prophet node 120 then determines candidate nodes 120 based on the latest log entries, and elects a leader node 120 from among the candidate nodes 120. The leader node 120 deletes its own logs after its latest log entry and generates the latest log information, thereby controlling all nodes 120 other than the leader node to process logs based on the latest log information, achieving log synchronization.

[0144] This ensures that during log synchronization, only uncommitted logs that meet the valid conditions are retained in the uncommitted logs, which can eliminate the impact of uncommitted logs to a certain extent and thus improve data consistency.

[0145] It should be understood that, although Figures 2-6 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-6 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0146] Please refer to Figure 7This is a block diagram of electronic device 130. Electronic device 130 includes a memory 140, a processor 150, and a communication module 160. The memory 140, processor 150, and communication module 160 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0147] The memory 140 is used to store programs or data. The memory 140 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0148] The processor 150 is used to read / write data, computer programs, or machine-executable instructions stored in the memory 140 and perform corresponding functions. In one embodiment, the processor 150 reads the computer program or machine-executable instructions from the memory 140 to implement the log synchronization method provided in this embodiment of the invention.

[0149] The communication module 160 is used to establish a communication connection between the server and other communication terminals via the network, and to send and receive data via the network.

[0150] It should be understood that, Figure 7 The structure shown is only a schematic diagram of the electronic device 130. The electronic device 130 may also include components that are larger than... Figure 7 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof.

[0151] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of the log synchronization device 170 is given below. Optionally, the log synchronization device 170 can adopt the above-described... Figure 7 The device structure of the electronic device 130 shown. Further, please refer to... Figure 8 , Figure 8This is a functional block diagram of a log synchronization device 170 provided in an embodiment of the present invention. It should be noted that the log synchronization device 170 provided in this embodiment has the same basic principle and technical effects as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. This log synchronization device 170 is applied to a distributed system 100, which has multiple nodes 120. The log synchronization device 170 includes a query module 180, an election module 190, and a synchronization module 200.

[0152] The query module 180 is used to determine the prophet node 120 from multiple nodes 120, and to control the prophet node 120 to send query requests to each node 120 other than the prophet node, and to receive log information returned by each node 120.

[0153] The election module 190 is used to control the oracle node 120 to retain uncommitted logs that meet the valid conditions based on log information, determine the latest log entry, and determine multiple candidate nodes 120 from each node 120 based on the latest log entry.

[0154] The election module 190 is also used to elect a leader node 120 from among the candidate nodes 120 according to preset voting rules.

[0155] The synchronization module 200 is used to control the leader node 120 to perform log synchronization operations on nodes 120 other than the leader node based on the latest log entries.

[0156] Optionally, the above modules can be stored in the form of software or firmware. Figure 7 The memory 140 shown is either stored in or embedded in the operating system (OS) of the electronic device 130, and can be used by... Figure 7 The processor 150 executes the program. Meanwhile, the data and program code required to execute the above modules can be stored in the memory 140.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0158] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0159] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of log synchronization, the method comprising: The method is applied to a distributed system comprising a plurality of nodes, and comprises the following steps: determining a prophet node from the plurality of nodes; controlling the prophet node to send a query request to each of the nodes other than the prophet node, and receiving log information returned by each of the nodes; controlling the prophet node to retain uncommitted logs satisfying an effective condition according to the log information of each of the nodes other than the prophet node and the log information of the prophet node, determine a latest log entry, and determine a plurality of candidate nodes from the plurality of nodes according to the latest log entry; electing a leader node from the plurality of candidate nodes according to a preset voting rule; controlling the leader node to delete log information after the latest log entry, and generate latest log information, wherein generating the latest log information comprises retaining all committed logs of all nodes in the distributed system, and updating uncommitted logs before the latest log entry to committed logs; controlling the leader node to send the latest log information to each of the nodes other than the leader node, so that each of the nodes other than the leader node synchronizes the log information thereof to the latest log information; the log information comprises whether a log is committed, a term, and a log entry, the effective condition comprises that a number of nodes having a currently analyzed log exceeds a preset number, and the step of controlling the prophet node to retain uncommitted logs satisfying the effective condition according to the log information of each of the nodes other than the prophet node and the log information of the prophet node, and determine a latest log entry, comprises: analyzing each log according to the log information in order from a latest term to an earliest term; if a currently analyzed log is a committed log, obtaining a latest log entry according to the term and the log entry of the currently analyzed log; if the currently analyzed log is an uncommitted log and the effective condition is satisfied, retaining the currently analyzed log, and obtaining a latest log entry according to the term and the log entry of the currently analyzed log; if the currently analyzed log is an uncommitted log and the effective condition is not satisfied, controlling each of the nodes to delete the currently analyzed log, and continuing to analyze a log after the currently analyzed log until a latest log entry is obtained.

2. The log synchronization method of claim 1, wherein, the step of determining a prophet node from the plurality of nodes comprises: controlling each of the nodes to broadcast heartbeat information to each of the nodes other than the node itself, the heartbeat information comprising a unique ID of the node; controlling each of the nodes to compare the ID of the node itself with the ID in the heartbeat information, and returning a first response to a node sending the heartbeat information when the ID in the heartbeat information is greater than the ID of the node itself, and returning a second response to the node sending the heartbeat information when the ID in the heartbeat information is less than the ID of the node itself; determining a node receiving only the second response as the prophet node.

3. The log synchronization method of claim 1, wherein, the step of determining a candidate node from each of the nodes according to the latest log entry comprises: controlling the prophet node to send the latest log entry to a node having the latest log entry, and controlling the node receiving the latest log entry to become a candidate node.

4. The log synchronization method of claim 1, wherein, The step of electing a leader node from the candidate nodes according to a preset voting rule comprises: controlling each of the candidate nodes to send a voting request to each of the nodes other than the candidate nodes; controlling each of the nodes other than the candidate nodes to vote for a candidate node meeting a preset condition of the node after receiving the voting request; determining a leader node from the candidate nodes according to the number of votes obtained by each candidate node.

5. A log synchronization apparatus characterized by comprising: The log synchronization device is applied to a distributed system comprising a plurality of nodes, and is configured to perform the log synchronization method according to any one of claims 1-4. The query module is configured to determine a prophet node from the plurality of nodes, and to control the prophet node to send a query request to each of the nodes other than the prophet node and receive log information returned by each node. The election module is configured to control the prophet node to retain uncommitted logs meeting a valid condition according to log information of each of the nodes other than the prophet node and log information of the prophet node, to determine a latest log entry, and to determine a plurality of candidate nodes from the plurality of nodes according to the latest log entry, wherein the log information comprises whether a log is committed. The election module is further configured to elect a leader node from the candidate nodes according to a preset voting rule. The synchronization module is configured to control the leader node to perform a log synchronization operation on each of the nodes other than the leader node according to the latest log entry. The synchronization module is further configured to: control the leader node to delete log information after the latest log entry and to generate latest log information, wherein generating the latest log information comprises retaining all committed logs of all nodes in the distributed system and updating uncommitted logs before the latest log entry to committed logs; control the leader node to send the latest log information to each of the nodes other than the leader node, so that each of the nodes other than the leader node synchronizes log information of the node to the latest log information.

6. An electronic device, comprising: The computer program is executed by a processor to implement the log synchronization method according to any one of claims 1-4.

7. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to implement the log synchronization method according to any one of claims 1-4.

Images