Data synchronization method and device, equipment and medium

Abstract

The invention discloses a data synchronization method, device and equipment and a medium, the method is applied to any target service node in a service cluster, and the method comprises the following steps: when the target service node is used as a candidate node, sending a main node election request to each survival service node in the service cluster, when election success responses fed back by all the survival service nodes are received, the candidate node is converted into the main node; when the target service node is used as a main node, sending a data acquisition request to each survival service node in the service cluster to acquire a consensus information version number fed back by each survival service node; and determining a first service node to which the latest consensus information belongs according to each consensus information version number, acquiring snapshot data of the latest consensus information from the first service node, and synchronizing the snapshot data to all survival service nodes in the service cluster. According to the scheme, the consistency of the consensus information in the service cluster can be accurately ensured on the premise of effectively ensuring the time delay performance of the system.

Description

Technical Field

[0001] The present application relates to the technical field of cluster systems, and in particular to a data synchronization method, and also to a data synchronization device, an electronic device, and a computer-readable storage medium. Background Art

[0002] A server cluster is built up of multiple servers, each of which is a node in the physical sense, and these nodes provide a certain service to the outside world. In order to coordinate these nodes to provide services to the outside world normally, these nodes will share some information, such as the cluster member list, user information, the mapping relationship between data partitions and cluster nodes, etc. This information is called the cluster's consensus information. The cluster needs to maintain the consistency and correctness of the consensus information in order to provide services to the outside world normally. Therefore, how to update the consensus information so that the reading and writing of the consensus information does not become a bottleneck for the entire system and ensure the consistency of the consensus information is the key point of the server cluster design.

[0003] At present, the methods for maintaining the consistency of consensus information in the industry are mainly divided into two categories: centralized and decentralized. Centralized means storing consensus information on the middleware and using the middleware to maintain consensus information. Decentralized means storing consensus information on each node in the cluster and updating and maintaining consensus information through communication between cluster nodes. However, under the centralized management solution, the acquisition of consensus information by cluster nodes usually becomes a bottleneck for the entire system, because each cluster node needs to send a request to the middleware to obtain consensus information in order to implement the corresponding service business, which will cause a significant increase in business time, that is, increase system latency. Under the decentralized management solution, although the acquisition of consensus information will not become a system bottleneck, it is easy to have problems with inconsistent consensus information.

[0004] Therefore, how to accurately ensure the consistency of consensus information within the service cluster while effectively ensuring the system latency performance is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0005] The purpose of the present application is to provide a data synchronization method, which can accurately ensure the consistency of consensus information within a service cluster under the premise of effectively ensuring the system latency performance; another purpose of the present application is to provide a data synchronization device, an electronic device and a computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0006] In a first aspect, the present application discloses a data synchronization method, which is applied to any target service node in a service cluster, and the method comprises:

[0007] When the target service node is used as a candidate node, it sends a master node election request to each surviving service node in the service cluster, and upon receiving election success responses fed back by all the surviving service nodes, the candidate node is converted into a master node;

[0008] When the target service node serves as the master node, a data acquisition request is sent to each of the survival service nodes in the service cluster to obtain the consensus information version number fed back by each of the survival service nodes;

[0009] Determine the first service node to which the latest consensus information belongs according to each consensus information version number, obtain snapshot data of the latest consensus information from the first service node, and synchronize the snapshot data to all the surviving service nodes in the service cluster.

[0010] Optionally, the data synchronization method further includes:

[0011] When the election success response fed back by all the surviving service nodes is not received, the candidate node is converted into a slave node;

[0012] When the target service node serves as the slave node, it starts the heartbeat timing monitoring for the master node, and when the heartbeat signal of the master node is not detected, the slave node is converted into the candidate node.

[0013] Optionally, the data synchronization method further includes:

[0014] When the target service node serves as the master node, periodically sending the heartbeat signal to each of the surviving service nodes in the service cluster;

[0015] When an abnormal heartbeat response including master node information is received, determining a second service node to which the abnormal heartbeat response belongs, and performing data communication with the second service node to determine a real master node;

[0016] When it is determined that the second service node is the real master node, the master node is converted into the slave node.

[0017] Optionally, performing data communication with the second service node to determine the real master node includes:

[0018] Obtaining the ID hash value of the second service node;

[0019] When the ID hash value of the second service node is greater than the ID hash value of the second service node, determining that the second service node is the real master node;

[0020] When the ID hash value of the second service node is smaller than its own ID hash value, it determines itself as the real master node.

[0021] Optionally, after determining the first service node to which the latest consensus information belongs according to each consensus information version number, obtaining snapshot data of the latest consensus information from the first service node, and synchronizing the snapshot data to all the surviving service nodes in the service cluster, the method further includes:

[0022] When a write request is received, the local consensus information is updated according to the write request, a version number of the updated consensus information is obtained, and the write request based on the updated consensus information version number is forwarded to each of the surviving service nodes in the service cluster;

[0023] If a write success response is received from all the surviving service nodes, it is determined that the write request is executed successfully;

[0024] If no write success response is received from all the surviving service nodes, a third service node that has not fed back the write success response is determined, and the snapshot data of the local consensus information is synchronized to the third service node;

[0025] When the synchronization success response and the write success response fed back by the third service node are received, it is determined that the write request is executed successfully.

[0026] Optionally, the data synchronization method further includes:

[0027] When the target service node serves as the slave node, if a write request based on the updated consensus information version number is received, it is determined whether the difference between the local consensus information version number and the updated consensus information version number is 1;

[0028] If it is determined that the difference between the local consensus information version number and the updated consensus information version number is 1, the write request is executed and the write success response is fed back to the write request node, and the local consensus information version number is increased by 1.

[0029] Optionally, the data synchronization method further includes:

[0030] When the target service node serves as the master node, a heartbeat signal is periodically sent to each non-surviving service node in the service cluster;

[0031] When a heartbeat response is received, determining a non-surviving service node that feeds back the heartbeat response;

[0032] The snapshot data of the local consensus information is synchronized to the non-survival service node, and the non-survival service node is added to the surviving service node set.

[0033] In a second aspect, the present application further discloses a data synchronization device, which is applied to any target service node in a service cluster, and the device includes:

[0034] A first sending module is used to send a master node election request to each surviving service node in the service cluster when the target service node is a candidate node, and convert the candidate node into a master node when receiving election success responses fed back by all the surviving service nodes;

[0035] A second sending module is used to send a data acquisition request to each of the survival service nodes in the service cluster when the target service node serves as the master node, so as to obtain the consensus information version number fed back by each of the survival service nodes;

[0036] A synchronization module is used to determine the first service node to which the latest consensus information belongs according to each consensus information version number, obtain snapshot data of the latest consensus information from the first service node, and synchronize the snapshot data to all the surviving service nodes in the service cluster.

[0037] In a third aspect, the present application further discloses an electronic device, comprising:

[0038] Memory for storing computer programs;

[0039] A processor is used to implement the steps of any one of the data synchronization methods described above when executing the computer program.

[0040] In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the data synchronization methods described above are implemented.

[0041] The present application provides a data synchronization method, which is applied to any target service node in a service cluster. The method includes: when the target service node acts as a candidate node, sending a master node election request to each surviving service node in the service cluster, and upon receiving an election success response fed back by all the surviving service nodes, the candidate node is converted to a master node; when the target service node acts as the master node, sending a data acquisition request to each surviving service node in the service cluster to obtain a consensus information version number fed back by each surviving service node; determining the first service node to which the latest consensus information belongs according to each consensus information version number, obtaining snapshot data of the latest consensus information from the first service node, and synchronizing the snapshot data to all the surviving service nodes in the service cluster.

[0042] Applying the technical solution provided by this application, different node roles can be added to any target service node in the service cluster, including candidate nodes, master nodes and slave nodes. When the target service node is a candidate node, it can participate in the master node election to become the master node in the service cluster with the permission of all surviving service nodes in the service cluster; when the target service node is a master node, the consensus information version number can be obtained from all surviving service nodes in the service cluster to determine the latest version of the consensus information, that is, the latest consensus information, and obtain the snapshot data of the latest consensus information to achieve data synchronization of the consensus information in the service cluster. It can be seen that this technical solution selects the master node in the service cluster by continuously switching the node role, and determines the latest consensus information by comparing the consensus information version numbers between the service nodes, thereby achieving data synchronization of the latest consensus information in the service cluster based on the master node, which can avoid the system delay problem caused by obtaining consensus information from the middleware under the centralized management consensus information solution, and can also solve the problem of inconsistent consensus information that is easy to occur under the decentralized management consensus information solution. Therefore, this technical solution can accurately ensure the consistency of consensus information in the service cluster under the premise of effectively ensuring the system delay performance.

[0043] A data synchronization device, electronic device and computer-readable storage medium provided in this application also have the above-mentioned technical effects, which will not be described in detail in this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings required for use in the description of the prior art and the embodiments of the present application are briefly introduced below. Of course, the drawings described below in relation to the embodiments of the present application are only part of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present application.

[0045] FIG1 is a flow chart of a data synchronization method provided by the present application;

[0046] FIG2 is a schematic diagram of a node state transfer state machine provided by the present application;

[0047] FIG3 is a schematic diagram of the structure of a data synchronization device provided by the present application;

[0048] FIG. 4 is a schematic diagram of the structure of an electronic device provided in the present application. DETAILED DESCRIPTION

[0049] The core of this application is to provide a data synchronization method, which can accurately ensure the consistency of consensus information within a service cluster under the premise of effectively ensuring the system latency performance; another core of this application is to provide a data synchronization device, electronic device and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0050] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] An embodiment of the present application provides a data synchronization method.

[0052] Please refer to FIG. 1 , which is a flow chart of a data synchronization method provided in the present application. The data synchronization method can be applied to any target service node in a service cluster, including the following S101 to S103 .

[0053] It should be noted that the data synchronization method provided in the embodiment of the present application is implemented based on a service cluster, and the service node includes multiple service nodes, aiming to achieve data synchronization between the service nodes in the service cluster, and the synchronized data is specifically consensus information. The data synchronization method provided in the embodiment of the present application can be applied to any service node in the service cluster, that is, the above-mentioned target service node.

[0054] S101: When the target service node is a candidate node, it sends a master node election request to each surviving service node in the service cluster, and upon receiving election success responses fed back by all surviving service nodes, the candidate node is converted into a master node.

[0055] This step is intended to achieve the election of the master node. Specifically, when the target service node acts as a candidate node, it can participate in the election of the master node in the service cluster. It can send a master node election request to each surviving service node in the service cluster, that is, inform each surviving service node in the service cluster that it will participate in the election of the master node. If all surviving service nodes in the service cluster feedback an election success response to the target service node, then the target service node can be converted from a candidate node to a master node. Of course, if the target service node does not receive an election success response from all surviving service nodes in the service cluster (for example, it receives 0 election success responses or fewer election success responses than the number of surviving service nodes), it can be determined that the target service node has failed to participate in the election of the master node and cannot become the master node in the service cluster.

[0056] Furthermore, in one embodiment of the present application, the data synchronization method may also include: when an election success response is not received from all surviving service nodes, converting the candidate node to a slave node; when the target service node acts as a slave node, starting the heartbeat timing monitoring of the master node, and converting the slave node to a candidate node when the heartbeat signal of the master node is not detected.

[0057] Specifically, when the target service node fails to participate in the master node election and is determined to be unable to become the master node, it can be converted from a candidate node to a slave node, which is essentially any surviving service node in the service cluster. It can monitor the heartbeat of the master node regularly. Once it finds that the heartbeat signal of the master node cannot be normally monitored, it can be converted from a slave node to a candidate node to re-participate in the master node election. Of course, if the heartbeat signal of the master node can be normally monitored, it means that there is no abnormality in the master node, and the target service node can continue to operate normally as a slave node.

[0058] S102: When the target service node serves as the master node, a data acquisition request is sent to each surviving service node in the service cluster to obtain the consensus information version number fed back by each surviving service node.

[0059] This step aims to achieve the acquisition of the consensus information version number in the service cluster. Specifically, when the target service node acts as the master node, it can send a data acquisition request to each surviving service node in the service cluster, so that each surviving service node can feedback the version number of the consensus information saved by itself to the master node. It can be understood that each service node in the service cluster maintains a service node list of the service cluster, and can achieve data interaction with other service nodes based on the node IP in the service node list (including the sending of the master node election request as mentioned above, the feedback of the election success response, the sending of the data acquisition request, the feedback of the consensus information version number, and the acquisition and synchronization of the snapshot data described later). In addition, data interaction between service nodes can be achieved through corresponding data communication protocols, such as the RPC (Remote Procedure Call) protocol.

[0060] S103: Determine the first service node to which the latest consensus information belongs according to the version numbers of the consensus information, obtain snapshot data of the latest consensus information from the first service node, and synchronize the snapshot data to all surviving service nodes in the service cluster.

[0061] This step aims to achieve the determination of the latest consensus information and its data synchronization. Specifically, after obtaining the consensus information version number fed back by each surviving service node in the service cluster, the target service node as the master node can determine the latest version of the consensus information, i.e., the above-mentioned latest consensus information, through comparative analysis between the version numbers, and then determine the service node for the latest consensus information, i.e., the above-mentioned first service node, thereby obtaining the snapshot data of the latest consensus information on the first service node, and using the snapshot data to achieve data synchronization of the consensus information between the service nodes in the service cluster.

[0062] In one embodiment of the present application, after determining the first service node to which the latest consensus information belongs according to each consensus information version number, obtaining snapshot data of the latest consensus information from the first service node, and synchronizing the snapshot data to all surviving service nodes in the service cluster, it also includes:

[0063] When a write request is received, the local consensus information is updated according to the write request, the updated consensus information version number is obtained, and the write request based on the updated consensus information version number is forwarded to each surviving service node in the service cluster;

[0064] If a write success response is received from all surviving service nodes, the write request is determined to be executed successfully;

[0065] If no write success response is received from all surviving service nodes, a third service node that has not fed back a write success response is determined, and the snapshot data of the local consensus information is synchronized to the third service node;

[0066] When the synchronization success response and the write success response fed back by the third service node are received, it is determined that the write request is executed successfully.

[0067] The data synchronization method provided in the embodiment of the present application is intended to realize the processing of write requests after the data synchronization operation. It is understandable that after completing the data synchronization of consensus information between the service nodes in the service cluster, the service cluster can receive and process the write request. It should be noted that in the service cluster, only the master node can process the write request.

[0068] During the implementation process, if the target service node, which is the master node, receives a write request, it can first update the local consensus information based on the write request. At this time, the local consensus information version number is increased by 1, and then the write request based on the new version number is forwarded to other storage service nodes in the service cluster; further, for other service nodes in the service cluster, when receiving the write request, they first perform a legitimacy check on it. If the legitimacy check passes, they will execute the write request and feedback a successful write response to the master node. If the legitimacy check fails, the write request will not be executed and the successful write response will not be fed back to the master node (or a failed write response will be fed back to the master node); finally, for the master node, if a successful write response is received from all surviving service nodes, it can be considered that the write request has been successfully executed. Otherwise, it can be considered that the write request has failed.

[0069] Among them, when it is determined that the write request execution fails, the service node that has not fed back a successful write response can be queried, that is, the above-mentioned third service node (the number of which is not unique). It can be understood that the third service node failed to successfully execute the write request because it failed to pass the legitimacy check of the write request, and the failure to pass the legitimacy check of the write request may be due to the fact that the consensus information in the third service node is not synchronized with other service nodes. Based on this, the master node can synchronize the snapshot data of its own consensus information to the third service node. Obviously, if the synchronization success response and write success response fed back by the third service node are received, it can be determined that the write request has been successfully executed. Of course, if the synchronization success response and write success response fed back by the third service node are still not received, it can be determined that the write request still failed to execute. At this time, it can return to S102 to re-execute the synchronization operation of the consensus information.

[0070] Furthermore, the data synchronization method may also include:

[0071] When the target service node acts as a slave node, if it receives a write request based on the updated consensus information version number, it determines whether the difference between the local consensus information version number and the updated consensus information version number is 1;

[0072] If it is determined that the difference between the local consensus information version number and the updated consensus information version number is 1, the write request is executed and a write success response is fed back to the write request node, and the local consensus information version number is increased by 1.

[0073] An embodiment of the present application provides a method for verifying the legitimacy of a write request. When the target service node acts as a slave node, during the process of executing a write request in the service cluster, it will receive a write request based on the updated consensus information version number sent by the master node. At this time, it is possible to determine whether the write request is legal by judging whether the difference between the updated consensus information version number and its own consensus information version number is 1. Therefore, the updated consensus information version number is obtained by adding 1 to the master node's own consensus information version number, and before receiving the write request, the version number of the consensus information on the master node and each slave node should be consistent. Furthermore, when the target service node as a slave node determines that the write request is legal, it can execute the write request and add 1 to its own consensus information version number to update the consensus information version number.

[0074] It can be seen that the data synchronization method provided in the embodiment of the present application can add different node roles for any target service node in the service cluster, including candidate nodes, master nodes and slave nodes. When the target service node is a candidate node, it can participate in the master node election to become the master node in the service cluster with the permission of all surviving service nodes in the service cluster; when the target service node is a master node, the consensus information version number can be obtained from all surviving service nodes in the service cluster to determine the latest version of the consensus information, that is, the latest consensus information, and obtain the snapshot data of the latest consensus information to achieve data synchronization of the consensus information in the service cluster. It can be seen that the technical solution selects the master node in the service cluster by continuously switching the node roles, and determines the latest consensus information by comparing the consensus information version numbers between the service nodes, thereby realizing the data synchronization of the latest consensus information in the service cluster based on the master node, which can avoid the system delay problem caused by obtaining consensus information from the middleware under the centralized management consensus information solution, and can also solve the problem of inconsistent consensus information that is easy to occur under the decentralized management consensus information solution. Therefore, the technical solution can accurately ensure the consistency of consensus information in the service cluster under the premise of effectively ensuring the system delay performance.

[0075] Based on the above embodiments:

[0076] In one embodiment of the present application, the data synchronization method may further include:

[0077] When the target service node acts as the master node, it periodically sends heartbeat signals to each surviving service node in the service cluster;

[0078] When an abnormal heartbeat response including master node information is received, a second service node to which the abnormal heartbeat response belongs is determined, and data communication is performed with the second service node to determine the real master node;

[0079] When it is determined that the second service node is the real master node, the master node is converted into a slave node.

[0080] It should be noted that in a service cluster, there should be only one master node. When it is detected that there is more than one master node in the service cluster, role verification is required to identify the real master node in the service cluster, that is, the real master node mentioned above.

[0081] Based on this, for the target service node that serves as the master node, it can periodically send heartbeat signals to each surviving service node in the service cluster. If it receives normal heartbeat responses from all surviving service nodes, it can be determined that there is only one master node in the service cluster at this time; if it receives an abnormal heartbeat response from a surviving service node (the heartbeat response contains the master node information, indicating that the service node that sends the abnormal heartbeat response is also the master node), it means that there is more than one master node in the service cluster. At this time, role verification is required to find the real master node, and the service node that is not the real master node can be converted into a slave node.

[0082] Furthermore, the search for the real master node can be determined by data communication between the service nodes that are currently the master node. A possible implementation method is as follows:

[0083] The data communication with the second service node to determine the real master node may include: obtaining the ID hash value of the second service node; when the ID hash value of the second service node is greater than its own ID hash value, determining the second service node as the real master node; when the ID hash value of the second service node is less than its own ID hash value, determining itself as the real master node. In other words, when there are multiple master nodes in the service cluster, the service node with the largest ID hash value may be selected as the real master node.

[0084] In one embodiment of the present application, the data synchronization method may further include:

[0085] When the target service node acts as the master node, it periodically sends heartbeat signals to each non-surviving service node in the service cluster;

[0086] When a heartbeat response is received, a non-surviving service node that feeds back the heartbeat response is determined;

[0087] Synchronize the snapshot data of the local consensus information to the non-survival service nodes, and add the non-survival service nodes to the set of surviving service nodes.

[0088] It can be understood that in addition to surviving service nodes, the service cluster also includes non-surviving service nodes, and the survival status of the service nodes in the service cluster is dynamically updated. There is a situation at any time that a service node fails and becomes a non-surviving service node, a non-surviving service node recovers from a fault and becomes a surviving service node, and a new surviving service node is added. Then, for each non-surviving service node in the service cluster, the target service node as the master node can perform regular heartbeat monitoring on it. Once a heartbeat response fed back by a non-surviving service node is detected, it can be considered that the non-surviving service node has recovered from the fault and become a surviving service node. At this time, the snapshot data of the local consensus information can be used to synchronize the consensus information to ensure the consistency of the consensus information in the service cluster. In addition, the non-surviving service node can also be added to the surviving service node set (such as the surviving service node list).

[0089] An embodiment of the present application provides another data synchronization method.

[0090] Please refer to FIG. 2 , which is a schematic diagram of a node state transfer state machine provided by the present application. Based on the node state transfer state machine, the implementation process of the data synchronization method provided by the embodiment of the present application is as follows:

[0091] First, the data communication between the service nodes in the embodiment of the present application is implemented based on RPC, and the predefined RPC message structure, node role, and consensus information structure are as follows:

[0092] 1. Request (request message):

[0093] The first field, Member Id, is of string type and is the unique ID of the node initiating the RPC.

[0094] The second field, Type, RPC type, enumeration type, including Election, HeartBeat, Push (update consensus information), Check (check the target member consensus information version number), Pull (pull consensus information), Merge (merge consensus information).

[0095] The third field, Sequence Id, is a long integer, which is the sequence number of the consensus information incremental update log.

[0096] The fourth field, Log, is a binary sequence, which is the incremental update log of consensus information.

[0097] The fifth field, Snapshot, is a binary sequence, which is a snapshot of the consensus information.

[0098] 2. Response (response message):

[0099] The first field, Member Id, is of string type and is the unique ID of the responding RPC node.

[0100] The second field, Response Data, is a binary sequence, indicating the response data.

[0101] 3. Consensus information:

[0102] The first field, Sequence Id, is a long integer and is the consensus information version number.

[0103] The second field, Consensus Data, is the content of consensus information.

[0104] Secondly, in the embodiment of the present application, the node roles in the service cluster are divided into three types: Leader (master node), Follower (slave node), and Candidate (candidate node).

[0105] 1. The node is initialized to Follower state.

[0106] 2. After the Follower node is started, it will monitor the heartbeat of the Leader node. If it does not receive the heartbeat of the Leader node within a period of time, its own state will be converted to Candidate.

[0107] 3. The Candidate node sends an Election-type RPC message to all surviving service nodes in the service cluster to elect a Leader. If the election is successful, the state is changed to Leader. If the Leader heartbeat is received, the state is changed to Follower.

[0108] 4. The Leader node will send a heartbeat signal to all service nodes in the service cluster. If it finds that there are other Leaders in the service cluster through the heartbeat response, it will put its own Member Id hash value and the Member Id hash values ​​of other Leaders into a set for sorting; if its own Member Id hash value is the largest, it will be ignored. Otherwise, it will send a Merge type RPC message to the service node with the largest Member Id hash value and the status will be converted to Follower after success.

[0109] Finally, in the embodiment of the present application, the node behavior of each service node in the service cluster can be divided into the following four aspects:

[0110] 1. Leader election behavior.

[0111] Candidate sends Election RPC messages to all surviving service nodes in the service cluster. It needs to get successful Responses from all surviving service nodes to become a Leader. If the hash value of the Member Id of the service node in the service cluster is less than that of Candidate and its role is Candidate, it returns a successful Response, otherwise it returns a failure. After the Candidate is successfully elected, it cannot process write requests because it cannot guarantee that its own consensus information is the latest at this time. It needs to perform initialization before responding to write requests. After the Candidate is successfully elected, it first sends Check RPC to all cluster nodes in the service cluster to obtain the consensus information version number on each service node in the current service cluster, and uses Pull RPC to pull the consensus information snapshot with the largest consensus information version number in the surviving service node, and then uses PushRPC to synchronize the consensus information snapshot to all service nodes in the service cluster.

[0112] 2. Synchronization of consensus information.

[0113] The service cluster is a strong master cluster. Only the Leader node can process write requests and modify consensus information. When the Leader node processes a write request, it first applies the request to its own consensus information, and increases the version number of its own consensus information by 1. Then, it propagates the request as a Log to the Follower node in the form of Push RPC, and carries its own consensus information version number as the Sequence Id in the Push RPC. The Follower node receives and verifies the legitimacy of the Push RPC. If it detects that the RPCSequence Id is not the version number of its own consensus information plus 1, it returns a "version number error" Response. Otherwise, it applies the Log to its own consensus information and increases the version number of the consensus information by 1, and returns a "success" Response. If the Leader receives a "version number error" Response, it sends the current consensus information snapshot to the node. If it receives a "success" Response, it does not do anything. After processing all successful Responses of the current service cluster, the write request is successfully processed.

[0114] 3. Node detection and configuration changes.

[0115] For scenarios where service nodes in the service cluster startup configuration are online or offline, the Leader can periodically send heartbeats to maintain information about surviving service nodes. If a service node is offline, a timeout or IO exception will occur when the Leader sends a heartbeat to the service node, and the Leader removes it from the set of surviving service nodes. The Leader will also periodically try to send heartbeats to non-surviving service nodes in the configuration. If the attempt is successful and the service node is not the Leader, the consensus information will be fully synchronized to the service node, and then it will be moved to the list of surviving service nodes. If it is the Leader, the merge logic will be executed. To add service nodes outside the configuration, you need to first use the command line tool to modify the node configuration, and then start the service node.

[0116] 4. Recovery after split brain.

[0117] The Leader node will send heartbeats to all service nodes in the service cluster. If other Leaders are found in the cluster through the heartbeat response, the hash value of its own Member Id and the hash values ​​of other Leaders' Member Ids will be put into a set for sorting; if its own Member Id hash value is the largest, it will be ignored, otherwise a Merge type RPC will be sent to the node with the largest Member Id hash value. After the Leader successfully processes it, its status will be converted to Follower.

[0118] It can be seen that the data synchronization method provided in the embodiment of the present application can add different node roles for any target service node in the service cluster, including candidate nodes, master nodes and slave nodes. When the target service node is a candidate node, it can participate in the master node election to become the master node in the service cluster with the permission of all surviving service nodes in the service cluster; when the target service node is a master node, the consensus information version number can be obtained from all surviving service nodes in the service cluster to determine the latest version of the consensus information, that is, the latest consensus information, and obtain the snapshot data of the latest consensus information to achieve data synchronization of the consensus information in the service cluster. It can be seen that the technical solution selects the master node in the service cluster by continuously switching the node roles, and determines the latest consensus information by comparing the consensus information version numbers between the service nodes, thereby realizing the data synchronization of the latest consensus information in the service cluster based on the master node, which can avoid the system delay problem caused by obtaining consensus information from the middleware under the centralized management consensus information solution, and can also solve the problem of inconsistent consensus information that is easy to occur under the decentralized management consensus information solution. Therefore, the technical solution can accurately ensure the consistency of consensus information in the service cluster under the premise of effectively ensuring the system delay performance.

[0119] An embodiment of the present application provides a data synchronization device.

[0120] Please refer to FIG. 3 , which is a schematic diagram of the structure of a data synchronization device provided by the present application. The data synchronization device can be applied to any target service node in a service cluster, including:

[0121] The first sending module 1 is used to send a master node election request to each surviving service node in the service cluster when the target service node is a candidate node, and convert the candidate node into a master node when receiving the election success response fed back by all surviving service nodes;

[0122] The second sending module 2 is used to send a data acquisition request to each surviving service node in the service cluster when the target service node serves as the master node, so as to obtain the consensus information version number fed back by each surviving service node;

[0123] Synchronization module 3 is used to determine the first service node to which the latest consensus information belongs according to the version number of each consensus information, obtain snapshot data of the latest consensus information from the first service node, and synchronize the snapshot data to all surviving service nodes in the service cluster.

[0124] It can be seen that the data synchronization device provided in the embodiment of the present application can add different node roles for any target service node in the service cluster, including candidate nodes, master nodes and slave nodes. When the target service node is a candidate node, it can participate in the master node election to become the master node in the service cluster with the permission of all surviving service nodes in the service cluster; when the target service node is a master node, the consensus information version number can be obtained from all surviving service nodes in the service cluster to determine the latest version of the consensus information, that is, the latest consensus information, and obtain the snapshot data of the latest consensus information to achieve data synchronization of the consensus information in the service cluster. It can be seen that the technical solution selects the master node in the service cluster by continuously switching the node roles, and determines the latest consensus information by comparing the consensus information version numbers between the service nodes, thereby achieving data synchronization of the latest consensus information in the service cluster based on the master node, which can avoid the system delay problem caused by obtaining consensus information from the middleware under the centralized management consensus information solution, and can also solve the problem of inconsistent consensus information that is easy to occur under the decentralized management consensus information solution. Therefore, the technical solution can accurately ensure the consistency of consensus information in the service cluster under the premise of effectively ensuring the system delay performance.

[0125] In one embodiment of the present application, the data synchronization device may also include a role conversion module, which is used to convert from a candidate node to a slave node when an election success response is not received from all surviving service nodes; when the target service node acts as a slave node, it starts the heartbeat timing monitoring of the master node, and when the heartbeat signal of the master node is not detected, it converts from a slave node to a candidate node.

[0126] In one embodiment of the present application, the data synchronization device may also include a role verification module, which is used to periodically send heartbeat signals to each surviving service node in the service cluster when the target service node acts as a master node; when an abnormal heartbeat response containing master node information is received, determine the second service node to which the abnormal heartbeat response belongs, and communicate data with the second service node to determine the real master node; when the second service node is determined to be the real master node, the master node is converted into a slave node.

[0127] In one embodiment of the present application, the above-mentioned role verification module can be specifically used to obtain the ID hash value of the second service node; when the ID hash value of the second service node is greater than its own ID hash value, the second service node is determined to be the real master node; when the ID hash value of the second service node is less than its own ID hash value, it is determined to be the real master node.

[0128] In one embodiment of the present application, the data synchronization device may also include a write processing module, which is used to update the local consensus information according to the write request when a write request is received, obtain the updated consensus information version number, and forward the write request based on the updated consensus information version number to each surviving service node in the service cluster; if a write success response is received from all surviving service nodes, it is determined that the write request is executed successfully; if a write success response is not received from all surviving service nodes, a third service node that has not fed back a write success response is determined, and the snapshot data of the local consensus information is synchronized to the third service node; when a synchronization success response and a write success response are received from the third service node, it is determined that the write request is executed successfully.

[0129] In one embodiment of the present application, the data synchronization device may further include a write response module, which is used to determine whether the difference between the local consensus information version number and the updated consensus information version number is 1 when the target service node acts as a slave node and receives a write request based on the updated consensus information version number; if it is determined that the difference between the local consensus information version number and the updated consensus information version number is 1, execute the write request and feedback a write success response to the write request node, and increase the local consensus information version number by 1.

[0130] In one embodiment of the present application, the data synchronization device may also include a node update module, which is used to periodically send heartbeat signals to each non-surviving service node in the service cluster when the target service node acts as the master node; when a heartbeat response is received, determine the non-surviving service node that feeds back the heartbeat response; synchronize the snapshot data of the local consensus information to the non-surviving service node, and add the non-surviving service node to the set of surviving service nodes.

[0131] For an introduction to the device provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0132] An embodiment of the present application provides an electronic device.

[0133] Please refer to FIG. 4 , which is a schematic diagram of the structure of an electronic device provided by the present application. The electronic device may include:

[0134] Memory for storing computer programs;

[0135] A processor, when used to execute a computer program, can implement the steps of any of the above-mentioned data synchronization methods.

[0136] As shown in Fig. 4, it is a schematic diagram of the composition structure of an electronic device, which may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0137] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0138] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of the data synchronization method.

[0139] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0140] When the target service node is a candidate node, it sends a master node election request to each surviving service node in the service cluster, and when it receives the election success response from all surviving service nodes, it is converted from a candidate node to a master node;

[0141] When the target service node acts as the master node, it sends a data acquisition request to each surviving service node in the service cluster to obtain the consensus information version number fed back by each surviving service node;

[0142] The first service node to which the latest consensus information belongs is determined according to the version number of each consensus information, the snapshot data of the latest consensus information is obtained from the first service node, and the snapshot data is synchronized to all surviving service nodes in the service cluster.

[0143] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0144] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0145] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0146] Of course, it should be noted that the structure shown in FIG. 4 does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include more or fewer components than those shown in FIG. 4 , or combine certain components.

[0147] An embodiment of the present application provides a computer-readable storage medium.

[0148] The computer-readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned data synchronization methods can be implemented.

[0149] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0150] For an introduction to the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0151] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0152] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0154] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples herein, and the description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A data synchronization method, characterized in that: Applied to any target service node in a service cluster, the method comprises: When the target service node is used as a candidate node, it sends a master node election request to each surviving service node in the service cluster, and upon receiving election success responses fed back by all the surviving service nodes, the candidate node is converted into a master node; When the target service node serves as the master node, a data acquisition request is sent to each of the survival service nodes in the service cluster to obtain the consensus information version number fed back by each of the survival service nodes; Determine the first service node to which the latest consensus information belongs according to each consensus information version number, obtain snapshot data of the latest consensus information from the first service node, and synchronize the snapshot data to all the surviving service nodes in the service cluster.

2. The data synchronization method according to claim 1, characterized in that: Also includes: When the election success response fed back by all the surviving service nodes is not received, the candidate node is converted into a slave node; When the target service node serves as the slave node, it starts the heartbeat timing monitoring for the master node, and when the heartbeat signal of the master node is not detected, the slave node is converted into the candidate node.

3. The data synchronization method according to claim 2, characterized in that: Also includes: When the target service node serves as the master node, periodically sending the heartbeat signal to each of the surviving service nodes in the service cluster; When an abnormal heartbeat response including master node information is received, determining a second service node to which the abnormal heartbeat response belongs, and performing data communication with the second service node to determine a real master node; When it is determined that the second service node is the real master node, the master node is converted into the slave node.

4. The data synchronization method according to claim 3, characterized in that: Performing data communication with the second service node to determine the real master node includes: Obtaining the ID hash value of the second service node; When the ID hash value of the second service node is greater than the ID hash value of the second service node, determining that the second service node is the real master node; When the ID hash value of the second service node is smaller than its own ID hash value, it determines itself as the real master node.

5. The data synchronization method according to claim 2, characterized in that: After determining the first service node to which the latest consensus information belongs according to each consensus information version number, obtaining snapshot data of the latest consensus information from the first service node, and synchronizing the snapshot data to all the surviving service nodes in the service cluster, the method further includes: When a write request is received, the local consensus information is updated according to the write request, a version number of the updated consensus information is obtained, and the write request based on the updated consensus information version number is forwarded to each of the surviving service nodes in the service cluster; If a write success response is received from all the surviving service nodes, it is determined that the write request is executed successfully; If no write success response is received from all the surviving service nodes, a third service node that has not fed back the write success response is determined, and the snapshot data of the local consensus information is synchronized to the third service node; When the synchronization success response and the write success response fed back by the third service node are received, it is determined that the write request is executed successfully.

6. The data synchronization method according to claim 5, characterized in that: Also includes: When the target service node serves as the slave node, if a write request based on the updated consensus information version number is received, it is determined whether the difference between the local consensus information version number and the updated consensus information version number is 1; If it is determined that the difference between the local consensus information version number and the updated consensus information version number is 1, the write request is executed and the write success response is fed back to the write request node, and the local consensus information version number is increased by 1.

7. The data synchronization method according to claim 1, characterized in that: Also includes: When the target service node serves as the master node, a heartbeat signal is periodically sent to each non-surviving service node in the service cluster; When a heartbeat response is received, determining a non-surviving service node that feeds back the heartbeat response; The snapshot data of the local consensus information is synchronized to the non-survival service node, and the non-survival service node is added to the surviving service node set.

8. A data synchronization device, characterized in that: Applied to any target service node in a service cluster, the device comprises: A first sending module is used to send a master node election request to each surviving service node in the service cluster when the target service node is a candidate node, and convert the candidate node into a master node when receiving election success responses fed back by all the surviving service nodes; A second sending module is used to send a data acquisition request to each of the survival service nodes in the service cluster when the target service node serves as the master node, so as to obtain the consensus information version number fed back by each of the survival service nodes; A synchronization module is used to determine the first service node to which the latest consensus information belongs according to each consensus information version number, obtain snapshot data of the latest consensus information from the first service node, and synchronize the snapshot data to all the surviving service nodes in the service cluster.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data synchronization method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 7 are implemented.

Images