Data replication method, device and equipment based on distributed cluster, medium and product

Abstract

The invention discloses a data replication method, device and equipment based on a distributed cluster, a medium and a product. The method can be applied to the field of financial science and technology, is applied to any cluster node in a distributed cluster, and comprises the following steps: if it is determined that the own cluster node is a leader node, obtaining change log data from a target data source, and sending the change log data to a distributed message service cluster, recording a current data replication point location of the change log data after sending is completed; generating a data replication request according to the current data replication point location, and sending the data replication request to the follower node, so that the follower node generates and feeds back a data confirmation response according to the data replication request; the follower nodes are other cluster nodes except the leader node in the distributed cluster; and determining a data replication result according to the received data confirmation response sent by the follower node. According to the technical scheme, data replication consistency is achieved, and data replication efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a data replication method, device, equipment, medium and product based on a distributed cluster. Background Art

[0002] When copying change log data from data sources to the consumer service cluster, data replication is typically performed using a replication component in a master-slave mode. The point information in the replication component is updated approximately every 10 seconds, which can result in the latest point information not being updated to the database in a timely manner. For example, if point information is stored at 09:00 and the replication component fails at 09:05, the backup server will take over and continue data replication. During the replication process, the backup server retrieves the point information stored at 09:00, meaning data replication continues from that time. Therefore, the replicated data from 09:00 to 09:05 is duplicated.

[0003] The existing data replication method described above suffers from point information lags, and the recovery process requires processing a large amount of old data. Furthermore, the backup server takes approximately ten minutes to boot and deploy, preventing timely service provision to the consumer service cluster. This impacts data replication efficiency and reduces device availability. Furthermore, the large amount of duplicate data required to process data makes it impossible to ensure data consistency between the primary and backup servers. Summary of the Invention

[0004] The present invention provides a data replication method, device, equipment, medium and product based on a distributed cluster to achieve data replication consistency, improve data replication efficiency, avoid processing large amounts of duplicate data, and avoid service interruption for downstream consumer service clusters.

[0005] According to one aspect of the present invention, a data replication method based on a distributed cluster is provided, which is applied to any cluster node in the distributed cluster. The method includes:

[0006] If it is determined that its own cluster node is the leader node, then obtaining the change log data from the target data source and sending the change log data to the distributed message service cluster;

[0007] After the change log data is sent, recording the current data copy point of the change log data;

[0008] Generate a data replication request according to the current data replication point, and send the data replication request to a follower node, so that the follower node generates and feeds back a data confirmation response according to the data replication request; the follower node is a cluster node other than the leader node in the distributed cluster;

[0009] The data replication result is determined according to the received data confirmation response sent by the follower node.

[0010] According to another aspect of the present invention, a data replication device based on a distributed cluster is provided, which is configured on any cluster node in the distributed cluster, and includes:

[0011] A change data sending module, configured to obtain change log data from a target data source and send the change log data to a distributed message service cluster if the node of the cluster itself is determined to be a leader node;

[0012] A point recording module, configured to record the current data copy point of the change log data after the change log data is sent;

[0013] a replication request generation module, configured to generate a data replication request based on the current data replication point, and send the data replication request to a follower node, so that the follower node generates and feeds back a data confirmation response based on the data replication request; the follower node is a cluster node other than the leader node in the distributed cluster;

[0014] The replication result determination module is used to determine the data replication result according to the received data confirmation response sent by the follower node.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed cluster-based data replication method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed cluster-based data replication method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the distributed cluster-based data replication method according to any embodiment of the present invention is implemented.

[0021] The technical scheme of the embodiment of the present application realizes the consistency of data replication in the process of data replication, ensures the real-time and accurate synchronization of the log point information in multiple nodes, improves the consistency of data, enhances the fault tolerance of the distributed cluster system, guarantees the integrity and continuity of data replication in the case of partial node failure, and does not need to consume long time to obtain point data and deploy nodes during node failure switching, avoids the service interruption of the downstream consumption service cluster, can quickly locate and continue to replicate data, and improves the data replication efficiency.

[0022] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0024] Figure 1 is a flow chart of a data replication method based on a distributed cluster according to the first embodiment of the present application;

[0025] Figure 2 is a flow chart of a data replication method based on a distributed cluster according to the second embodiment of the present application;

[0026] Figure 3A is a flow chart of an existing data replication method according to the third embodiment of the present application;

[0027] Figure 3B is a flow chart of an improved data replication method according to the third embodiment of the present application;

[0028] Figure 4 is a structural schematic diagram of a data replication device based on a distributed cluster according to the fourth embodiment of the present application;

[0029] Figure 5The present invention is a schematic structural diagram of an electronic device for implementing the distributed cluster-based data replication method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Example 1

[0033] Figure 1 This is a flow chart of a data replication method based on a distributed cluster provided in the first embodiment of the present invention. This embodiment is applicable to the case where changed log data is replicated from a data source to a downstream consumer service cluster. This method can be executed by a data replication device based on a distributed cluster. The data replication device based on a distributed cluster can be implemented in the form of hardware and / or software. The data replication device based on a distributed cluster can be configured in an electronic device. Figure 1 As shown, the method can be applied to a distributed cluster, and can be specifically executed by any cluster node in the distributed cluster. The method includes:

[0034] S110: If it is determined that the own cluster node is the leader node, obtain the change log data from the target data source, and send the change log data to the distributed message service cluster.

[0035] S120: After the change log data is sent, the current data copy point of the change log data is recorded.

[0036] S130. Generate a data replication request based on the current data replication point, and send the data replication request to the follower node, so that the follower node can generate and feedback a data confirmation response based on the data replication request; the follower node is the other cluster node in the distributed cluster except the leader node.

[0037] S140: Determine the data replication result according to the received data confirmation response sent by the follower node.

[0038] The distributed cluster can be a replicator cluster built based on the Raft protocol, and the cluster nodes can be replicators, components that read change data from the source database and transmit it to the target system. The target system is a distributed messaging service cluster, for example, a distributed stream processing platform for building real-time data stream processing and message queues.

[0039] The Raft protocol is a consensus algorithm for achieving consistency in distributed systems. Through leader election, log replication, and consistency mechanisms, the Raft protocol ensures that multiple cluster nodes reach consensus when processing data changes.

[0040] For any cluster node in a distributed cluster, if it is determined to be the leader node during a data replication task, it retrieves the change log data from the target data source and sends it to the distributed messaging cluster. At any given execution time, there is only one leader node processing all data replication requests. The target data source is the upstream database cluster, which primarily provides Binary Log (a log file format in relational database management systems) logs for the distributed cluster to call and replicate data to the downstream consumer service cluster.

[0041] After copying the change log data to the distributed messaging service cluster, the current data replication point of the change log data is recorded. The current data replication point is the latest breakpoint in the replication process of the cluster nodes in the target data source. The leader node records the current data replication point information. The leader node encapsulates the current data replication point and its own term number as a log entry and writes it to a local file.

[0042] The leader node initiates a data replication request to a follower node and sends it to the follower node. This request carries the most recently updated log entry. Upon receiving the request, the follower node performs a consistency check on the request. If the consistency check passes, the follower node generates a data confirmation response and sends it back to the leader node.

[0043] The consistency verification manner of the follower node based on the data replication request is that: if it is determined according to the data replication request that a preset consistency verification condition is met, a data confirmation response is generated. The consistency verification condition includes: a check on the leader term number, a continuity check on the log entry, and a check on the log entry index.

[0044] The check on the leader term number, taking any follower node as an example, if the leader term number of the leader node obtained by analyzing the data replication request is not less than the term number of the follower node itself, the check on the leader term number is passed.

[0045] The continuity check on the log entry, taking any follower node as an example, the latest log entry index sent by the leader node is obtained by analyzing the data replication request. According to the latest log entry index, the previous log entry index corresponding to the latest log entry index is determined, the log entry of the follower node at the previous log entry index corresponding to the latest log entry index is determined, and it is judged whether the term number of the log entry is equal to the term number of the leader node obtained by analysis, if yes, the continuity check on the log entry is passed.

[0046] The check on the log entry index, taking any follower node as an example, it is judged whether the latest log entry index sent by the leader node obtained by analysis is greater than the latest log entry index of the follower node itself, if yes, the check on the log entry index is passed.

[0047] If the number of data confirmation responses sent by the follower node and received by the leader node is greater than a preset number threshold, it is considered that the log entry has been safely replicated to each cluster node in the distributed cluster, and the data replication task is ended. If the number of data confirmation responses sent by the follower node and received by the leader node is not greater than the preset number threshold, it is considered that most of the cluster nodes in the distributed cluster have not been successfully replicated, then the leader node checks whether the node itself has failed, if it has failed, a new leader node needs to be selected, if it has not failed, the data replication request is initiated again.

[0048] The technical solution of the embodiment of the present invention obtains change log data from the target data source when determining that its own cluster node is the leader node, sends the change log data to the distributed message service cluster, records the current data replication point of the change log data, generates a data replication request based on the current data replication point, and sends the data replication request to the follower node. According to the data confirmation response received from the follower node, the data replication result is determined, thereby achieving replication consistency in the data replication process, ensuring real-time and accurate synchronization of log point information across multiple nodes, improving data consistency, and enhancing the fault tolerance of the distributed cluster system. Even if some nodes fail, the integrity and continuity of data replication can be guaranteed. Moreover, when switching node failures, there is no need to consume a long time to obtain point data and node deployment, thereby avoiding service interruption for the downstream consumer service cluster. Data can be quickly located and continued to be replicated, thereby improving data replication efficiency.

[0049] Example 2

[0050] Figure 2 This is a flow chart of a data replication method based on a distributed cluster provided in the second embodiment of the present invention. This embodiment is optimized and improved on the basis of the above technical solutions.

[0051] Furthermore, before the step "If it is determined that the own cluster node is the leader node, obtain the change log data from the target data source", add the step "When it is monitored that the own cluster node meets the judgment conditions of the candidate node, send the own cluster node's own term number and its own log entry index to the candidate node, and at the same time, receive the candidate term number and candidate log entry index sent by the candidate node; the candidate node is the other cluster nodes in the distributed cluster except the own cluster node; obtain the first voting result generated by the candidate node based on its own term number and its own log entry index, and at the same time, generate the second voting result according to the candidate term number and candidate log entry index of the candidate node; determine the leader node according to the first voting result and the second voting result." to improve the selection method of the leader node in the distributed cluster.

[0052] It should be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments. Figure 2 As shown, the method includes the following specific steps:

[0053] S210. When it is monitored that its own cluster node meets the judgment conditions of the candidate node, its own cluster node's own term number and its own log entry index are sent to the candidate node. At the same time, the candidate term number and candidate log entry index sent by the candidate node are received; the candidate node is the other cluster node in the distributed cluster except its own cluster node.

[0054] The criteria for determining a candidate node can be pre-set by relevant technical personnel. For example, the criteria for determining a candidate node can be failure to detect the leader node's heartbeat within a preset time. If a leader node is unable to continue to serve as the leader due to a failure or other reasons during its term, follower nodes can compete as candidate nodes for the leader node.

[0055] Taking any cluster node as an example, when it participates in the competition as a candidate node, it sends its own term number and its own log entry index of the cluster node to the candidate node, and at the same time, receives the candidate term number and candidate log entry index sent by other candidate nodes except itself.

[0056] The own log entry index is the index value of the most recent change log data in the current log entry recorded by the own cluster node; the candidate log entry index is the index value of the most recent change log data in the current log entry recorded by the candidate node. The own term number is the latest term number of the own cluster node; the candidate term number is the latest term number of the candidate node.

[0057] S220. Obtain a first voting result generated by the candidate node based on its own term number and its own log entry index. At the same time, generate a second voting result based on the candidate term number and candidate log entry index of the candidate node.

[0058] For any cluster node, the second voting result is generated by the candidate node based on its own term number and its own log entry index. At the same time, the candidate node also generates the first voting result based on its own term number and its own log entry index and the candidate term number and candidate log entry index of other candidate nodes.

[0059] The node sends its second voting result to other candidate nodes and receives the first voting results sent by other candidate nodes.

[0060] In an optional embodiment, a second voting result is generated based on the candidate term number and candidate log entry index of the candidate node, including: judging whether the candidate node meets the preset grant voting judgment conditions based on the candidate term number and candidate log entry index of the candidate node; if so, voting for the candidate node to generate a second voting result.

[0061] It should be noted that a candidate node is only granted voting rights, meaning that other candidate nodes may vote for it, if its candidate term number and candidate log entry index meet specific conditions. Therefore, the first step is to determine whether a candidate node meets the preset voting conditions based on its candidate term number and candidate log entry index. This determines whether to grant voting rights. If so, then vote for the candidate node that meets the conditions, resulting in a second voting result.

[0062] Furthermore, a vote is usually cast for the first candidate node received that meets the voting judgment criteria.

[0063] The above technical solution determines whether the candidate node meets the preset grant voting judgment conditions based on the candidate term number and candidate log entry index of the candidate node, and votes for the candidate node when it is determined that the conditions are met, generating a second voting result, thereby achieving accurate voting for the candidate node, avoiding voting for candidate nodes that do not meet the conditions, resulting in errors in the subsequent selection of the leader node, and thereby improving the accuracy of the selection of the leader node.

[0064] Optionally, the voting judgment conditions are: judging whether the candidate term number of the candidate node is not less than its own term number; and judging whether the candidate log entry index of the candidate node is greater than its own log entry index of its own cluster node.

[0065] If the candidate term number of the candidate node is not less than its own term number, and the candidate log entry index of the candidate node is greater than the own log entry index of its own cluster node, then it is determined that the candidate node meets the judgment conditions for granting voting; if the candidate term number of the candidate node is less than its own term number, and the candidate log entry index of the candidate node is not greater than the own log entry index of its own cluster node, then it is determined that the candidate node does not meet the judgment conditions for granting voting.

[0066] The above technical solution achieves accurate setting of the voting granting judgment conditions by judging whether the candidate term number of the candidate node is not less than its own term number; and judging whether the candidate log entry index of the candidate node is greater than the own log entry index of its own cluster node, thereby improving the accuracy of judging whether the candidate node meets the requirements for voting granting, and further improving the accuracy of subsequent selection of leader nodes.

[0067] S230. Determine a leader node according to the first voting result and the second voting result.

[0068] The node with the highest number of votes becomes the leader node. Each cluster node can obtain the first and second voting results, and therefore can determine whether it is the leader node based on the first and second voting results.

[0069] S240, if it is determined that the self cluster node is the leader node, obtaining change log data from the target data source, and sending the change log data to the distributed message service cluster.

[0070] S250, after the change log data is sent, recording a current data replication point of the change log data.

[0071] S260, generating a data replication request according to the current data replication point, and sending the data replication request to a follower node, so that the follower node generates and feeds back a data confirmation response according to the data replication request; the follower node is a cluster node other than the leader node in the distributed cluster.

[0072] S270, determining a data replication result according to the data confirmation response sent by the follower node.

[0073] The technical scheme of the embodiment sends the self term number and the self log item index of the self cluster node to the candidate node, simultaneously receives the candidate term number and the candidate log item index sent by the candidate node, obtains a first voting result of the candidate node, simultaneously generates a second voting result according to the candidate term number and the candidate log item index of the candidate node, and determines the leader node according to the first voting result and the second voting result, so that the leader node is accurately selected. In the process of selecting the leader node, the cluster nodes can all participate in voting and being voted as the candidate nodes, and can obtain the final voting result to determine the final leader node, so that the leader node is quickly selected in the case of cluster node failure, so that the service interruption of the downstream consumption service cluster is avoided, the data can be quickly located and replicated, and the data replication efficiency is improved.

[0074] Further, in the initial deployment stage of the cluster nodes of the distributed cluster, the leader node election in the initialization stage can be performed by a third-party platform. In an optional embodiment, before obtaining the change log data from the target data source if it is determined that the self cluster node is the leader node, the method further includes: if the current time is the cluster startup time of the distributed cluster, and the leader node confirmation information sent by the data replication monitoring platform is received within the preset time, the self cluster node is determined as the leader node, and the heartbeat information is sent to the follower node.

[0075] The cluster startup time of the distributed cluster is the initial deployment time of the initial cluster node, that is, in the initial deployment stage of the distributed cluster, the leader election is performed on the cluster nodes by the data replication monitoring platform.

[0076] Optionally, the data replication monitoring platform generates the leader node confirmation information in the following manner:

[0077] The data replication monitoring platform obtains node performance data of each cluster node in the distributed cluster, selects a target cluster node according to the node performance data of each cluster node, determines the target cluster node as a leader node, and generates leader node confirmation information and sends it to the target cluster node.

[0078] The node performance data can include node computing capability such as CPU (Central Processing Unit) performance and memory capacity, can also include disk I / O (Input / Output) performance such as disk throughput and disk capacity, and can also include network bandwidth and delay and node load.

[0079] The data replication monitoring platform comprehensively evaluates the node performance data of each cluster node, and selects a cluster node with optimal node performance data in each dimension as a target cluster node. Optionally, the data replication monitoring platform can select the target cluster node based on a pre-trained node selection model. The node performance data of each cluster node is input into the node selection model to obtain the target cluster node predicted and output by the model. The node selection model can be pre-trained based on a large amount of historical node performance data in a historical time period. In addition, in the use stage after the deployment of the cluster nodes of the distributed cluster, the data replication monitoring platform can also monitor the node performance data of each cluster node in real time, so as to continuously optimize the node selection model based on the real-time monitored node performance data.

[0080] The data replication monitoring platform determines the selected target cluster node as a leader node, generates leader node confirmation information, and sends it to the target cluster node. The target cluster node sends heartbeat content to the follower node as the leader node.

[0081] The above technical solution obtains the node performance data of each cluster node in the distributed cluster by the data replication monitoring platform, selects a target cluster node according to the node performance data of each cluster node, and determines the target cluster node as a leader node, which realizes the accurate selection of the initial leader node in the initial deployment stage of the cluster nodes of the distributed cluster, the stage without leader node, so as to facilitate the subsequent data replication process based on the leader node.

[0082] If the cluster node receives the leader node confirmation information sent by the data replication monitoring platform within a preset time, it determines itself as a leader node and sends heartbeat information to the follower node. The preset time can be pre-set by relevant technical personnel according to actual needs.

[0083] The above technical solution determines its own cluster node as the leader node when it determines that the current time is the cluster startup time of the distributed cluster and receives the leader node confirmation information sent by the data replication monitoring platform within the preset time, and sends heartbeat information to the follower node, thereby realizing the selection of the leader node in the initial deployment stage, and continuously monitoring whether its own node is the leader node, and then sending heartbeat information to the follower node after determination, thereby improving the accuracy of the selection of the leader node and follower node in the initial stage, thereby facilitating subsequent data replication.

[0084] Example 3

[0085] This embodiment provides a preferred example based on the above embodiment. In order to further illustrate the improvements of the technical solution of the present invention, this embodiment uses the technical solution of the present invention and the existing technical solution to provide a detailed description.

[0086] Part I: Figure 3A The schematic diagram of the process structure of an existing data replication method is shown in FIG. The specific implementation process is as follows:

[0087] Step 1: The upstream target data source database cluster provides Binlog logs for data replication.

[0088] Step 2: The database cluster in the distributed data replication system is mainly used to store the replication points of the Binlog log and the performance data of the replica cluster.

[0089] Step 3: The replica in the replica cluster requests the changed log data in the target data source through the I / O thread based on the database internal communication protocol, and reads the update events in the log.

[0090] Step 4: The replicator reassembles the change log data into a JSON (JavaScript Object Notation) message and sends it to the downstream consumer service cluster for consumer service subscription.

[0091] Step 5: The distributed data replication monitoring platform monitors the replica machine's operating status every 10 minutes and implements a master-slave switchover when a failure of the primary replica machine is detected.

[0092] In the above existing distributed data replication system includes several key issues: first, when the primary replication machine failure, distributed data replication monitoring platform needs up to 10 minutes to perceive and implement the primary and backup switching, backup replication machine takeover replication services, this delay may lead to system during the primary replication machine failure service interruption, affecting the high availability of the system. Second, point information every 10s record to the database, in the process of resuming data consistency and accuracy.

[0093] The first part: as shown in a flow structure diagram of an improved data replication method provided by the embodiment, the specific implementation process is as follows: Figure 3B

[0094] Step 1: the replication machine (cluster node) in the distributed cluster starts to be a follower (Follow) node, when the node term does not receive the heartbeat content of the leader node or the log synchronization request, it will be converted into a candidate (Canadidate) node, and the term of its own node is added by 1, and then a new round of election is initiated, and the leader (Leader) node is elected.

[0095] In the initial state, the leader node can also be selected by the distributed data replication monitoring platform based on the node performance data of each cluster node obtained.

[0096] In the process of leader election, the candidate sends a voting request to other cluster nodes in the distributed cluster, and each cluster node can only vote for one candidate in the term. After receiving the voting request, the other cluster nodes respond to the candidate node with the voting result, and the candidate with more than half of the votes is elected as the leader node.

[0097] Among them, the election voting rule is: the log entry of the candidate node needs to be newer than the entry of the node itself; the term number of the candidate node needs to be greater than the term number of the node itself; vote for the candidate node that the voting request arrives first.

[0098] Step 2: after the leader node is elected, the other candidate nodes automatically become follower nodes. The leader node replicates the Binlog data changed in the target data source and sends it to the downstream consumption service cluster.

[0099] Step 3: the leader node synchronously encapsulates the replication point and its corresponding index and term number as a log entry and writes it into the local log file.

[0100] Step 4: the leader node sends a log replication request to all follower nodes, which carries new log entry information, including the pre-checkpoint (prevLogIndex) used to ensure log consistency. ​

[0101] Step 5: After receiving the log replication request, the follower node performs a strict consistency check, appends the log entry after passing it, and then sends a confirmation response to the leader node.

[0102] Step 6: When the leader node collects confirmation responses from the majority of follower nodes, it can be considered that the log entry has been safely replicated, and all follower nodes are notified to submit the corresponding log entry to achieve consistency of data replication points within the cluster nodes.

[0103] The above consistency check rules are as follows:

[0104] 1) Term number comparison: The follower node will check whether the term number in the log replication request is greater than or equal to its own current term number. If so, the follower node will update its own term number and accept the log entry; if not, the follower node will reject the log replication request, which may be an expired request.

[0105] 2) Log Continuity Verification: The follower node compares the previous log entry index (prevLogIndex) and term number specified in the RPC request with the corresponding entry in its own log. If the two match, it means that the follower node's log is consistent with the leader node's log at that location and log appending can continue. If they do not match, the follower node sends a response to the leader node containing the index and term number of the latest matching entry in its log and resends the log entries from that point on.

[0106] 3) Log entry index check: The follower node checks whether the index of the new log entry is greater than the index of the last entry in its own log. If not, it indicates that the leader's log may have been truncated at some point, and the follower node will ignore the log entry in the request.

[0107] The above-mentioned improved data replication methods based on the Raft protocol are all aimed at solving the problems of data consistency, fault recovery time, data duplication, and service interruption during the process of master database failure and backup database takeover. This solution solves the reliability and consistency problems of data replication by deploying a replica cluster. The cluster contains multiple nodes. Through the election mechanism in the Raft protocol, each node can act as a leader (Leader) or a follower (Follower). The leader node reads the data change record from the target data source and sends it to the downstream consumer service cluster. It asynchronously records the current replication site in the log entries of all follower nodes and sends heartbeat content. When the leader node fails, the Raft protocol quickly elects a new leader node service to achieve fast fault recovery and high availability. At the same time, the log entry mechanism in the Raft protocol ensures that the replication sites of all nodes are consistent, solving the problems of data duplication and service interruption in the traditional master-slave architecture.

[0108] The information collected in the technical solution of the present invention is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0109] Example 4

[0110] Figure 4 This is a structural diagram of a data replication device based on a distributed cluster provided by the fourth embodiment of the present invention. The data replication device based on a distributed cluster provided by the embodiment of the present invention is applicable to the case where the changed log data is replicated from the data source to the downstream consumer service cluster. The data replication device based on the distributed cluster can be implemented in the form of hardware and / or software, such as Figure 4 As shown, the device includes: a change data sending module 401, a point recording module 402, a copy request generating module 403 and a copy result determining module 404.

[0111] The change data sending module 401 is configured to obtain change log data from a target data source and send the change log data to a distributed message service cluster if the node of the cluster itself is determined to be a leader node.

[0112] A point recording module 402 is configured to record the current data replication point of the change log data after the change log data is sent;

[0113] A replication request generation module 403 is configured to generate a data replication request based on the current data replication point, and send the data replication request to a follower node, so that the follower node generates and feeds back a data confirmation response based on the data replication request; the follower node is a cluster node other than the leader node in the distributed cluster;

[0114] The replication result determination module 404 is configured to determine the data replication result according to the received data confirmation response sent by the follower node.

[0115] The technical solution of the embodiment of the present invention obtains change log data from the target data source when determining that its own cluster node is the leader node, sends the change log data to the distributed message service cluster, records the current data replication point of the change log data, generates a data replication request based on the current data replication point, and sends the data replication request to the follower node. According to the data confirmation response received from the follower node, the data replication result is determined, thereby achieving replication consistency in the data replication process, ensuring real-time and accurate synchronization of log point information across multiple nodes, improving data consistency, and enhancing the fault tolerance of the distributed cluster system. Even if some nodes fail, the integrity and continuity of data replication can be guaranteed. Moreover, when switching node failures, there is no need to consume a long time to obtain point data and node deployment, thereby avoiding service interruption for the downstream consumer service cluster. Data can be quickly located and continued to be replicated, thereby improving data replication efficiency.

[0116] Optionally, the device further includes:

[0117] An associated data sending module is used for, before obtaining the change log data from the target data source if the own cluster node is determined to be the leader node, sending the own term number and the own log entry index of the own cluster node to the candidate node when it is monitored that the own cluster node meets the judgment conditions of the candidate node, and at the same time, receiving the candidate term number and candidate log entry index sent by the candidate node; the candidate node is the other cluster node in the distributed cluster except the own cluster node;

[0118] A voting module, configured to obtain a first voting result generated by the candidate node based on its own term number and its own log entry index, and at the same time, generate a second voting result based on the candidate term number and candidate log entry index of the candidate node;

[0119] A leader node selection module is used to determine a leader node according to the first voting result and the second voting result.

[0120] Optional voting module, specifically used for:

[0121] Determine whether the candidate node meets the preset voting determination conditions based on the candidate term number and candidate log entry index of the candidate node;

[0122] If so, voting is performed for the candidate node to generate a second voting result.

[0123] Optionally, the voting granting judgment condition is:

[0124] Determine whether the candidate term number of the candidate node is not less than its own term number; and

[0125] Determine whether the candidate log entry index of the candidate node is greater than the own log entry index of the own cluster node.

[0126] Optionally, the device further includes:

[0127] A heartbeat information sending module is used to determine its own cluster node as a leader node and send heartbeat information to the follower node if the current time is the cluster startup time of the distributed cluster and the leader node confirmation information sent by the data replication monitoring platform is received within the preset time before obtaining the change log data from the target data source.

[0128] Optionally, the data replication monitoring platform generates leader node confirmation information in the following manner:

[0129] Obtaining node performance data of each cluster node in the distributed cluster by a data replication monitoring platform;

[0130] Selecting a target cluster node based on the node performance data of each cluster node;

[0131] The target cluster node is determined as a leader node, and leader node confirmation information is generated and sent to the target cluster node.

[0132] The distributed cluster-based data replication device provided in the embodiment of the present invention can execute the distributed cluster-based data replication method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0133] Example 5

[0134] Figure 5A structural diagram of an electronic device 50 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0135] As shown in Figure 5 The electronic device 50 includes at least one processor 51, and a memory, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., connected to the at least one processor 51, where the memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 52 or loaded into the random access memory (RAM) 53 from the storage unit 58. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other through a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0136] Various components in the electronic device 50 are connected to the I / O interface 55, including an input unit 56, such as a keyboard, a mouse, etc., an output unit 57, such as various types of displays, a speaker, etc., a storage unit 58, such as a magnetic disk, an optical disk, etc., and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0137] The processor 51 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 performs various methods and processes described above, such as the distributed cluster-based data replication method.

[0138] In some embodiments, the distributed cluster-based data replication method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the distributed cluster-based data replication method described above can be performed. Alternatively, in other embodiments, processor 51 can be configured to perform the distributed cluster-based data replication method in any other appropriate manner (e.g., by means of firmware).

[0139] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0143] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0144] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0145] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0146] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data replication method based on a distributed cluster, characterized in that: Applicable to any cluster node in a distributed cluster, including: If it is determined that its own cluster node is the leader node, then obtaining the change log data from the target data source and sending the change log data to the distributed message service cluster; After the change log data is sent, recording the current data copy point of the change log data; Generate a data replication request according to the current data replication point, and send the data replication request to a follower node, so that the follower node generates and feeds back a data confirmation response according to the data replication request; the follower node is a cluster node other than the leader node in the distributed cluster; The data replication result is determined according to the received data confirmation response sent by the follower node.

2. The method according to claim 1, characterized in that Before obtaining the change log data from the target data source if the own cluster node is determined to be the leader node, the method further includes: When it is detected that its own cluster node meets the judgment conditions of the candidate node, it sends its own term number and its own log entry index to the candidate node, and at the same time, receives the candidate term number and candidate log entry index sent by the candidate node; the candidate node is the other cluster node in the distributed cluster except its own cluster node; Obtain a first voting result generated by the candidate node based on its own term number and its own log entry index, and at the same time, generate a second voting result based on the candidate term number and candidate log entry index of the candidate node; A leader node is determined according to the first voting result and the second voting result.

3. The method according to claim 2, characterized in that Generating a second voting result according to the candidate term number and the candidate log entry index of the candidate node includes: Determine whether the candidate node meets the preset voting determination conditions based on the candidate term number and candidate log entry index of the candidate node; If so, voting is performed for the candidate node to generate a second voting result.

4. The method according to claim 3, characterized in that The voting conditions are as follows: Determine whether the candidate term number of the candidate node is not less than its own term number; and Determine whether the candidate log entry index of the candidate node is greater than the own log entry index of the own cluster node.

5. The method according to claim 1, wherein Before obtaining the change log data from the target data source if the own cluster node is determined to be the leader node, the method further includes: If the current time is the cluster startup time of the distributed cluster and the leader node confirmation information sent by the data replication monitoring platform is received within the preset time, the cluster node itself is determined as the leader node and a heartbeat message is sent to the follower node.

6. The method according to claim 5, characterized in that The data replication monitoring platform generates leader node confirmation information in the following manner: Obtaining node performance data of each cluster node in the distributed cluster by a data replication monitoring platform; Selecting a target cluster node based on the node performance data of each cluster node; The target cluster node is determined as a leader node, and leader node confirmation information is generated and sent to the target cluster node.

7. A data replication device based on a distributed cluster, characterized in that: Configured on any cluster node in a distributed cluster, including: A change data sending module, configured to obtain change log data from a target data source and send the change log data to a distributed message service cluster if the node of the cluster itself is determined to be a leader node; A point recording module, configured to record the current data copy point of the change log data after the change log data is sent; a replication request generation module, configured to generate a data replication request based on the current data replication point, and send the data replication request to a follower node, so that the follower node generates and feeds back a data confirmation response based on the data replication request; the follower node is a cluster node other than the leader node in the distributed cluster; The replication result determination module is used to determine the data replication result according to the received data confirmation response sent by the follower node.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the distributed cluster-based data replication method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed cluster-based data replication method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the distributed cluster-based data replication method according to any one of claims 1 to 6.

Images