Zookeeper data synchronization method and device

Abstract

The invention relates to the technical field of data synchronization, and discloses a Zookeeper data synchronization method and device, and the method comprises the steps: carrying out the full-amount reading of the data of a Zookeeper cluster of a production center, and obtaining a full-amount change set based on a previous full-amount compensation site and the full-amount read data; monitoring an event occurring in a Zookeeper cluster of a production center, and performing incremental reading on the monitored change event to obtain incremental buffer data; merging the total change set and the incremental buffer data, and loading the merged data to a data queue; and delivering the data in the data queue to a Zookeeper cluster of a disaster recovery center. According to the method, the disaster recovery RPO and the disaster recovery RTO can be reduced at the same time, in the process of synchronizing the data to the disaster recovery center, connection configuration does not need to be restarted or modified, the data synchronization process is smooth, and the synchronization effect of Zookeeper cluster remote disaster recovery is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a Zookeeper data synchronization method and device. Background Art

[0002] Zookeeper is a distributed, open source distributed application coordination service. It is an open source implementation of Google Chubby and is widely used in many open source components, such as Kafka, Hadoop, Dubbo, etc. Zookeeper distributed cluster service consists of a group of hosts. In the production environment, it is usually composed of more than 3 hosts to form a highly available consistent distributed system. The client program will choose to create a TCP connection with any server in the cluster, and the client can connect to other servers after the server is disconnected. The core principle of the architectural design to ensure the high availability of the system is the redundancy of components and the automatic transfer of faults. In addition to achieving redundancy by forming a cluster with multiple nodes, it is also necessary to consider multiple factors such as disaster recovery in multiple computer rooms and multiple regions (in the same city and in different places) to ensure that the service can operate normally from a single node to a data center failure or even in the case of regional unavailability caused by natural disasters.

[0003] Currently, there are generally two ways to achieve disaster recovery in a Zookeeper cluster:

[0004] One is to build a cross-computer room cluster and use the cluster's high availability mechanism for disaster recovery, but there are problems such as high deployment cost, poor synchronization performance, and uneven recovery.

[0005] The second is to build an independent cluster and use data synchronization between clusters for disaster recovery. Generally, there are two ways to synchronize the metadata files and the logical way to synchronize the znode data. There is no mature and easy-to-use continuous replication solution in the industry for the logical synchronization method of replicating node data. The physical synchronization method of replicating metadata files depends on a certain operation sequence, which makes it difficult to achieve real-time synchronization. The recovery process is not smooth, and the recovery point objective (RPO) and recovery time objective (RTO) of disaster recovery data are large. Summary of the invention

[0006] In view of this, the present invention provides a Zookeeper data synchronization method and device to solve the technical problem of poor synchronization effect of the existing Zookeeper cluster remote disaster recovery.

[0007] In a first aspect, the present invention provides a Zookeeper data synchronization method, comprising: fully reading the data of the Zookeeper cluster of the production center, and obtaining a full change set based on the previous full compensation site and the fully read data; monitoring events occurring in the Zookeeper cluster of the production center, and performing incremental reading of the monitored change events to obtain incremental buffered data; merging the full change set and the incremental buffered data and loading them into a data queue; and delivering the data in the data queue to the Zookeeper cluster of the disaster recovery center.

[0008] The Zookeeper data synchronization method of the present invention performs full reading of the data of the Zookeeper cluster of the production center, obtains a full change set based on the last full compensation site and the full read data, monitors the events occurring in the Zookeeper cluster of the production center, performs incremental reading of the monitored change events, obtains incremental buffer data, merges the full change set and the incremental buffer data and loads them into a data queue, delivers the data in the data queue to the Zookeeper cluster of the disaster recovery center, realizes the full reading and incremental monitoring reading process of the data local cache interleaving operation, builds the node data change flow from the Zookeeper cluster of the production center to the Zookeeper cluster of the disaster recovery center, realizes near real-time RPO, can reduce the disaster recovery RPO and RTO at the same time, and in the process of synchronizing the data to the disaster recovery center, it is not necessary to restart or modify the connection configuration, the data synchronization process is smooth, the synchronization effect of the remote disaster recovery of the Zookeeper cluster is improved, and it is not necessary to build multiple cross-computer room clusters, and the deployment cost is low.

[0009] Optionally, the data of the Zookeeper cluster in the production center is fully read, and a full change set is obtained based on the last full compensation site and the fully read data, including: periodically or when the data synchronization process is interrupted and restored, the data of the Zookeeper cluster in the production center is traversed using a tree structure; the local tree cache data is updated after comparing the version and site of the traversed data with the local tree cache data of the last full compensation site to obtain a full change set.

[0010] In this way, by traversing the tree structure and comparing version sites, the full set of changes can be obtained more accurately, improving the accuracy and efficiency of data synchronization.

[0011] Optionally, monitor the events occurring in the Zookeeper cluster of the production center, and perform incremental reading of the monitored change events to obtain incremental buffered data, including: monitor the events occurring in the Zookeeper cluster of the production center, and query the change nodes corresponding to the monitored change events; compare the data of the change nodes with the corresponding node data in the local cache by version and location, and then update the local cache to obtain incremental buffered data.

[0012] In this way, data changes can be captured in real time through monitoring and comparison, and the local cache can be updated in a timely manner to ensure the real-time and consistency of the data. Data can be quickly and smoothly synchronized and restored after synchronization is interrupted.

[0013] Optionally, the data in the data queue is delivered to the Zookeeper cluster of the disaster recovery center, including: comparing the data of the Zookeeper cluster of the production center with the data of the current local cache to determine whether the current local cache is expired; if the current local cache is not expired, deleting, adding or updating the nodes of the Zookeeper cluster of the disaster recovery center according to the data in the data queue.

[0014] In this way, by comparing and judging whether the local cache is expired, the synchronization of outdated data can be avoided, thereby improving the timeliness and accuracy of data synchronization.

[0015] Optionally, after deleting, adding or updating the nodes of the Zookeeper cluster of the disaster recovery center according to the data in the data queue, it also includes: judging whether a deletion conflict, a addition conflict or an update conflict occurs in the process of deleting, adding or updating the nodes of the Zookeeper cluster of the disaster recovery center; if a deletion conflict occurs, the deletion operation is skipped; if a addition conflict occurs, the addition operation is adjusted to an update operation; if an update conflict occurs, the update operation is adjusted to a addition operation.

[0016] In this way, conflicts in the data synchronization process can be effectively resolved through conflict detection and corresponding processing strategies, thereby improving the stability and reliability of data synchronization.

[0017] Optionally, before merging the full change set and the incremental buffer data and loading them into the data queue, the process includes: merging the full change set and the incremental buffer data, and compressing and encrypting the merged data; correspondingly, before delivering the data in the data queue to the Zookeeper cluster of the disaster recovery center, the process includes: decompressing and decrypting the data in the data queue.

[0018] Compression and encryption can reduce the amount of data transmission, improve transmission efficiency, and increase data security during transmission to prevent data leakage.

[0019] In the second aspect, the present invention provides a synchronization device, including: a full reading module, which is used to read the data of the Zookeeper cluster of the production center in full, and obtain a full change set based on the previous full compensation site and the full read data; an incremental reading module, which is used to monitor events occurring in the Zookeeper cluster of the production center, and perform incremental reading of the monitored change events to obtain incremental buffered data; a data loading module, which is used to merge the full change set and the incremental buffered data and load them into the data queue; a data delivery module, which is used to deliver the data in the data queue to the Zookeeper cluster of the disaster recovery center.

[0020] In the third aspect, the present invention provides a synchronization system, comprising: a management device and the synchronization device described in the second aspect of the present invention, the management device is used to create data synchronization tasks and issue synchronization tasks to the synchronization device, and obtain and output status data and task data reported by the synchronization device.

[0021] In a fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the Zookeeper data synchronization method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0022] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the Zookeeper data synchronization method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flowchart of the Zookeeper data synchronization method according to an embodiment of the present invention;

[0025] Figure 2 is a structural block diagram of a synchronization device according to an embodiment of the present invention;

[0026] Figure 3 is a structural block diagram of a synchronization system according to an embodiment of the present invention;

[0027] Figure 4It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] The terms involved in the embodiments of the present invention are explained as follows:

[0030] RPO (Recovery Point Objective): It refers to the amount of data loss that the business system can tolerate, that is, the time point corresponding to the recovered data when the business system restores its service function.

[0031] RTO (Recovery Time Objective): Recovery time objective, mainly refers to the longest tolerable service outage time, that is, the shortest time period from the occurrence of a disaster to the restoration of service functions of the business system.

[0032] Chubby: Chubby is a file system designed by Google that provides coarse-grained locking services. It is based on a loosely coupled distributed system and solves the problem of distributed consistency.

[0033] Kafka: Kafka is an open source stream processing platform developed by the Apache Software Foundation. It is written in Scala and Java and is a high-throughput distributed publish-subscribe messaging system.

[0034] Hadoop: Hadoop is a distributed system infrastructure developed by the Apache Foundation that implements a distributed file system.

[0035] Dubbo: Dubbo is an open-source, high-performance service framework developed by Alibaba. It enables applications to implement service output and input functions through high-performance RPC and can be seamlessly integrated with the Spring framework.

[0036] zkcopy: zkcopy is a simple data replication tool between zookeepers that can quickly copy data from the source cluster to the target cluster.

[0037] Leader: A Zookeeper cluster node role. The leader is responsible for initiating and resolving votes and updating the system status.

[0038] Observer: A node in the Zookeeper cluster. The observer does not participate in the voting process and only synchronizes the Leader status.

[0039] znode: The Zookeeper namespace has a tree-like memory model, in which each node is called a znode.

[0040] Watch: Zookeeper clients can subscribe through the Watch mechanism to receive corresponding notifications when the data or status of a node on the server changes.

[0041] ETL: It is the abbreviation of Extract-Transform-Load, which is used to describe the process of extracting (extract), transforming (transform), and loading (load) data from the source end to the destination end.

[0042] Currently, there are generally two ways to achieve disaster recovery in a Zookeeper cluster:

[0043] (1) Build a cross-computer room cluster and use the cluster high availability mechanism for disaster recovery, but there are problems such as deployment cost, synchronization performance and uneven recovery: (A) Dual computer rooms cannot guarantee availability (more than half of the nodes need to work properly), and at least three computer rooms are deployed, with no more than half of the nodes in each computer room; (B) The biggest problem with building clusters in different locations is network latency, which leads to a long decision time for cluster voting nodes, affecting cluster processing performance. After the Observer node is started, it synchronizes data from the Leader and does not participate in elections and voting, and has no impact on the transactional requests and elections of the cluster. Using the production center cluster deployment and the Observer node method in other centers can optimize the disaster recovery reading performance in different locations to a certain extent, but the production center cannot recover quickly after a failure, and the node roles of other centers need to be adjusted; (C) Zookeeper cluster information is configured in the configuration file of each node. The method of building a unified cluster is not smooth when nodes are changed and restored, which affects the cluster configuration modification and restart of running nodes and the connection configuration modification and reconnection of clients.

[0044] (2) Build an independent cluster and use data synchronization between clusters for disaster recovery. Generally, there are two methods: physical synchronization of replicating metadata files and logical synchronization of replicating znode node data. Among them, there is no mature and easy-to-use continuous replication solution in the industry for the logical synchronization of replicating node data. The physical synchronization of replicating metadata files depends on a certain operation sequence, which makes it difficult to achieve real-time synchronization. The recovery process is not smooth, and the disaster recovery RPO and RTO are large. The main steps are: (A) Start a new cluster and automatically create a data directory; (B) Stop the new cluster and clean up the metadata files in the data directory of each node; (C) Find the original cluster leader node and copy the metadata files in the data directory of the node; (D) Distribute the copied metadata files to the data directories corresponding to each node in the new cluster; (E) Restart the new cluster.

[0045] In order to solve at least one of the above problems, an embodiment of the present invention provides a Zookeeper data synchronization method.

[0046] According to an embodiment of the present invention, a Zookeeper data synchronization method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] In this embodiment, a Zookeeper data synchronization method is provided, which can be used in intelligent terminals, such as synchronization devices in production centers or disaster recovery centers, and the synchronization work of the synchronization devices is managed by the customs management device. Figure 2 As shown in the figure, the synchronization device is deployed in the production center and the disaster recovery center, and the management device is deployed in other intelligent terminals. The main functions of the synchronization device and the management device are as follows:

[0048] The management device is responsible for creating data synchronization tasks, generating instruction templates, issuing task instructions to synchronization devices, and managing synchronization device status and task data.

[0049] The synchronization device has the ability to interact with the management device, report local status and task data, accept task instructions from the management device, and complete data collection and transmission. It implements the following functional modules and supports continuous data replication:

[0050] 1. TreeCache: implements local snapshots of Zookeeper cluster data and manages data and versions;

[0051] 2.FullExtractor: Extracts full data through tree traversal;

[0052] 3.IncrementExtractor: Extract incremental data through event monitoring;

[0053] 4.DataTransformer: implements data conversion, converting the full-extracted and incremental-extracted data into the data to be loaded;

[0054] 5.DataQueue: implements the queue of data to be loaded;

[0055] 6.DataLoader: implements data loading and delivers data to the target cluster.

[0056] like Figure 1 As shown, the Zookeeper data synchronization method of the embodiment of the present invention includes the following steps:

[0057] Step S101, fully read the data of the Zookeeper cluster of the production center, and obtain a full change set based on the last full compensation point and the full read data.

[0058] Specifically, the synchronization task issued by the receiving management device will monitor the incremental event when the synchronization task is started. If the synchronization task is interrupted for any reason and then restarted, a full read will be performed because the incremental data is based on events rather than logs and data must be compensated.

[0059] In addition, full reading can be performed periodically.

[0060] The current point is recorded each time the full data is read. When dealing with the problems of discontinuous native Watch monitoring and loss of interruption events, it is necessary to rebuild the local snapshot, compare the current full data read with the data of the previous full compensation point, filter the full extraction data to generate a full change set.

[0061] Step S102, monitor the events occurring in the Zookeeper cluster of the production center, and perform incremental reading on the monitored change events to obtain incremental buffer data.

[0062] Specifically, Zookeeper's native Watch is used to monitor node events in the production center, Zookeeper's cross-cluster data synchronization streaming monitoring of data changes is implemented, and pull / ack support similar to MySQL Binlog is provided.

[0063] After the synchronization task is interrupted and restarted, full reading is performed while event monitoring is performed. After the event is monitored, incremental reading is performed on the incremental events of the production center. Incremental reading only needs to obtain the data content that has changed in the production center without having to read all the data in the production center.

[0064] Step S103: merge the full change set and the incremental buffer data and load them into the data queue.

[0065] The full change set and incremental buffer data are merged by the local TreeCache to achieve data conversion and prevent version / time conflicts.

[0066] Manage data through local snapshots of Zookeeper cluster data, merge full and incremental extracted data according to version and time rules, generate change sets and push them to downstream data queues, implement push-pull delivery mode through data queues, and use change time as the location.

[0067] Step S104: delivering the data in the data queue to the Zookeeper cluster of the disaster recovery center.

[0068] The Zookeeper clusters in the disaster recovery center and the production center both have a tree-structured memory model, where each node is called a znode. Data in the data queue is synchronized to the corresponding node in the Zookeeper cluster of the disaster recovery center to achieve data synchronization.

[0069] The Zookeeper data synchronization method of the present invention performs full reading of the data of the Zookeeper cluster of the production center, obtains a full change set based on the last full compensation site and the full read data, monitors the events occurring in the Zookeeper cluster of the production center, performs incremental reading of the monitored change events, obtains incremental buffer data, merges the full change set and the incremental buffer data and loads them into a data queue, delivers the data in the data queue to the Zookeeper cluster of the disaster recovery center, realizes the full reading and incremental monitoring reading process of the data local cache interleaving operation, builds the node data change flow from the Zookeeper cluster of the production center to the Zookeeper cluster of the disaster recovery center, realizes near real-time RPO, can reduce the disaster recovery RPO and RTO at the same time, and in the process of synchronizing the data to the disaster recovery center, it is not necessary to restart or modify the connection configuration, the data synchronization process is smooth, the synchronization effect of the remote disaster recovery of the Zookeeper cluster is improved, and it is not necessary to build multiple cross-computer room clusters, and the deployment cost is low.

[0070] In some embodiments, step S101, fully read the data of the Zookeeper cluster of the production center, and obtain a full change set based on the last full compensation site and the fully read data, including:

[0071] Step S1011, periodically or when the data synchronization process is interrupted and restored, the data of the Zookeeper cluster of the production center is traversed and read using a tree structure.

[0072] Specifically, the synchronization device periodically extracts full data and performs full compensation after a certain time interval, or extracts full data when reconnecting after synchronization is interrupted.

[0073] Step S1012, compare the traversed data with the local tree cache data of the last full compensation site in terms of version and site, and then update the local tree cache data to obtain a full change set.

[0074] Each time the local tree cache data is updated, the changed point and the current version are recorded and reported to the management device for storage. The data read in full this time is compared with the local tree cache data of the previous version, the old data is discarded, and the full data is put into the local cache to obtain the full change set.

[0075] In this way, by traversing the tree structure and comparing version sites, the full set of changes can be obtained more accurately, improving the accuracy and efficiency of data synchronization.

[0076] In some embodiments, step S102 monitors events occurring in the Zookeeper cluster of the production center, and performs incremental reading of the monitored change events to obtain incremental buffer data, including:

[0077] Step S1021, monitor the events occurring in the Zookeeper cluster of the production center, and query the change nodes corresponding to the monitored change events.

[0078] Specifically, when a synchronization task is received, the native Watch is used to monitor data changes in the production center and find the changed nodes.

[0079] Step S1022, compare the data of the changed node with the corresponding node data in the local cache in terms of version and location, and then update the local cache to obtain incremental buffer data.

[0080] Find the location of the changed node in the local tree cache data, compare the data read incrementally this time with the local tree cache data of the previous version, discard the old data, and put the newly read incremental data into the local cache to obtain the incremental buffer data.

[0081] In this way, data changes can be captured in real time through monitoring and comparison, and the local cache can be updated in a timely manner to ensure the real-time and consistency of the data. Data can be quickly and smoothly synchronized and restored after synchronization is interrupted.

[0082] In some embodiments, step S104, delivering the data in the data queue to the Zookeeper cluster of the disaster recovery center, includes:

[0083] Step S1041, compare the data of the Zookeeper cluster of the production center with the data of the current local cache to determine whether the current local cache is expired.

[0084] Specifically, whether the current local cache is expired is determined by comparing the version and modification time of the current local cache znode node with the version and modification time of the data of the Zookeeper cluster of the production center currently being read. If the last repair time of the current local cache version is the same as the data of the Zookeeper cluster of the production center currently being read, the current local cache is not expired. Otherwise, the current local cache is expired.

[0085] By comparing and determining whether the local cache is expired, the synchronization of outdated data can be avoided, thereby improving the timeliness and accuracy of data synchronization.

[0086] Step S1042: If the current local cache is not expired, the nodes of the Zookeeper cluster of the disaster recovery center are deleted, added, or updated according to the data in the data queue.

[0087] Specifically, the synchronization device processes the data in the queue to be loaded and performs corresponding processing according to the processing type. If the processed data is deleted, the current local cache is compared, and if it is not expired, the disaster recovery center node is deleted; if the processed data is newly added, the current local cache is compared, and if it is not expired, a new disaster recovery center node is added; if the processed data is updated, the current local cache is compared, and if it is not expired, the disaster recovery center node is updated.

[0088] Further, in step S1042, after deleting, adding or updating the nodes of the Zookeeper cluster of the disaster recovery center according to the data in the data queue, the method further includes:

[0089] Step a1, determining whether a deletion conflict, a addition conflict or an update conflict occurs during the process of deleting, adding or updating a node of the Zookeeper cluster of the disaster recovery center.

[0090] Step a2: if a deletion conflict occurs, the deletion operation is skipped; if a new addition conflict occurs, the new addition operation is adjusted to an update operation; if an update conflict occurs, the update operation is adjusted to a new addition operation.

[0091] Specifically, in the process of data processing, corresponding adjustments are made according to possible deletion conflicts, addition conflicts, and update conflicts. If a deletion conflict occurs, it is considered that the disaster recovery center node does not exist and the deletion operation is skipped; if a addition conflict occurs, it is considered that the disaster recovery center node already exists, and the addition operation is adjusted to an update operation; if an update conflict occurs, it is considered that the disaster recovery center node does not exist, and the update operation is adjusted to a addition operation.

[0092] In this way, conflicts in the data synchronization process can be effectively resolved through conflict detection and corresponding processing strategies, thereby improving the stability and reliability of data synchronization.

[0093] In some embodiments, in step S103, merging the full change set and the incremental buffered data and loading them into the data queue includes:

[0094] Step b1, merge the full change set and the incremental buffer data, and compress and encrypt the merged data.

[0095] Correspondingly, before delivering the data in the data queue to the Zookeeper cluster of the disaster recovery center in step S104, the following steps are included:

[0096] Step c1, decompressing and decrypting the data in the data queue.

[0097] Compression and encryption can reduce the amount of data transmission, improve transmission efficiency, and increase data security during transmission to prevent data leakage.

[0098] A Zookeeper data synchronization method according to an embodiment of the present invention has the following beneficial effects:

[0099] By fully reading the data of the Zookeeper cluster in the production center, and obtaining the full change set based on the previous full compensation site and the full read data, the events occurring in the Zookeeper cluster of the production center are monitored, and the monitored change events are incrementally read to obtain incremental buffered data. The full change set and incremental buffered data are merged and loaded into the data queue, and the data in the data queue is delivered to the Zookeeper cluster in the disaster recovery center, so as to realize the full read and incremental monitoring and reading process of the data local cache interleaving operation, and build the node data change flow from the Zookeeper cluster of the production center to the Zookeeper cluster of the disaster recovery center, effectively reducing the difficulty of implementing Zookeeper data migration and Zookeeper cluster disaster recovery, and reducing the disaster recovery RPO and RTO. In the process of synchronizing data to the disaster recovery center, there is no need to restart or modify the connection configuration, and the data synchronization process is smooth, which improves the synchronization effect of the Zookeeper cluster disaster recovery, and does not need to build multiple cross-computer room clusters, so the deployment cost is low.

[0100] Combining Zookeeper's native Watch mechanism and the full traversal process, incremental data and full data can be merged to ensure the reliability and efficiency of data synchronization.

[0101] The data is extracted, transformed, and loaded to the destination through the ETL process in the production center to achieve data synchronization process, support Extract / Transform / Load node expansion and separate deployment, and achieve pluggable capabilities such as data compression and transmission encryption, which facilitates real-time synchronization of Zookeeper cluster data at the production site to a remote read-only cluster or shadow cluster. In actual application scenarios, Zookeeper namespace isolation, cluster data bidirectional synchronization, and data loopback control can also be used to build a Zookeeper active-active cluster.

[0102] The embodiment of the present invention also provides a synchronization device, such as Figure 3 As shown, including:

[0103] The full read module 301 is used to read the data of the Zookeeper cluster of the production center in full, and obtain a full change set based on the last full compensation point and the full read data;

[0104] The incremental reading module 302 is used to monitor the events occurring in the Zookeeper cluster of the production center, and perform incremental reading on the monitored change events to obtain incremental buffer data;

[0105] The data loading module 303 is used to merge the full change set and the incremental buffer data and load them into the data queue;

[0106] The data delivery module 304 is used to deliver the data in the data queue to the Zookeeper cluster of the disaster recovery center.

[0107] The Zookeeper data synchronization device of the embodiment of the present invention fully reads the data of the Zookeeper cluster of the production center, obtains a full change set based on the last full compensation site and the fully read data, monitors the events occurring in the Zookeeper cluster of the production center, and performs incremental reading on the monitored change events to obtain incremental buffered data, merges the full change set and the incremental buffered data and loads them into a data queue, delivers the data in the data queue to the Zookeeper cluster of the disaster recovery center, implements the full read and incremental monitoring read process of staggered operation of the local data cache, builds a node data change flow from the Zookeeper cluster of the production center to the Zookeeper cluster of the disaster recovery center, implements near real-time RPO, can reduce disaster recovery RPO and RTO at the same time, and in the process of synchronizing data to the disaster recovery center, there is no need to restart or modify the connection configuration, the data synchronization process is smooth, the synchronization effect of the remote disaster recovery of the Zookeeper cluster is improved, and there is no need to build multiple cross-computer room clusters, and the deployment cost is low.

[0108] Furthermore, the full reading module 301 includes:

[0109] The full traversal module is used to traverse and read the data of the Zookeeper cluster in the production center periodically or when the data synchronization process is interrupted and restored.

[0110] The full data acquisition module is used to update the local tree cache data after comparing the version and location of the traversed data with the local tree cache data of the previous full compensation location to obtain the full change set.

[0111] Furthermore, the incremental reading module 302 includes:

[0112] The event monitoring module is used to monitor the events occurring in the Zookeeper cluster of the production center and query the change nodes corresponding to the monitored change events;

[0113] The incremental data acquisition module is used to compare the data of the changed node with the corresponding node data in the local cache in terms of version and location, and then update the local cache to obtain incremental buffer data.

[0114] Furthermore, the data loading module 303 includes:

[0115] The expiration judgment module compares the data of the Zookeeper cluster in the production center with the data in the current local cache to determine whether the current local cache is expired;

[0116] The data processing module is used to delete, add or update the nodes of the Zookeeper cluster of the disaster recovery center according to the data in the data queue if the current local cache has not expired.

[0117] The data processing module also includes:

[0118] The conflict judgment module is used to judge whether a deletion conflict, addition conflict or update conflict occurs during the process of deleting, adding or updating nodes in the Zookeeper cluster of the disaster recovery center;

[0119] The conflict handling module is used to skip the deletion operation if a deletion conflict occurs, adjust the addition operation to an update operation if a addition conflict occurs, and adjust the update operation to a addition operation if an update conflict occurs.

[0120] Furthermore, the data loading module 303 includes:

[0121] The compression and encryption module is used to merge the full change set and the incremental buffer data, and compress and encrypt the merged data;

[0122] Correspondingly, the data delivery module 304 includes:

[0123] The decompression and decryption module is used to decompress and decrypt the data in the data queue.

[0124] The embodiment of the present invention also provides a synchronization system, such as Figure 2 As shown, it includes: a management device and a synchronization device as in the above embodiment, the management device is used to create a data synchronization task and send the synchronization task to the synchronization device, and obtain and output the status data and task data reported by the synchronization device.

[0125] Specifically, taking the two clusters of the production center and the disaster recovery center as examples, the steps and technical solutions included in the synchronization method and device provided by the present invention are explained:

[0126] 1. Install the synchronization device on the designated machine that communicates with the two centers;

[0127] 2. Create a data synchronization task on the management device;

[0128] 2.1. Configure cluster access information for the production center and disaster recovery center;

[0129] 2.2. Configure user authorization information for the production center and disaster recovery center;

[0130] 2.3. Configure the root path set that needs to be synchronized;

[0131] 3. Start the synchronization task on the management device;

[0132] 3.1. The management device generates a template instruction;

[0133] 3.2. The management device sends a start command to the synchronization device;

[0134] 4. The synchronization device performs task initialization;

[0135] 4.1. Create the Zookeeper client of the disaster recovery center;

[0136] 4.2. Create the production center Zookeeper client;

[0137] 4.3. Initialize local cache TreeCache;

[0138] 4.4. Monitor data changes in the production center;

[0139] 5. The synchronization device starts the full reading process;

[0140] 5.1. Tree traversal root path node;

[0141] 5.2. Try to put the data into the local cache, compare the data version and location, and discard the old data;

[0142] 5.3. After the data is put into the local cache, it is pushed to the queue to be loaded;

[0143] 6. The synchronization device handles the incremental change event;

[0144] 6.1. Query the changed nodes;

[0145] 6.2. Try to put the data into the local cache, compare the data version and location, and discard the old data;

[0146] 6.3. After the data is put into the local cache, it is pushed to the queue to be loaded;

[0147] 7. Synchronous device processing data compensation;

[0148] 7.1. The production center client reconnects at certain intervals to make full compensation;

[0149] 7.2. Carry out periodic full compensation;

[0150] 7.3. Perform full compensation after interruption and reconnection;

[0151] 8. The synchronization device processes the queue data to be loaded;

[0152] 8.1. Processing data deletion: Compare the latest local cache, and delete the disaster recovery center node if it is not expired;

[0153] 8.2. Processing data addition: Compare the latest local cache, and if it is not expired, add a disaster recovery center node;

[0154] 8.3. Process data updates: compare the latest local cache, and update the disaster recovery center node if it is not expired;

[0155] 9. Synchronous device number processing data loading;

[0156] 9.1. Handling deletion conflicts: If the disaster recovery center node does not exist, skip it;

[0157] 9.2. Handling new conflicts: If the disaster recovery center node already exists, adjust it to an update operation;

[0158] 9.3. Handling update conflicts: If the disaster recovery center node does not exist, adjust it to a new operation;

[0159] 10. The synchronization device periodically reports status and task data to the management device.

[0160] The synchronization system of the embodiment of the present invention realizes automated task process management through a management device and a synchronization device. The management device is responsible for creating synchronization tasks, dispatching task instructions, managing the status of the synchronization device, etc., which can improve the efficiency of data synchronization management and effectively reduce the difficulty of configuring the synchronization device and the difficulty of task management.

[0161] The embodiment of the present invention also provides a schematic diagram of the structure of a computer device, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0162] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0163] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0164] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0166] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0167] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0168] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0169] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0170] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope of protection.

Claims

1. A Zookeeper data synchronization method, characterized in that: include: Fully read the data of the Zookeeper cluster in the production center, and obtain the full change set based on the previous full compensation point and the full read data; Monitor the events occurring in the Zookeeper cluster of the production center, and perform incremental reading of the monitored change events to obtain incremental buffer data; Merge the full change set and incremental buffer data and load them into the data queue; Deliver the data in the data queue to the Zookeeper cluster in the disaster recovery center.

2. The Zookeeper data synchronization method according to claim 1, characterized in that: The data of the Zookeeper cluster in the production center is fully read, and the full change set is obtained based on the previous full compensation point and the full read data, including: Periodically or when the data synchronization process is interrupted and restored, the data of the Zookeeper cluster in the production center is traversed using a tree structure; The traversed data is compared with the local tree cache data of the previous full compensation site in terms of version and site, and then the local tree cache data is updated to obtain the full change set.

3. The Zookeeper data synchronization method according to claim 1, characterized in that: Monitor the events of the Zookeeper cluster in the production center and perform incremental reading of the monitored change events to obtain incremental buffer data, including: Monitor the events occurring in the Zookeeper cluster of the production center and query the change nodes corresponding to the monitored change events; After comparing the version and location of the data of the changed node with the corresponding node data in the local cache, the local cache is updated to obtain the incremental buffer data.

4. The Zookeeper data synchronization method according to claim 1, characterized in that: Deliver the data in the data queue to the Zookeeper cluster in the disaster recovery center, including: Compare the data of the Zookeeper cluster in the production center with the data in the current local cache to determine whether the current local cache is out of date; If the current local cache has not expired, the nodes of the Zookeeper cluster in the disaster recovery center are deleted, added, or updated according to the data in the data queue.

5. The Zookeeper data synchronization method according to claim 4, characterized in that: After deleting, adding or updating the nodes of the Zookeeper cluster in the disaster recovery center according to the data in the data queue, it also includes: Determine whether deletion conflicts, addition conflicts, or update conflicts occur during the process of deleting, adding, or updating nodes in the Zookeeper cluster of the disaster recovery center; If a deletion conflict occurs, the deletion operation is skipped. If a addition conflict occurs, the addition operation is adjusted to an update operation. If an update conflict occurs, the update operation is adjusted to a addition operation.

6. The Zookeeper data synchronization method according to claim 1, characterized in that: Before merging the full change set and incremental buffered data and loading them into the data queue, the following steps are performed: Merge the full change set and the incremental buffer data, and compress and encrypt the merged data; Correspondingly, before delivering the data in the data queue to the Zookeeper cluster in the disaster recovery center, the following steps are required: Decompress and decrypt the data in the data queue.

7. A synchronization device, characterized in that: include: The full read module is used to read the data of the Zookeeper cluster in the production center in full, and obtain the full change set based on the previous full compensation point and the full read data; The incremental reading module is used to monitor the events occurring in the Zookeeper cluster of the production center and perform incremental reading on the monitored change events to obtain incremental buffer data. The data loading module is used to merge the full change set and the incremental buffer data and load them into the data queue; The data delivery module is used to deliver the data in the data queue to the Zookeeper cluster in the disaster recovery center.

8. A synchronization system, characterized in that: include: A management device and a synchronization device as described in claim 7, wherein the management device is used to create a data synchronization task and issue the synchronization task to the synchronization device, and obtain and output status data and task data reported by the synchronization device.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the Zookeeper data synchronization method according to any one of claims 1 to 6 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the Zookeeper data synchronization method according to any one of claims 1 to 6.

Images