Data cross-cluster synchronization methods, electronic devices and storage media
By combining full synchronization with incremental synchronization, the system responds to node changes in real time and detects event loss, solving the real-time and reliability issues in cross-cluster synchronization and achieving data consistency and business continuity.
Patent Information
- Application Number
- CN202611272028.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing cross-cluster data synchronization methods lack real-time incremental synchronization capabilities, cannot reflect changes in the source cluster in real time, and have reliability issues such as the inability to detect and compensate for lost events and the need to start synchronization from scratch after an interruption, making it difficult to meet the real-time requirements of disaster recovery scenarios.
After initially replicating the data using a full synchronization strategy, incremental synchronization is performed in real time in response to node change events. By comparing the consistency of node data between the source path and the target path, missing or faulty nodes are detected, and the full synchronization strategy is used for compensatory replication. Combined with progress information recording and breakpoint resume mechanism, synchronization efficiency and reliability are improved.
It achieves real-time and reliable cross-cluster data synchronization, ensuring data consistency between the target cluster and the source cluster, and guaranteeing business continuity and data accuracy in disaster recovery scenarios.
Smart Images

Figure CN123123362A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of independent station technology, and in particular to a data cross-cluster synchronization method, electronic device and storage medium. Background Technology
[0002] ZooKeeper distributed clusters, as a distributed coordination service, are widely used in scenarios such as service registration, distributed locks, and configuration management. The stored metadata needs to be kept synchronized across multiple clusters to meet business requirements such as disaster recovery switching, consistency across multiple environments, and cross-regional deployment.
[0003] Currently, data migration across clusters is typically achieved by copying snapshot files from the source cluster's data directory (including snapshot files and transaction logs) to the target cluster and then restarting. However, this method lacks real-time synchronization capabilities, and changes to the source cluster cannot be reflected in the target cluster in real time after synchronization. This makes it difficult to meet the real-time requirements of disaster recovery scenarios, and it suffers from weak real-time incremental synchronization capabilities, as well as reliability issues such as the inability to detect and compensate for lost events and the need to start synchronization from scratch after an interruption. Summary of the Invention
[0004] In view of the above, it is necessary to propose a data cross-cluster synchronization method, electronic device and storage medium to solve the technical problem of weak real-time incremental synchronization capability during cross-cluster data synchronization.
[0005] This application provides a data cross-cluster synchronization method, the method comprising: determining the source cluster and target cluster corresponding to the data synchronization task based on the configuration information of the full synchronization strategy; copying the data of all nodes under the source path corresponding to the source cluster to the target path corresponding to the target cluster by executing the full synchronization strategy; after the full synchronization strategy is executed, performing incremental synchronization in response to node change events under the source path, the incremental synchronization being used to synchronize the nodes corresponding to the node change events to the target path; determining the faulty node that caused the event loss failure during the incremental synchronization process by comparing the data consistency between the nodes under the source path and the corresponding nodes under the target path; and copying the data of the faulty node and its corresponding child nodes to the target path based on the full synchronization strategy.
[0006] In some embodiments, the method further includes: for each node under the source path, registering a data monitoring service and a child node monitoring service, wherein the data monitoring service is used to monitor data change events and deletion events in the node change events, and the child node monitoring service is used to monitor child node change events in the node change events; if a data change event or a deletion event is detected under the source path, and if the node corresponding to the data change event or the deletion event exists under the source path, re-registering the data monitoring service for that node; if a child node change event is detected for a node under the source path, re-registering the child node monitoring service for the node under the source path, and registering the data monitoring service and the child node monitoring service for the newly added child node corresponding to the child node change event.
[0007] In some embodiments, incremental synchronization in response to node change events under the source path includes: in response to a creation event in the node change event, adding a node corresponding to the creation event in the target cluster, and synchronizing the child nodes of the corresponding node using recursive operations; in response to a data change event, updating the data of the node corresponding to the data change event in the target cluster; in response to a deletion event, deleting the corresponding node and its child nodes in the target cluster; in response to a child node change event, determining the newly added child node and the deleted child node in the source cluster by comparing the child node lists in the source cluster and the target cluster; and synchronizing the newly added child node and the deleted child node in the target cluster using recursive operations.
[0008] In some embodiments, the method further includes: during the execution of the full synchronization strategy, after any node under the source path has been synchronized, updating the pre-stored progress information based on the storage path of the arbitrary node; when the full synchronization strategy is restarted after being interrupted, determining the synchronized nodes based on the progress information, and continuing to perform full synchronization on the remaining unsynchronized nodes; after completing the full synchronization of all nodes under the source path, restarting the listening to node change events under the source path, and obtaining the node change events generated during the interruption of the full synchronization strategy.
[0009] In some embodiments, the method further includes: re-establishing the communication session when the communication session of the source cluster or the target cluster fails; re-executing the full synchronization strategy based on the progress information after the communication session is re-established; and restarting the monitoring of node change events under the source path after the full synchronization strategy is completed.
[0010] In some embodiments, the method further includes: when any existing node on the target path corresponds to the same path as the node to be synchronized, obtaining the version number, modification time, and data content of the node to be synchronized and the any existing node; if it is determined based on the version number, the modification time, and the data content that there is a conflict between the node to be synchronized and the any existing node, handling the conflict based on a preset conflict resolution strategy, the conflict resolution strategy including an overwrite strategy, a skip strategy, and a merge strategy; the overwrite strategy includes: using the data of the node to be synchronized to overwrite the data of the any existing node; the skip strategy includes: retaining the data of the any existing node and terminating the synchronization process of the node to be synchronized; the merge strategy includes: determining updated node data from the any existing node and the node to be synchronized based on the version number and the modification time; and writing the updated node data into the any existing node.
[0011] In some embodiments, the method further includes: during the process of copying nodes under the source path to the target path, if the node to be synchronized meets a preset filtering condition, then no synchronization operation is performed on the node to be synchronized; the filtering condition includes any one of the following: the node to be synchronized is a temporary node or an ordered node; the data volume of the node to be synchronized is less than a preset first threshold or greater than a preset second threshold; the path of the node to be synchronized matches a preset regular expression.
[0012] In some embodiments, determining the fault node that caused the event loss during the incremental synchronization process by comparing the data consistency between the node under the source path and the corresponding node under the target path includes: randomly selecting an existing node from the source path at preset time intervals; determining the corresponding node of the existing node from the target path based on the path of the existing node; and determining the existing node as the fault node that caused the event loss if the data of the existing node and the corresponding node are inconsistent.
[0013] This application also provides an electronic device, which includes a processor and a memory. The processor is used to implement the data cross-cluster synchronization method when executing a computer program stored in the memory.
[0014] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the data cross-cluster synchronization method.
[0015] As can be seen from the above technical solutions, the embodiments of this application combine full synchronization and incremental synchronization. After the full synchronization is completed, incremental synchronization is performed in real time in response to node change events, achieving real-time data synchronization across clusters. By comparing the data consistency between nodes under the source path and the corresponding nodes under the target path, faulty nodes that have experienced event loss during incremental synchronization are identified. Based on the full synchronization strategy, data replication is re-executed for the faulty node and its corresponding child nodes, realizing the detection and compensation of event loss and ensuring data consistency. The progress information records synchronized nodes, and synchronization can continue from the breakpoint after an interruption, improving the efficiency of large-scale data synchronization. Conflict faults are handled through overlay, skip, and merge strategies, and the system supports filtering by node type, data volume, path mode, and other conditions to select nodes to be synchronized, improving the flexibility and controllability of the synchronization process. Attached Figure Description
[0016] Figure 1 This is an application scenario diagram of a data cross-cluster synchronization method provided in an embodiment of this application.
[0017] Figure 2 This is an architecture diagram of an independent website provided in one embodiment of this application.
[0018] Figure 3 This is a flowchart of a data cross-cluster synchronization method provided in an embodiment of this application.
[0019] Figure 4 This is a flowchart of a method for registering a listening service provided in an embodiment of this application.
[0020] Figure 5 This is a flowchart of an incremental synchronization method provided in an embodiment of this application.
[0021] Figure 6 This is a flowchart of a method for resuming interrupted downloads provided in an embodiment of this application.
[0022] Figure 7 This is a flowchart of a session expiration handling method provided in an embodiment of this application.
[0023] Figure 8 This is a flowchart of a conflict resolution method provided in an embodiment of this application.
[0024] Figure 9 This is a flowchart of a timed verification method provided in an embodiment of this application.
[0025] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0026] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.
[0027] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.
[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0029] This application provides a data cross-cluster synchronization method that can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0030] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.
[0031] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.
[0032] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).
[0033] like Figure 1 The diagram shown is an application scenario illustration of a cross-cluster data synchronization method provided in an embodiment of this application. Electronic device 100 is communicatively connected to source cluster 200 and target cluster 300. Electronic device 100 is a device that executes the cross-cluster data synchronization method, exemplarily a standalone server or any device in a standalone server cluster. Source cluster 200 is the source of synchronization data for the data synchronization task, storing a tree of nodes to be synchronized. Target cluster 300 is the data receiver for the data synchronization task, used to receive data copies from source cluster 200.
[0034] Taking a disaster recovery scenario as an example, the source cluster 200 is a production environment cluster that stores key metadata such as storage registration information and configuration information, while the target cluster 300 is a disaster recovery environment cluster.
[0035] When node data in the production environment changes, electronic device 100 performs a full synchronization, copying all nodes under the source path of source cluster 200 to the target path of target cluster 300. After the full synchronization is completed, electronic device 100 responds to node change events in source cluster 200 by performing incremental synchronization to keep the data of target cluster 300 consistent with that of source cluster 200. When event loss occurs during incremental synchronization, electronic device 100 identifies the faulty node by comparing node data consistency and performs compensatory replication on the faulty node and its corresponding child nodes.
[0036] When the source cluster 200 fails, the business can quickly switch to the disaster recovery cluster with consistent data to ensure business continuity.
[0037] like Figure 2 The diagram shown is an architecture diagram of an independent site provided in an embodiment of this application. The independent site 400 provided in this application can run on... Figure 1 The electronic device 100 shown provides a trading platform for merchants and users in cross-border e-commerce. The independent site 400 includes a monitoring module 410 and a synchronization module 420.
[0038] The monitoring module 410 is used to monitor node change events under the source path of the source cluster 200, and the synchronization module 420 is used to perform node data replication and synchronization. The source cluster 200 is the source of synchronization data for the data synchronization task and stores the node tree to be synchronized. The target cluster 300 is the data receiver for the data synchronization task and is used to receive data copies from the source cluster 200. The independent station 400 establishes communication connections with the source cluster 200 and the target cluster 300.
[0039] Taking a disaster recovery scenario as an example, the source cluster 200 is a production environment cluster that stores key metadata such as storage registration information and configuration information, while the target cluster 300 is a disaster recovery environment cluster. Based on the configuration information of the full synchronization strategy, the independent site 400 determines the source cluster 200 and the target cluster 300 corresponding to the data synchronization task. The synchronization module 420 copies the data of all nodes under the source path of the source cluster 200 to the target path of the target cluster 300 by executing the full synchronization strategy. After the full synchronization strategy is completed, the listening module 410 registers a listener for the nodes under the source path of the source cluster 200. In response to node change events under the source path, the synchronization module 420 synchronizes the nodes corresponding to the node change events to the target path of the target cluster 300, achieving incremental synchronization and ensuring that the data of the target cluster 300 and the source cluster 200 remain consistent in real time.
[0040] When event loss occurs during incremental synchronization, the monitoring module 410 identifies the faulty node causing the event loss by comparing the data consistency between nodes on the source path in the source cluster 200 and the corresponding nodes on the target path in the target cluster 300. The synchronization module 420 then replicates the data of the faulty node and its corresponding child nodes to the target cluster 300 based on a full synchronization strategy, completing the compensation synchronization. When the source cluster 200 fails, services can quickly switch to a data-consistent disaster recovery cluster, ensuring business continuity.
[0041] like Figure 3 The diagram shown is a flowchart of a data cross-cluster synchronization method provided in an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The data cross-cluster synchronization method provided in this embodiment includes the following steps.
[0042] S10, based on the configuration information of the full synchronization strategy, determines the source cluster and target cluster corresponding to the data synchronization task.
[0043] In one embodiment of this application, the data cross-cluster synchronization method needs to first determine the object of the data synchronization task, that is, which cluster to synchronize data from and to which cluster; otherwise, the execution scope cannot be located for subsequent full synchronization and incremental synchronization.
[0044] In one embodiment of this application, the execution device obtains the configuration information of the full synchronization strategy, parses the identification information of the source cluster and the target cluster from the configuration information, and uses the source cluster and the target cluster as the objects of the data synchronization task.
[0045] In one embodiment of this application, the configuration information may originate from user input, configuration files, or parameters issued by an automated operation and maintenance platform.
[0046] For example, the configuration information is the command-line parameters "synchronization source cluster address, target cluster address, source path, target path", and the execution device determines the source cluster and target cluster from the parameters; if the configuration information does not specify a source cluster or target cluster, the execution device can use the preset default cluster configuration, for example, using the pre-stored disaster recovery cluster mapping relationship to determine the source cluster and target cluster.
[0047] S11, by executing the full synchronization strategy, the data of all nodes under the source path corresponding to the source cluster is copied to the target path corresponding to the target cluster.
[0048] In one embodiment of this application, after determining the source cluster and the target cluster, it is necessary to establish an initial data foundation for synchronization, that is, to first copy the data of the existing nodes in the source cluster to the target cluster. Otherwise, the target cluster will not have the initial data corresponding to the source cluster, and subsequent incremental synchronization will be impossible.
[0049] In one embodiment of this application, the execution device traverses all nodes under the source path of the source cluster, reads the data content of each node one by one, and writes the read data content to the corresponding node under the target path of the target cluster; the traversal method is, for example, depth-first traversal; during the copying process, the node data and the node's metadata are processed together, including, for example, the node path, data content and version information.
[0050] In one embodiment of this application, the execution device establishes a connection with the source cluster and the target cluster, and obtains a list of nodes under the source path; it traverses each node in the node list in depth-first order, reads the data content of the node and writes it to the corresponding path of the target cluster; for sub-paths existing under the source path, the above traversal process is recursively executed until all nodes under the source path have been copied.
[0051] For example, the replication process uses a batch write method, with each batch written to multiple nodes. The amount of data written in each batch is limited by the single transaction capacity of the target cluster, which is, for example, no more than 1 megabyte.
[0052] S12, after the full synchronization strategy is completed, incremental synchronization is performed in response to the node change event under the source path. The incremental synchronization is used to synchronize the node corresponding to the node change event to the target path.
[0053] In one embodiment of this application, full synchronization only replicates the node data of the source cluster at the time of execution. After the full synchronization is completed, the node data of the source cluster will continue to change. If these changes are not synchronized, the target cluster will not be able to keep the data consistent with the source cluster. Therefore, incremental synchronization is required to synchronize the node changes generated after full synchronization.
[0054] In one embodiment of this application, the execution device initiates incremental synchronization after the full synchronization strategy is completed. Incremental synchronization is achieved by monitoring node change events under the source path of the source cluster. Node change events include node creation, data modification, node deletion, and child node changes. In response to the monitored node change events, the execution device synchronizes the node corresponding to the event to the target path of the target cluster.
[0055] In one embodiment of this application, the execution device registers with the source cluster to listen to the source path. When a node change occurs under the source path of the source cluster, the source cluster pushes the node change event to the execution device.
[0056] For example, the node change event is a ZooKeeper Watcher event; when the execution device receives a node creation event, it creates the corresponding node in the target cluster; when it receives a data change event, it updates the data of the corresponding node in the target cluster; when it receives a deletion event, it deletes the corresponding node in the target cluster; when it receives a child node change event, it synchronizes the newly added or deleted child nodes. For specific processing methods, please refer to the detailed descriptions corresponding to steps S30 to S33.
[0057] S13, by comparing the data consistency between the nodes under the source path and the corresponding nodes under the target path, the faulty node that caused the event loss fault during the incremental synchronization process is determined.
[0058] In one embodiment of this application, incremental synchronization relies on an event notification mechanism. However, the event notification mechanism may lose events under special circumstances, such as network anomalies causing event notifications to not be delivered, or communication session failures causing changes during the session to not generate event notifications. Event loss will cause some nodes in the target cluster to fail to synchronize with the changes in the source cluster. Therefore, it is necessary to detect event loss and locate the nodes where event loss has occurred.
[0059] In one embodiment of this application, the execution device obtains data of nodes under the source path and corresponding nodes under the target path, and compares whether the data content of the two is consistent; for nodes with inconsistent data, they are identified as faulty nodes that have caused event loss during incremental synchronization; the comparison method is exemplarily a periodic comparison, and for specific comparison methods, please refer to the detailed descriptions corresponding to steps S70 to S72.
[0060] In one embodiment of this application, the execution device determines the corresponding node in the target path based on the node path in the source path; reads the data content of the source node and the corresponding node and compares them; if the comparison result shows that the data is inconsistent, the node is marked as a faulty node.
[0061] For example, the execution device selects a portion of nodes for comparison at fixed time intervals (30 seconds for example), and the comparison content includes data content, data length and version information.
[0062] S14, based on the full synchronization strategy, copy the data of the faulty node and its corresponding child nodes to the target path.
[0063] In one embodiment of this application, the data of the faulty node and its corresponding child nodes in the target cluster are inconsistent with the source cluster. It is necessary to re-perform data replication to compensate for the data differences caused by the event loss. Otherwise, the data of the faulty node and its subtree will be inconsistent with the source cluster for a long time, affecting the accuracy of the target cluster data.
[0064] In one embodiment of this application, the execution device re-executes the full synchronization strategy on the determined faulty node and its corresponding child nodes, that is, traverses the subtree where the faulty node and its child nodes are located, and copies the data of all nodes in the subtree back to the corresponding path of the target cluster; after the copying is completed, the data of the subtree in the target cluster is consistent with the source cluster.
[0065] In one embodiment of this application, the execution device takes the faulty node as the root node, obtains the subtree composed of the root node and its child nodes, and performs the same traversal and copying process as step S11 on the subtree, copying the data of each node in the subtree to the corresponding path of the target cluster.
[0066] For example, the breakpoint resume mechanism is reused during the replication process, skipping already synchronized nodes based on progress information. For details on how to implement breakpoint resume, please refer to the detailed explanations corresponding to steps S40 to S42. After the compensation replication is completed, the comparison verification in step S13 can be performed again to confirm that the data of the faulty node and its child nodes are consistent.
[0067] like Figure 4 The diagram shown is a flowchart of a method for registering a listening service according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for registering a listening service provided in this embodiment of the application includes the following steps.
[0068] S20, for each node under the source path, register a data monitoring service and a child node monitoring service. The data monitoring service is used to monitor data change events and deletion events in the node change events, and the child node monitoring service is used to monitor child node change events in the node change events.
[0069] In one embodiment of this application, incremental synchronization needs to be aware of changes in each node under the source path. However, the event listening mechanism can usually only register listening for a single node, and different types of changes are covered by different types of listening services. Therefore, it is necessary to register a data listening service and a child node listening service for each node separately to cover two types of events: node data changes and child node changes.
[0070] In one embodiment of this application, the execution device traverses each node under the source path and registers a data monitoring service and a child node monitoring service for each node. The data monitoring service is used to monitor data change events and deletion events that occur at the node, and the child node monitoring service is used to monitor child node change events of the node.
[0071] In one embodiment of this application, the execution device obtains a list of nodes under the source path and performs a registration operation on each node in the list in turn.
[0072] For example, the data listening service is implemented by carrying a Watcher through ZooKeeper's getData interface, and the child node listening service is implemented by carrying a Watcher through ZooKeeper's getChildren interface; the above registration is performed on the root node and each of its child nodes under the source path to achieve coverage of the entire node tree under the source path.
[0073] S21, if a data change event or a deletion event is detected under the source path, and the node corresponding to the data change event or deletion event exists under the source path, the data monitoring service of the node is re-registered.
[0074] In one embodiment of this application, the event triggering mechanism is usually a one-time action, that is, the listening service becomes invalid after the event is triggered, and subsequent changes to the node will not be perceived if it is not re-registered.
[0075] In one embodiment of this application, it is necessary to re-register the listening service for the still existing nodes after the event is triggered in order to maintain continuous listening.
[0076] In one embodiment of this application, after receiving a data change event or deletion event, the execution device determines whether the node corresponding to the event still exists under the source path; if the node still exists, it re-registers the data monitoring service for the node to continue monitoring subsequent data change events and deletion events of the node; if the node no longer exists, it does not perform re-registration.
[0077] In one embodiment of this application, for example, after the execution device receives a data change event for node "Configuration A", it confirms through the exists interface that "Configuration A" still exists under the source path, and then re-registers the data listening service for "Configuration A"; if it is confirmed that "Configuration A" has been deleted, it will not be registered again, and its child nodes will be uniformly handled by the deletion events generated by the corresponding child node listening service.
[0078] S22, if a child node change event is detected in the node under the source path, the child node listening service of the node under the source path is re-registered, and the data listening service and the child node listening service are registered for the newly added child node corresponding to the child node change event.
[0079] In one embodiment of this application, after the list of child nodes of a node changes, the child node listening service is triggered and becomes invalid, and the newly added child node does not yet have any listening service. If the child node listening service is not re-registered or the new child node is not registered with a listening service, the subsequent changes to the new child node will not be perceived.
[0080] In one embodiment of this application, re-registration and new registration need to be completed simultaneously.
[0081] In one embodiment of this application, after receiving a child node change event, the execution device re-registers the child node listening service for that node; at the same time, it obtains the list of child nodes after the child node change, determines the newly added child nodes, and registers a data listening service and a child node listening service for each newly added child node.
[0082] In one embodiment of this application, for example, when a new child node "Service A / Instance 1" is added to the node "Service A", after the execution device receives the child node change event of "Service A", it re-registers the child node listening service of "Service A" and obtains the new child node list through the getChildren interface, identifies the newly added child node "Service A / Instance 1", and registers a data listening service and a child node listening service for "Service A / Instance 1". Thereafter, data changes, deletions and child node changes of "Service A / Instance 1" can be detected.
[0083] like Figure 5The diagram shown is a flowchart of an incremental synchronization method according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The incremental synchronization method provided in this embodiment includes the following steps.
[0084] S30, in response to the creation event in the node change event, add a node corresponding to the creation event in the target cluster, and synchronize the child nodes of the corresponding node using recursive operations.
[0085] In one embodiment of this application, a new node is created in the source cluster, but there is no corresponding node in the target cluster. If the new node and its child nodes are not synchronized to the target cluster, the target cluster will lack the data of the node and its subtree.
[0086] In one embodiment of this application, the execution device needs to fully synchronize the newly created node and its child nodes to the target cluster.
[0087] In one embodiment of this application, after receiving a creation event, the execution device determines the node corresponding to the creation event, adds the corresponding node under the target path of the target cluster, and synchronizes the child nodes of the corresponding node through recursive operation.
[0088] In one embodiment of this application, for example, the source cluster creates a node "Configuration B". After receiving the creation event, the execution device creates a "Configuration B" node under the target path of the target cluster, and recursively obtains all child nodes of "Configuration B" (such as "Configuration B / Parameter 1"), and copies them one by one to the target cluster to ensure that the new node subtree is completely synchronized.
[0089] S31, in response to the data change event, update the data of the corresponding node in the target cluster for the data change event.
[0090] In one embodiment of this application, if the data content of a node in the source cluster changes, the data of the corresponding node in the target cluster will remain the old value. If it is not updated, the target cluster will retain the expired data.
[0091] In one embodiment of this application, the execution device needs to update the latest data of the nodes in the source cluster to the target cluster.
[0092] In one embodiment of this application, after receiving a data change event, the execution device determines the node corresponding to the data change event, reads the latest data of that node in the source cluster, and updates the data of the corresponding node in the target cluster.
[0093] In one embodiment of this application, for example, the data of the source cluster node "Configuration A" is updated from version 1 to version 2. After receiving the data change event, the execution device reads the latest data content of "Configuration A" and writes it to the "Configuration A" node of the target cluster.
[0094] S32, in response to the deletion event, delete the corresponding node and its child nodes in the target cluster.
[0095] In one embodiment of this application, if a node in the source cluster is deleted, the corresponding node and its child nodes in the target cluster are still retained. If the nodes are not deleted synchronously, the target cluster will have residual nodes that no longer exist, resulting in data redundancy and inconsistency.
[0096] In one embodiment of this application, the execution device needs to delete the node and its child nodes corresponding to the deleted node in the target cluster.
[0097] In one embodiment of this application, after receiving a deletion event, the execution device determines the node corresponding to the deletion event and deletes the corresponding node and all its child nodes in the target cluster.
[0098] In one embodiment of this application, for example, when the source cluster deletes the node "Service A / Instance 1", after the execution device receives the deletion event, it deletes the "Service A / Instance 1" node and its child nodes in the target cluster, so as to keep the node structure of the target cluster consistent with that of the source cluster.
[0099] S33, in response to the child node change event, by comparing the child node lists in the source cluster and the target cluster, the newly added child node and the deleted child node in the source cluster are determined; the newly added child node and the deleted child node are synchronized in the target cluster using a recursive operation.
[0100] In one embodiment of this application, the list of child nodes of a node changes, including the addition of child nodes and the deletion of child nodes, which need to be handled separately; if the child node list is not compared, it will be impossible to determine the specific added or deleted child nodes.
[0101] In one embodiment of this application, the execution device needs to compare the child node lists of the source cluster and the target cluster to determine and synchronize the added and deleted child nodes.
[0102] In one embodiment of this application, after receiving a child node change event, the execution device obtains the child node list of the node in the source cluster, compares it with the child node list of the corresponding node in the target cluster, determines the newly added child node and the deleted child node in the source cluster, and uses recursive operations to synchronize the newly added child node and delete the deleted child node in the target cluster.
[0103] In one embodiment of this application, for example, the list of child nodes of the source cluster node "Service A" is changed from "Instance 1, Instance 2" to "Instance 2, Instance 3". After performing device comparison, it is determined to add a child node "Instance 3" and delete a child node "Instance 1". In the target cluster, "Instance 3" and its child nodes are recursively created, and "Instance 1" and its child nodes are recursively deleted.
[0104] like Figure 6 The diagram shown is a flowchart of a method for resuming interrupted downloads according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for resuming interrupted downloads according to an embodiment of this application includes the following steps.
[0105] S40, during the execution of the full synchronization strategy, after any node under the source path has completed synchronization, the pre-stored progress information is updated based on the storage path of the node.
[0106] In one embodiment of this application, large-scale full data synchronization takes a long time. The synchronization process may be interrupted due to network failures, program abnormalities, or other reasons. If there is no progress record after the interruption, all nodes need to be resynchronized from the beginning, resulting in low synchronization efficiency.
[0107] In one embodiment of this application, it is necessary to record the synchronization progress. During the full synchronization process, each time the device completes the synchronization of a node, the pre-stored progress information is updated based on the storage path of that node. The progress information records the storage path of the nodes that have been synchronized.
[0108] In one embodiment of this application, for example, progress information is stored in a local progress file. The progress file is in JSON format and records the list of synchronized node paths and the last synchronization time. Nodes under the source path are synchronized in depth-first order. After node "Configuration A" is synchronized, the execution device appends its path "Configuration A" to the progress file.
[0109] S41, when the full synchronization strategy is restarted after being interrupted, the synchronized nodes are determined based on the progress information, and full synchronization is continued for the remaining unsynchronized nodes.
[0110] In one embodiment of this application, when full synchronization is interrupted and restarted, if synchronization starts from the beginning, the already synchronized nodes will be copied repeatedly, wasting resources; by using progress information to identify the synchronized nodes, only the nodes that have not yet been completed can be synchronized.
[0111] In one embodiment of this application, the execution device determines the synchronization breakpoint based on progress information and continues synchronization from the unsynchronized node.
[0112] In one embodiment of this application, when the execution device restarts full synchronization, it reads progress information, determines the set of synchronized nodes, skips the synchronized nodes, and continues to perform full synchronization on the unsynchronized nodes.
[0113] In one embodiment of this application, for example, the progress information shows that nodes "Configuration A" and "Configuration B" have been synchronized, and the execution device continues to copy from the unsynchronized node "Configuration C" under the source path, without repeatedly copying "Configuration A" and "Configuration B".
[0114] S42, after completing the full synchronization of all nodes under the source path, restart the listening for node change events under the source path and obtain the node change events generated during the interruption of the full synchronization strategy.
[0115] In one embodiment of this application, during a full synchronization interruption, the nodes in the source cluster may continue to change, and these changes are not captured by incremental synchronization during the interruption. After the full synchronization is completed, listening needs to be restarted to obtain the node change events generated during the interruption and to catch up on the data differences.
[0116] In one embodiment of this application, the execution device resumes monitoring of node change events after completing full synchronization. After completing full synchronization of the remaining nodes, it restarts monitoring of node change events under the source path, acquires node change events generated during the interruption, and performs incremental synchronization on the nodes corresponding to the events.
[0117] In one embodiment of this application, for example, during the synchronization interruption, the source cluster creates a node "Configuration D". After the full synchronization resumes, the device restarts listening, obtains the creation event of "Configuration D", and synchronizes "Configuration D" to the target cluster.
[0118] like Figure 7 The diagram shown is a flowchart of a session expiration handling method according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The session expiration handling method provided in this embodiment includes the following steps.
[0119] S50, when the communication session of the source cluster or the target cluster fails, the communication session is re-established.
[0120] In one embodiment of this application, the execution device interacts with the source cluster and the target cluster through a communication session. The communication session may fail due to network anomalies, cluster restarts, or other reasons. Node change events generated during the session failure period cannot be captured, resulting in a significant risk of event loss.
[0121] In one embodiment of this application, it is necessary to re-establish the communication session to restore synchronization capability. Specifically, the device monitors the communication session status with the source cluster and the target cluster, and re-establishes the communication session when it fails.
[0122] In one embodiment of this application, for example, the session between the execution device and the source cluster expires due to network jitter. After detecting the session failure, the execution device re-initiates a connection request to establish a new session. For example, the communication session is a ZooKeeper client session, and the session timeout is 40 seconds.
[0123] S51, after the communication session is rebuilt, the full synchronization strategy is re-executed based on the progress information.
[0124] In one embodiment of this application, during the communication session failure, node change events in the source cluster are not captured, and there may be significant data differences between the target cluster and the source cluster; after re-establishing the session, a full synchronization needs to be re-executed to repair the differences.
[0125] In one embodiment of this application, the execution device re-executes the full synchronization strategy based on progress information to repair data differences during session failure.
[0126] In one embodiment of this application, after the communication session is rebuilt, the execution device re-executes the full synchronization strategy based on the progress information, skips the synchronized nodes using the progress information, performs full synchronization on the unsynchronized nodes, and repairs the data differences caused during the session failure.
[0127] In one embodiment of this application, for example, during the communication session failure, multiple nodes are added to the source cluster. After the session is rebuilt, the execution device determines the synchronized nodes based on the progress file, re-executes full synchronization on the unsynchronized nodes, and copies the newly added nodes to the target cluster.
[0128] S52, after the full synchronization strategy is completed, restart the monitoring of node change events under the source path.
[0129] In one embodiment of this application, full synchronization repairs data differences during session failure. However, after full synchronization is completed, the nodes in the source cluster may continue to change, requiring a restart of listening to continue incremental synchronization.
[0130] In one embodiment of this application, incremental synchronization is resumed after full synchronization is completed. Furthermore, after the full synchronization strategy is executed, the monitoring of node change events under the source path is restarted, allowing incremental synchronization to resume operation.
[0131] In one embodiment of this application, for example, after the full synchronization is completed, the execution device re-registers to listen to the source path, and thereafter all node change events in the source cluster are responded to and synchronized to the target cluster.
[0132] like Figure 8 The diagram shown is a flowchart of a conflict resolution method according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The conflict resolution method provided in this embodiment includes the following steps.
[0133] S60, if any existing node under the target path corresponds to the same path as the node to be synchronized, obtain the version number, modification time and data content of the node to be synchronized and any existing node.
[0134] In one embodiment of this application, there may already be a node with the same path as the node to be synchronized under the target path of the target cluster. For example, there may be historical data or previously synchronized data in the target cluster. If the data is directly overwritten, it may overwrite the newer data in the target cluster.
[0135] In one embodiment of this application, it is necessary to first obtain information from both parties to determine a conflict. When the execution device detects that an existing node on the target path corresponds to the same path as the node to be synchronized, it obtains the version number, modification time, and data content of the node to be synchronized and the existing node.
[0136] In one embodiment of this application, for example, a node "Configuration E" (version 3) already exists under the target path, and the node to be synchronized is "Configuration E" (version 5) of the source cluster. The execution device reads the version number, modification time and data content of both. For example, the version number and modification time are obtained through the node's metadata, and the data content is obtained by reading the node's data.
[0137] S61, if it is determined that there is a conflict between the node to be synchronized and any existing node based on the version number, the modification time and the data content, the conflict is handled based on a preset conflict resolution strategy, which includes an overwrite strategy, a skip strategy and a merge strategy.
[0138] In one embodiment of this application, after obtaining information from both parties, it is necessary to determine whether there is a conflict. If there is no conflict, synchronization can be performed directly. If there is a conflict, it needs to be processed according to a preset strategy to avoid blind overwriting or missing updates.
[0139] In one embodiment of this application, after determining that a conflict fault exists, the execution device handles the conflict based on a preset conflict resolution strategy.
[0140] In one embodiment of this application, the execution device determines whether there is a conflict between the node to be synchronized and the existing node based on the version number, modification time and data content; if there is a conflict, the conflict is handled based on a preset conflict resolution strategy; the conflict resolution strategy includes an overwrite strategy, a skip strategy and a merge strategy.
[0141] In one embodiment of this application, for example, the version number of the node to be synchronized "Configuration E" (version 5) is different from that of the existing node "Configuration E" (version 3) and the data content is different, and a conflict is determined to exist; the conflict resolution strategy is specified by the user in advance through parameters, and for example, it is specified as the overwrite strategy.
[0142] In one embodiment of this application, the overlay strategy includes: using the data of the node to be synchronized to overlay the data of any existing node.
[0143] In scenarios where the target cluster contains outdated data and the source cluster data should be the sole criterion, it is necessary to overwrite the target cluster data with the source cluster data. Conflict resolution is handled using the source cluster data. When the conflict resolution strategy is to overwrite, the data of the node to be synchronized is written to the existing node, overwriting the existing data.
[0144] For example, the target cluster "Configuration E" contains expired data (version 3), while the source cluster "Configuration E" contains the latest data (version 5). The execution device writes the version 5 data to the target cluster "Configuration E" node to complete the overwrite.
[0145] In one embodiment of this application, the skipping strategy includes: retaining the data of any one of the existing nodes and terminating the synchronization process of the node to be synchronized.
[0146] In one embodiment of this application, where the existing data in the target cluster is the latest data and should not be overwritten by the data in the source cluster, it is necessary to retain the target cluster data and terminate the synchronization of that node. The execution device retains the existing data of the target cluster and terminates the synchronization of that node. When the conflict resolution strategy is a skip strategy, the execution device retains the data of the existing node, terminates the synchronization process of the node to be synchronized, and does not perform data writing.
[0147] For example, the target cluster "Configuration E" contains the latest data (version 6), while the source cluster "Configuration E" contains older data (version 5). The executing device retains the target cluster data, skips the synchronization of "Configuration E", and continues to process the next node.
[0148] In one embodiment of this application, the merging strategy includes: determining updated node data from the arbitrary existing node and the node to be synchronized based on the version number and the modification time; and writing the updated node data into the arbitrary existing node.
[0149] In one embodiment of this application, in scenarios where it is impossible to determine which side's data is updated and intelligent judgment is required, the newer data needs to be selected as the merging result based on the version number or modification time. The execution device selects the newer data as the merging result based on the version number and modification time. When the conflict resolution strategy is a merging strategy, the execution device determines the updated node data from the existing nodes and the nodes to be synchronized based on the version number and modification time, and writes the latest node data into the existing nodes.
[0150] For example, the execution device compares the modification times on both sides of "Configuration E" and determines that the data on the side with the later modification time is the latest data, and writes the data to the "Configuration E" node in the target cluster; when the version number and modification time are inconsistent, the data with the later modification time is used first for example.
[0151] In one embodiment of this application, during the process of copying nodes from the source path to the target path, if the node to be synchronized meets the preset filtering conditions, the synchronization operation on the node to be synchronized is not performed.
[0152] In one embodiment of this application, some nodes under the source path may not need to be synchronized to the target cluster, such as temporary state nodes, nodes with extremely large data volumes, etc. If all nodes are synchronized, it will cause data redundancy and unnecessary network overhead in the target cluster.
[0153] In one embodiment of this application, nodes to be synchronized need to be filtered according to filtering conditions. During the node replication process, the execution device applies preset filtering conditions to the nodes to be synchronized and judges them; if the nodes to be synchronized meet the filtering conditions, no synchronization operation is performed on the nodes to be synchronized; if the nodes to be synchronized do not meet the filtering conditions, normal synchronization operation is performed.
[0154] For example, the filtering conditions are specified by the user through configuration information. When the execution device traverses the source path nodes, it checks whether each node meets the filtering conditions one by one. Nodes that meet the conditions are skipped, and nodes that do not meet the conditions are copied.
[0155] In one embodiment of this application, the filtering conditions include any one of the following: the node to be synchronized is a temporary node or an ordered node; the data volume of the node to be synchronized is less than a preset first threshold or greater than a preset second threshold; the path of the node to be synchronized matches a preset regular expression.
[0156] In one embodiment of this application, ephemeral nodes and ordered nodes typically have time-sensitive or session-dependent characteristics, and synchronization to the target cluster may generate invalid data, which can be used as a filtering condition. The execution device determines whether the node to be synchronized is an ephemeral node or an ordered node. If the node to be synchronized is an ephemeral node or an ordered node, the synchronization operation for that node is not performed.
[0157] For example, the execution device determines the node type by checking the node's metadata attributes. For example, the session owner attribute of a temporary node is not empty, and an ordered node is a node whose path includes a sequence number suffix. For example, the node "lock / temporary lock 1" is a temporary node, and the execution device skips the synchronization of this node.
[0158] In one embodiment of this application, nodes with too little data may be placeholder nodes, while nodes with too much data may contain business data rather than metadata; neither is suitable for synchronization. A data volume threshold can be used as a filtering condition. The execution device obtains the data volume of the node to be synchronized and compares it with preset first and second thresholds. If the data volume is less than the first threshold or greater than the second threshold, synchronization operation is not performed on that node.
[0159] For example, the first threshold is 1 byte, the second threshold is 10 megabytes, and the data size of the node to be synchronized, "Large Object 1", is 100 megabytes, which is greater than the second threshold. Therefore, the execution device skips the synchronization of this node.
[0160] In one embodiment of this application, the user may only need to synchronize nodes whose paths conform to a specific pattern, such as only synchronizing nodes under the configuration class path. Path pattern matching can be used as a filtering condition. The execution device matches the path of the node to be synchronized with a preset regular expression. If the path matches the regular expression, the synchronization operation for that node is not performed.
[0161] For example, the preset regular expression matches nodes containing the keyword "temporary". The node to be synchronized has the path "temporary / node1" and matches the regular expression, so the execution device skips the synchronization of that node.
[0162] like Figure 9 The diagram shown is a flowchart of a timing verification method provided in an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The timing verification method provided in this embodiment includes the following steps.
[0163] S70, at preset time intervals, randomly select existing nodes from the source path.
[0164] In one embodiment of this application, event loss is usually not directly perceptible and needs to be discovered indirectly by comparing the data of the source cluster and the target cluster; comparing all nodes one by one is costly.
[0165] In one embodiment of this application, a periodic sampling comparison method is adopted to control overhead while ensuring detection capability. The execution device randomly selects existing nodes from the source path as verification samples at preset time intervals.
[0166] In one embodiment of this application, for example, the preset time interval is 30 seconds, there are 1000 nodes in the source path, and the execution device randomly selects 50 nodes as verification samples in each cycle; the random selection method is, for example, random number sampling.
[0167] S71, Based on the path of the existing node, determine the corresponding node of the existing node from the target path.
[0168] In one embodiment of this application, to compare the data of the source cluster and the target cluster, it is necessary to first locate the node in the target cluster that corresponds to the node in the source cluster, that is, the node corresponding to the path.
[0169] In one embodiment of this application, the execution device locates the corresponding node in the target cluster based on the path mapping relationship.
[0170] In one embodiment of this application, the execution device determines the corresponding node under the target path based on the path of the selected existing node.
[0171] In one embodiment of this application, for example, the existing node selected is "Configuration A", and the execution device locates the "Configuration A" node under the target path of the target cluster; if there is no corresponding node under the target path, it is determined that the node data is inconsistent.
[0172] S72, if the data of the existing node is inconsistent with that of the corresponding node, the existing node is identified as the faulty node that caused the event loss fault.
[0173] In one embodiment of this application, if the data of the corresponding nodes in the source cluster and the target cluster are inconsistent, it indicates that the changes of the node were not synchronized during the incremental synchronization process, that is, event loss has occurred, and the node needs to be marked as a faulty node for compensation.
[0174] In one embodiment of this application, the execution device compares the data of an existing node with that of the corresponding node, and if the data is inconsistent, the existing node is identified as a faulty node.
[0175] In one embodiment of this application, by way of example, the execution device compares the data of "Configuration A" in the source cluster and the target cluster, finds that the data content of the two is inconsistent, identifies "Configuration A" as a faulty node, and performs compensation replication on the faulty node and its child nodes in accordance with S14.
[0176] Please see Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement a data cross-cluster synchronization method as described in any of the above embodiments.
[0177] In one embodiment of this application, the electronic device 100 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as a data cross-cluster synchronization program.
[0178] Figure 10 Only an electronic device 100 with memory 12 and processor 13 is shown; those skilled in the art will understand that... Figure 10 The structure shown does not constitute a limitation on the electronic device 100, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0179] Combination Figure 3 The memory 12 in the electronic device 100 stores multiple computer-readable instructions to implement the cross-cluster data synchronization method. The processor 13 can execute the multiple instructions to achieve: determining the source cluster and target cluster corresponding to the data synchronization task based on the configuration information of the full synchronization strategy; copying the data of all nodes under the source path corresponding to the source cluster to the target path corresponding to the target cluster by executing the full synchronization strategy; after the full synchronization strategy is completed, performing incremental synchronization in response to node change events under the source path, the incremental synchronization being used to synchronize the nodes corresponding to the node change events to the target path; determining the faulty node that caused the event loss failure during the incremental synchronization process by comparing the data consistency between the nodes under the source path and the corresponding nodes under the target path; and copying the data of the faulty node and its corresponding child nodes to the target path based on the full synchronization strategy.
[0180] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.
[0181] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, the electronic device 100 may also include input / output devices, network access devices, etc.
[0182] It should be noted that electronic device 100 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.
[0183] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as the portable hard drive of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 100. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 100, such as the code of a data cross-cluster synchronization program, but also to temporarily store data that has been output or will be output.
[0184] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 100, connecting various components of the electronic device 100 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing cross-cluster data synchronization programs) and calls data stored in the memory 12 to perform various functions of the electronic device 100 and process data.
[0185] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes these applications to implement the steps in the various data cross-cluster synchronization method embodiments described above, for example... Figure 3 The steps are shown.
[0186] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 100.
[0187] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the data cross-cluster synchronization method described in the various embodiments of this application.
[0188] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.
[0189] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.
[0190] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.
[0191] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 10 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.
[0192] This application also provides a computer-readable storage medium (not shown), which stores computer-readable instructions that are executed by a processor in an electronic device to implement the data cross-cluster synchronization method described in any of the above embodiments.
[0193] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0194] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0195] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0196] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.
[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.
Claims
1. A method for synchronizing data across clusters, the method comprising: The method includes: Based on the configuration information of the full synchronization strategy, determine the source cluster and target cluster corresponding to the data synchronization task; By executing the full synchronization strategy, the data of all nodes under the source path corresponding to the source cluster is copied to the target path corresponding to the target cluster. After the full synchronization strategy is completed, incremental synchronization is performed in response to node change events under the source path. The incremental synchronization is used to synchronize the nodes corresponding to the node change events to the target path. By comparing the data consistency between the nodes under the source path and the corresponding nodes under the target path, the faulty nodes that caused event loss during the incremental synchronization process are identified. Based on the full synchronization strategy, the data of the faulty node and its corresponding child nodes are copied to the target path.
2. The data cross-cluster synchronization method as described in claim 1, characterized in that, The method further includes: For each node under the source path, register a data monitoring service and a child node monitoring service. The data monitoring service is used to monitor data change events and deletion events in the node change events, and the child node monitoring service is used to monitor child node change events in the node change events. If a data change event or a deletion event is detected under the source path, and the node corresponding to the data change event or deletion event exists under the source path, the data monitoring service of the node is re-registered. If a child node change event is detected in a node under the source path, the child node listening service of the node under the source path is re-registered, and the data listening service and the child node listening service are registered for the newly added child node corresponding to the child node change event.
3. The method for synchronizing data across clusters of claim 2, wherein, The incremental synchronization in response to node change events under the source path includes: In response to the creation event in the node change event, a new node corresponding to the creation event is added in the target cluster, and the child nodes of the corresponding node are synchronized using recursive operations; In response to the data change event, update the data of the corresponding node in the target cluster where the data change event occurred; In response to the deletion event, the corresponding node and its child nodes in the target cluster are deleted; In response to the child node change event, the newly added and deleted child nodes in the source cluster are determined by comparing the child node lists in the source cluster and the target cluster; the newly added and deleted child nodes are then synchronized in the target cluster using a recursive operation.
4. The method for synchronizing data across clusters of claim 1, wherein, The method further includes: During the execution of the full synchronization strategy, once any node under the source path has completed synchronization, the pre-stored progress information is updated based on the storage path of that node. When the full synchronization strategy is restarted after being interrupted, the synchronized nodes are determined based on the progress information, and full synchronization is continued for the remaining unsynchronized nodes. After completing the full synchronization of all nodes under the source path, restart the listening for node change events under the source path to obtain node change events generated during the interruption of the full synchronization strategy.
5. The method for synchronizing data across clusters of claim 4, wherein, The method further includes: When the communication session of the source cluster or the target cluster fails, the communication session is re-established; After the communication session is rebuilt, the full synchronization strategy is re-executed based on the progress information; After the full synchronization strategy is completed, the monitoring of node change events under the source path will be restarted.
6. The method for synchronizing data across clusters of claim 1, wherein, The method further includes: If any existing node on the target path corresponds to the same path as the node to be synchronized, obtain the version number, modification time, and data content of the node to be synchronized and any existing node. If, based on the version number, the modification time, and the data content, it is determined that there is a conflict between the node to be synchronized and any existing node, the conflict is handled based on a preset conflict resolution strategy, which includes an overwrite strategy, a skip strategy, and a merge strategy. The overlay strategy includes: using the data of the node to be synchronized to overwrite the data of any existing node; The skipping strategy includes: retaining the data of any one of the existing nodes and terminating the synchronization process of the node to be synchronized; The merging strategy includes: determining the updated node data from any existing node and the node to be synchronized based on the version number and the modification time; and writing the updated node data into any existing node.
7. The data cross-cluster synchronization method as described in claim 6, characterized in that, The method further includes: During the process of copying nodes from the source path to the target path, if the node to be synchronized meets the preset filtering conditions, the synchronization operation on the node to be synchronized will not be performed. The filtering conditions include any one of the following: The node to be synchronized is a temporary node or an ordered node; The amount of data in the node to be synchronized is less than a preset first threshold or greater than a preset second threshold. The path of the node to be synchronized matches a preset regular expression.
8. The method for synchronizing data across clusters of claim 1, wherein, The step of determining the faulty node that caused event loss during the incremental synchronization process by comparing the data consistency between the nodes under the source path and the corresponding nodes under the target path includes: At preset time intervals, existing nodes are randomly selected from the source path; Based on the path of the existing node, determine the corresponding node of the existing node from the target path; If the data of the existing node is inconsistent with that of the corresponding node, the existing node will be identified as the faulty node that caused the event loss fault.
9. An electronic device, comprising: The electronic device includes a processor and a memory, the processor being configured to implement the data cross-cluster synchronization method as described in any one of claims 1 to 8 when executing a computer program stored in the memory.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data cross-cluster synchronization method as described in any one of claims 1 to 8.
