Cross-cluster data synchronization method and apparatus

By constructing global synchronization information and monitoring topology changes in real time, the data to be synchronized is accurately determined, solving the consistency and complexity problems in traditional cross-cluster data synchronization and achieving efficient and reliable data synchronization.

CN121456053BActive Publication Date: 2026-11-10HUNAN HAPPLY SUNSHINE INTERACTIVE ENTERTAINMENT MEDIA CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511562829.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-10-28
Publication Date
2026-11-10
Estimated Expiration
2045-10-28

AI Technical Summary

Technical Problem

Traditional cross-cluster data synchronization methods struggle to guarantee data consistency in asymmetric scenarios, and the complexity of adjusting synchronization strategies increases system complexity and maintenance costs.

Method used

By constructing global synchronization information, the data synchronization methods of the source and target clusters are periodically obtained, the location information of synchronized data is determined, and the incremental data to be synchronized is accurately determined under symmetric or asymmetric conditions. Data synchronization is performed by atomic writing or hash value mapping, and real-time monitoring of cluster topology changes triggers synchronization.

Benefits of technology

In scenarios involving changes in cluster topology, accurately identifying the data to be synchronized avoids data omissions or duplicate synchronizations, improving the efficiency and reliability of data synchronization, and reducing system complexity and maintenance costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121456053B_ABST
    Figure CN121456053B_ABST
Patent Text Reader

Abstract

This application discloses a cross-cluster data synchronization method and apparatus. The method includes: periodically acquiring global synchronization information, which includes the data synchronization methods of the source cluster and the target cluster in the previous synchronization cycle; when the data synchronization method is symmetric synchronization, acquiring the synchronized data location information of each first node in the source cluster from each second node in the target cluster; when the data synchronization method is asymmetric synchronization, determining the synchronized data location of each first node from the global synchronization information; for each first node, determining the incremental data to be synchronized in the first node based on the synchronized data location, and synchronizing the incremental data to the second node corresponding to the first node in the target cluster; after data synchronization is completed, updating the global synchronization information based on the synchronization result. This application solves the technical problem that traditional cross-cluster data synchronization schemes cannot perform data synchronization normally in cluster topology change scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and more specifically, to a cross-cluster data synchronization method and apparatus. Background Technology

[0002] In today's internet era, data real-time performance and consistency have become key metrics for measuring system performance. As business scales expand, single database clusters often struggle to meet the storage and access demands of massive amounts of data. Distributed databases are widely used due to their high availability, scalability, and load balancing advantages. However, the complexity of distributed database architectures also presents challenges for data synchronization, especially in cross-cluster synchronization scenarios.

[0003] Traditional distributed database synchronization methods, such as Redis-shake and xpipe, typically assume that the source and target clusters have a symmetrical topology, meaning that the number of nodes and data sharding layout are exactly the same. This premise simplifies the synchronization process, making the transmission and replay of data logs more direct. However, in real-world applications, this assumption is often difficult to meet. For example, in cold data backup, hot standby, or resource optimization scenarios, due to cost control and resource allocation considerations, the number of nodes and structure of the target cluster may be significantly fewer or different from those of the source cluster, resulting in asymmetry in data synchronization.

[0004] The main challenges of asymmetric data synchronization include: difficulty in guaranteeing data consistency, with incremental data from the source cluster potentially failing to map accurately to the destination cluster; and difficulty in adjusting synchronization strategies when the cluster topology changes, especially when the number of nodes increases or decreases, which can cause existing synchronization processes to fail. To address these issues, existing synchronization methods often rely on complex mapping tables and additional metadata management, which not only increases system complexity but may also introduce additional maintenance costs and potential points of failure.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a cross-cluster data synchronization method and apparatus to at least solve the technical problem that traditional cross-cluster data synchronization schemes cannot perform data synchronization normally in cluster topology change scenarios.

[0007] According to one aspect of the embodiments of this application, a cross-cluster data synchronization method is provided, comprising: periodically acquiring global synchronization information, wherein the global synchronization information includes the data synchronization mode of the source cluster and the target cluster in the previous synchronization cycle of the current synchronization cycle; when the data synchronization mode is symmetric synchronization, acquiring the synchronized data location information of each first node in the source cluster from each second node in the target cluster; when the data synchronization mode is asymmetric synchronization, determining the synchronized data location information of each first node in the source cluster from the global synchronization information; for each first node in the source cluster, determining the incremental data to be synchronized in the first node based on the synchronized data location information of the first node, and synchronizing the incremental data to the second node in the target cluster corresponding to the first node; and updating the global synchronization information based on the data synchronization result after the data synchronization is completed.

[0008] Optionally, if global synchronization information is not obtained, it is determined that the source cluster and the target cluster have not performed data synchronization; for each first node in the source cluster, all data in the first node is taken as incremental data to be synchronized, and the incremental data is synchronized to the second node in the target cluster corresponding to the first node; after the data synchronization is completed, global synchronization information is generated based on the data synchronization result.

[0009] Optionally, synchronizing incremental data to the second node corresponding to the first node in the target cluster includes: obtaining the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster; if the first distribution state and the second distribution state are symmetrical, writing the incremental data to the second node corresponding to the first node in the target cluster in an atomic write manner, and writing the end position of the incremental data in the first node as the synchronized data position information of the first node to the second node; if the first distribution state and the second distribution state are asymmetrical, for each sub-incremental data in the incremental data, determining the target second node corresponding to the sub-incremental data in the target cluster based on the hash value of the sub-incremental data, and writing the sub-incremental data to the target second node.

[0010] Optionally, determining the target second node in the target cluster corresponding to the sub-incremental data based on the hash value of the sub-incremental data includes: determining the number of second nodes in the target cluster and sequentially numbering each second node; performing a modulo operation on the hash value and the number of nodes to determine the second node corresponding to the node number and the obtained remainder value as the target second node corresponding to the sub-incremental data.

[0011] Optionally, determining the target second node in the target cluster corresponding to the sub-incremental data based on the hash value of the sub-incremental data includes: constructing a hash ring and mapping all second nodes in the target cluster to the hash ring, wherein each second node corresponds to at least one virtual node on the hash ring; determining the target position of the hash value on the hash ring and determining the target virtual node closest to the target position, and taking the second node corresponding to the target virtual node as the target second node corresponding to the sub-incremental data.

[0012] Optionally, the global synchronization information is updated based on the data synchronization results, including: when the first distribution state and the second distribution state are symmetrical, the data synchronization method in the global synchronization information is updated to symmetric synchronization; when the first distribution state and the second distribution state are asymmetrical, the data synchronization method in the global synchronization information is updated to asymmetric synchronization. For each first node in the source cluster, the end position of the incremental data of the first node in the first node is used as the synchronized data position information of the first node and updated to the global synchronization information.

[0013] Optionally, the system monitors in real time whether the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster have changed; when the first distribution state and / or the second distribution state change, the system actively triggers a new synchronization cycle to synchronize data.

[0014] According to another aspect of the embodiments of this application, a cross-cluster data synchronization device is also provided, comprising: an acquisition module, configured to periodically acquire global synchronization information, wherein the global synchronization information includes the data synchronization method of the source cluster and the target cluster in the previous synchronization period of the current synchronization period; a first position determination module, configured to acquire synchronized data position information of each first node in the source cluster from each second node in the target cluster when the data synchronization method is symmetric synchronization; a second position determination module, configured to determine synchronized data position information of each first node in the source cluster from the global synchronization information when the data synchronization method is asymmetric synchronization; a synchronization module, configured to determine the incremental data to be synchronized in each first node in the source cluster based on the synchronized data position information of the first node, and synchronize the incremental data to the second node in the target cluster corresponding to the first node; and an information update module, configured to update the global synchronization information based on the data synchronization result after the data synchronization is completed.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described cross-cluster data synchronization method.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described cross-cluster data synchronization method through the computer program.

[0017] In this embodiment, by constructing a global synchronization information record of the data synchronization method between the source cluster and the target cluster within the synchronization period, when it is symmetric synchronization, the location information of the synchronized data of each first node in the source cluster can be stored at the corresponding second node in the target cluster. When it is asymmetric synchronization, the location information of the synchronized data of each first node in the source cluster can be uniformly stored in the global synchronization information. Based on this method, in the next synchronization period, regardless of whether the topology of the target cluster changes, the location information of the synchronized data in the source cluster can be accurately obtained, thereby accurately determining the incremental data to be synchronized and avoiding data omission or duplicate synchronization. This scheme effectively solves the technical problem that traditional cross-cluster data synchronization schemes cannot perform data synchronization normally in cluster topology change scenarios. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart illustrating an optional cross-cluster data synchronization method according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of an optional data symmetric synchronization according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an optional asymmetric data synchronization according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of an optional cross-cluster data synchronization device according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0027] Symmetric synchronization: When the nodes (data shards) of the source cluster and the target cluster are symmetrical (i.e., the number of nodes in the source cluster and the target cluster are the same and the distribution area is consistent, and there is a one-to-one correspondence), the data of each node in the source cluster is synchronized to the corresponding node in the target cluster.

[0028] Asymmetric synchronization: When the nodes of the source cluster and the target cluster are asymmetric, it is necessary to determine one by one which node in the target cluster each piece of data in each node of the source cluster should be synchronized to. In the end, data of the same node in the source cluster may be synchronized to different nodes in the target cluster, and data of different nodes in the source cluster may be synchronized to the same node in the target cluster.

[0029] Atomic write: also known as atomic operation, refers to an operation in a multi-threaded environment that either completes entirely or fails entirely, without being interrupted by other threads. In a file system, atomic write means that the write operation to a file is indivisible; even in the event of a system crash, the file will not be in a partially written state.

[0030] Example 1

[0031] According to an embodiment of this application, a cross-cluster data synchronization method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] Figure 1 This is a flowchart illustrating a cross-cluster data synchronization method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0033] Step S102: Periodically acquire global synchronization information, wherein the global synchronization information includes the data synchronization method between the source cluster and the target cluster in the previous synchronization cycle of the current synchronization cycle;

[0034] Step S104: When the data synchronization method is symmetric synchronization, obtain the synchronized data location information of each first node in the source cluster from each second node in the target cluster.

[0035] Step S106: When the data synchronization method is asymmetric synchronization, determine the synchronized data location information of each first node in the source cluster from the global synchronization information.

[0036] Step S108: For each first node in the source cluster, determine the incremental data to be synchronized in the first node based on the synchronized data location information of the first node, and synchronize the incremental data to the second node corresponding to the first node in the target cluster.

[0037] Step S110: After data synchronization is completed, update the global synchronization information based on the data synchronization results.

[0038] This application constructs a global synchronization information record to track the data synchronization methods of the source cluster and the target cluster within the synchronization cycle. When it is symmetric synchronization, the location information of the synchronized data of each first node in the source cluster is stored at the corresponding second node in the target cluster. When it is asymmetric synchronization, the location information of the synchronized data of each first node in the source cluster is uniformly stored in the global synchronization information. Based on this method, in the next synchronization cycle, regardless of whether the topology of the target cluster changes, the location information of the synchronized data in the source cluster can be accurately obtained, thereby accurately determining the incremental data to be synchronized and avoiding data omission or duplicate synchronization.

[0039] The following section describes each step of the cross-cluster data synchronization method in conjunction with the specific implementation process.

[0040] As an optional implementation, the existence of global synchronization information is first detected.

[0041] If global synchronization information is not obtained, it can be determined that the source cluster and the target cluster have not performed data synchronization before. That is, this synchronization is the first time that the source cluster and the target cluster have performed data synchronization. In this case, for each first node in the source cluster, all data in the first node can be regarded as incremental data to be synchronized, and the incremental data is synchronized to the second node in the target cluster corresponding to the first node. After the data synchronization is completed, global synchronization information is generated based on the data synchronization result, and the data synchronization method of this synchronization is recorded.

[0042] Once global synchronization information is obtained, it can be determined that the source and target clusters have already synchronized data. At this point, the data synchronization method of the source and target clusters in the previous synchronization cycle can be directly read from the global synchronization information. If it is symmetric synchronization, the synchronized data location information of each first node in the source cluster can be obtained from each second node in the target cluster. If it is asymmetric synchronization, the synchronized data location information of each first node in the source cluster can be directly read from the global synchronization information. Then, for each first node in the source cluster, the incremental data to be synchronized in that first node can be determined based on the synchronized data location information of that first node, and the incremental data can be synchronized to the second node in the target cluster corresponding to that first node. Finally, after the data synchronization is completed, the global synchronization information is updated based on the data synchronization results.

[0043] As an optional implementation, when synchronizing incremental data to the second node corresponding to the first node in the target cluster, the following steps can be taken: First, obtain the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster; if the first distribution state and the second distribution state are symmetrical, write the incremental data to the second node corresponding to the first node in the target cluster in an atomic write manner, and write the end position of the incremental data in the first node as the synchronized data position information of the first node to the second node; if the first distribution state and the second distribution state are asymmetrical, for each sub-incremental data in the incremental data, determine the target second node corresponding to the sub-incremental data in the target cluster based on the hash value of the sub-incremental data, and write the sub-incremental data to the target second node.

[0044] Understandably, when performing data synchronization, we can first compare whether the topology of the source cluster and the target cluster is consistent, that is, whether the distribution of their respective nodes is symmetrical, and then determine the subsequent data synchronization method based on the comparison results.

[0045] Optionally, when the node distribution of the source and target clusters is symmetrical, a symmetrical synchronization method can be used for data synchronization. Specifically, since there is a one-to-one correspondence between the nodes in the source and target clusters, for each first node in the source cluster, the incremental data in that first node can be directly written to the corresponding second node in the target cluster. During the actual writing, an atomic write method can be used to ensure the integrity and reliability of the synchronized data. After the write is completed, the end position of the incremental data in the first node can be used as the synchronized data position information of the first node and written to the corresponding second node.

[0046] Optionally, when the node distribution of the source cluster and the target cluster is asymmetrical, an asymmetric synchronization method can be used for data synchronization. In order to ensure the balanced distribution of synchronized data in the target cluster as much as possible, this application does not directly establish a mapping relationship between the nodes of the source cluster and the target cluster for data synchronization at the node level. Instead, it further uses the individual data within a node as the synchronization granularity, and for each sub-incremental data (which may be related data logs) in the incremental data of each first node in the source cluster, it determines which second node in the target cluster to synchronize it to, and specifically provides the following two node selection methods.

[0047] The first method is primarily based on the idea of ​​hash modulo. Optionally, for each sub-incremental data in the incremental data of each first node in the source cluster, the hash value corresponding to the sub-incremental data can be determined first, and the number of second nodes in the target cluster can be determined simultaneously. Each second node is then sequentially numbered 0, 1, 2, ..., m. A modulo operation is performed on the hash value of the sub-incremental data and the number of second nodes in the target cluster to obtain the remainder n = hash value % number of nodes. The second node corresponding to the node number and the obtained remainder value is determined as the target second node corresponding to the sub-incremental data, and the sub-incremental data will be synchronized to the target second node.

[0048] The second approach is primarily based on the concept of consistent hashing. Optionally, a virtual hash ring can be constructed first, and all second nodes in the target cluster can be mapped onto the hash ring. Each second node corresponds to one or more virtual nodes on the hash ring. For each sub-incremental data in the incremental data of each first node in the source cluster, the hash value corresponding to the sub-incremental data is determined, and the target position of the hash value on the hash ring is determined. Then, the target virtual node closest to the target position is determined, and the second node corresponding to the target virtual node is the target second node corresponding to the sub-incremental data.

[0049] As an optional implementation, when updating the global synchronization information based on the data synchronization results, the following method can be adopted: When the first distribution state and the second distribution state are symmetrical, since the synchronized data position information of each first node has been written to the corresponding second node, it is only necessary to update the data synchronization method in the global synchronization information to symmetrical synchronization; When the first distribution state and the second distribution state are asymmetrical, it is not only necessary to update the data synchronization method in the global synchronization information to asymmetric synchronization, but also for each first node in the source cluster, it is necessary to update the global synchronization information with the end position of the incremental data of the first node as the synchronized data position information of the first node.

[0050] Figure 2 This is a schematic diagram of an optional symmetric data synchronization method, such as... Figure 2 As shown, the source cluster includes nodes A, B, and C, and the target cluster includes nodes X, Y, and Z. The node distribution of the two clusters is symmetrical. During data synchronization, the synchronized data location information in nodes A, B, and C is first obtained: A=100, B=60, C=40. Based on this information, the incremental data to be synchronized in node A are determined to be data 101 and data 102, in node B are data 61 and data 62, and in node C is data 41. Then, data 101 and data 102 are packaged into dataset 102 and atomically written to node X. After successful writing, the synchronized data location information corresponding to node A (A=102) is added to node X. Data 61 and data 62 are packaged into dataset 62 and atomically written to node Y. After successful writing, the synchronized data location information corresponding to node B (B=62) is added to node Y. Data 41 is packaged into dataset 41 and atomically written to node Z. After successful writing, the synchronized data location information corresponding to node C (C=41) is added to node Z. After data synchronization is complete, the global synchronization information is updated to symmetric synchronization.

[0051] Figure 3 This is a schematic diagram of an optional asymmetric data synchronization method, such as... Figure 3As shown, the source cluster includes nodes A and B, and the target cluster includes nodes X, Y, and Z. The node distribution of the two clusters is asymmetrical. During data synchronization, the synchronized data location information in nodes A and B is first obtained: A = 100, B = 60. Then, based on this information, the incremental data to be synchronized in node A is determined to be data 101, data 102, and data 103, and the incremental data to be synchronized in node B is data 41, data 42, and data 43. Next, based on the data hash values, data 101 and data 102 are written to node X, data 103 and data 43 are written to node Y, and data 41 and data 42 are written to node Z. After data synchronization is completed, the global synchronization information is updated to asymmetric synchronization, and the synchronized data location information A = 103 for node A and B = 43 for node B is added to the global synchronization information.

[0052] To further improve the adaptability of the data synchronization process to cluster topology change scenarios, this application provides a scheme for actively triggering data synchronization based on cluster topology changes, in addition to performing data synchronization according to a preset synchronization cycle.

[0053] Optionally, it is possible to monitor in real time whether the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster have changed; when the first distribution state and / or the second distribution state change, a new synchronization cycle is actively triggered to synchronize data.

[0054] In this embodiment, by constructing a global synchronization information record of the data synchronization method between the source cluster and the target cluster within the synchronization period, when it is symmetric synchronization, the location information of the synchronized data of each first node in the source cluster can be stored at the corresponding second node in the target cluster. When it is asymmetric synchronization, the location information of the synchronized data of each first node in the source cluster can be uniformly stored in the global synchronization information. Based on this method, in the next synchronization period, regardless of whether the topology of the target cluster changes, the location information of the synchronized data in the source cluster can be accurately obtained, thereby accurately determining the incremental data to be synchronized and avoiding data omission or duplicate synchronization. This scheme effectively solves the technical problem that traditional cross-cluster data synchronization schemes cannot perform data synchronization normally in cluster topology change scenarios.

[0055] Example 2

[0056] According to an embodiment of this application, a cross-cluster data synchronization apparatus for implementing the cross-cluster data synchronization method in Embodiment 1 is also provided, such as... Figure 4 As shown, the cross-cluster data synchronization device includes at least: an acquisition module 41, a first location determination module 42, a second location determination module 43, a synchronization module 44, and an information update module 45, wherein:

[0057] The acquisition module 41 is used to periodically acquire global synchronization information, which includes the data synchronization method between the source cluster and the target cluster in the previous synchronization cycle of the current synchronization cycle.

[0058] The first position determination module 42 is used to obtain the synchronized data position information of each first node in the source cluster from each second node in the target cluster when the data synchronization method is symmetric synchronization.

[0059] The second position determination module 43 is used to determine the synchronized data position information of each first node in the source cluster from the global synchronization information when the data synchronization method is asymmetric synchronization.

[0060] Synchronization module 44 is used to determine the incremental data to be synchronized in each first node in the source cluster based on the location information of the synchronized data of the first node, and synchronize the incremental data to the second node in the target cluster corresponding to the first node.

[0061] The information update module 45 is used to update the global synchronization information based on the data synchronization results after the data synchronization is completed.

[0062] The following section describes the functions of each module in the cross-cluster data synchronization device, using a specific implementation process as an example.

[0063] As an optional implementation, the acquisition module first checks whether global synchronization information exists.

[0064] If global synchronization information is not obtained, it can be determined that the source cluster and the target cluster have not performed data synchronization before. That is, this synchronization is the first time that the source cluster and the target cluster have performed data synchronization. In this case, for each first node in the source cluster, the synchronization module can treat all the data in the first node as incremental data to be synchronized and synchronize the incremental data to the second node in the target cluster corresponding to the first node. After the data synchronization is completed, the information update module generates global synchronization information based on the data synchronization result and records the data synchronization method of this synchronization.

[0065] Once global synchronization information is obtained, it can be determined that the source cluster and the target cluster have already synchronized data. At this point, the data synchronization method of the source cluster and the target cluster in the previous synchronization cycle is directly read from the global synchronization information. If it is symmetric synchronization, the first position determination module can obtain the synchronized data position information of each first node in the source cluster from each second node in the target cluster. If it is asymmetric synchronization, the second position determination module can directly read the synchronized data position information of each first node in the source cluster from the global synchronization information. Then, for each first node in the source cluster, the synchronization module can determine the incremental data to be synchronized in the first node based on the synchronized data position information of the first node, and synchronize the incremental data to the second node corresponding to the first node in the target cluster. Finally, after the data synchronization is completed, the information update module updates the global synchronization information based on the data synchronization result.

[0066] As an optional implementation, the synchronization module can synchronize incremental data to the second node corresponding to the first node in the target cluster through the following steps: First, obtain the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster; if the first distribution state and the second distribution state are symmetrical, write the incremental data to the second node corresponding to the first node in the target cluster in an atomic write manner, and write the end position of the incremental data in the first node as the synchronized data position information of the first node to the second node; if the first distribution state and the second distribution state are asymmetrical, for each sub-incremental data in the incremental data, determine the target second node corresponding to the sub-incremental data in the target cluster based on the hash value of the sub-incremental data, and write the sub-incremental data to the target second node.

[0067] Understandably, when performing data synchronization, we can first compare whether the topology of the source cluster and the target cluster is consistent, that is, whether the distribution of their respective nodes is symmetrical, and then determine the subsequent data synchronization method based on the comparison results.

[0068] Optionally, when the node distribution of the source and target clusters is symmetrical, the synchronization module can use a symmetrical synchronization method for data synchronization. Specifically, since there is a one-to-one correspondence between the nodes in the source and target clusters, for each first node in the source cluster, the incremental data in that first node can be directly written to the corresponding second node in the target cluster. During the writing process, an atomic write method can be used to ensure the integrity and reliability of the synchronized data. After the write is completed, the end position of the incremental data in the first node can be used as the synchronized data position information of the first node and written to the corresponding second node.

[0069] Optionally, when the node distribution of the source cluster and the target cluster is asymmetrical, the synchronization module can use an asymmetric synchronization method for data synchronization. To ensure a balanced distribution of synchronized data in the target cluster as much as possible, this application does not directly establish a mapping relationship between nodes in the source cluster and the target cluster for data synchronization at the node level. Instead, it further uses individual data entries within a node as the synchronization granularity, determining which second node in the target cluster each sub-incremental data entry (which can be related data logs) in the incremental data of each first node in the source cluster should be synchronized to. Specifically, it provides the following two node selection methods.

[0070] The first method is primarily based on the idea of ​​hash modulo. Optionally, for each sub-incremental data in the incremental data of each first node in the source cluster, the hash value corresponding to the sub-incremental data can be determined first, and the number of second nodes in the target cluster can be determined simultaneously. Each second node is then sequentially numbered 0, 1, 2, ..., m. A modulo operation is performed on the hash value of the sub-incremental data and the number of second nodes in the target cluster to obtain the remainder n = hash value % number of nodes. The second node corresponding to the node number and the obtained remainder value is determined as the target second node corresponding to the sub-incremental data, and the sub-incremental data will be synchronized to the target second node.

[0071] The second approach is primarily based on the concept of consistent hashing. Optionally, a virtual hash ring can be constructed first, and all second nodes in the target cluster can be mapped onto the hash ring. Each second node corresponds to one or more virtual nodes on the hash ring. For each sub-incremental data in the incremental data of each first node in the source cluster, the hash value corresponding to the sub-incremental data is determined, and the target position of the hash value on the hash ring is determined. Then, the target virtual node closest to the target position is determined, and the second node corresponding to the target virtual node is the target second node corresponding to the sub-incremental data.

[0072] As an optional implementation, the information update module can update the global synchronization information based on the data synchronization results in the following way: When the first distribution state and the second distribution state are symmetrical, since the synchronized data position information of each first node has been written to the corresponding second node, it is only necessary to update the data synchronization method in the global synchronization information to symmetrical synchronization; When the first distribution state and the second distribution state are asymmetrical, it is not only necessary to update the data synchronization method in the global synchronization information to asymmetric synchronization, but also for each first node in the source cluster, it is necessary to update the global synchronization information with the end position of the incremental data of the first node as the synchronized data position information of the first node.

[0073] To further improve the adaptability of the data synchronization process to cluster topology change scenarios, this application provides a scheme for actively triggering data synchronization based on cluster topology changes, in addition to performing data synchronization according to a preset synchronization cycle.

[0074] Optionally, the device of this application further includes a cluster topology detection module, which is used to monitor in real time whether the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster have changed; when the first distribution state and / or the second distribution state change, it actively triggers a new synchronization cycle to synchronize data.

[0075] It should be noted that each module in the cross-cluster data synchronization device in this application embodiment corresponds one-to-one with each implementation step of the cross-cluster data synchronization method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0076] Example 3

[0077] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the cross-cluster data synchronization method in embodiment 1.

[0078] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the cross-cluster data synchronization method in Embodiment 1 by running the computer program.

[0079] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the cross-cluster data synchronization method of embodiment 1 during runtime.

[0080] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the cross-cluster data synchronization method of Embodiment 1 through the computer program.

[0081] Specifically, the computer program executes the following steps during runtime: periodically acquiring global synchronization information, which includes the data synchronization method of the source cluster and the target cluster in the previous synchronization period; when the data synchronization method is symmetric synchronization, acquiring the synchronized data location information of each first node in the source cluster from each second node in the target cluster; when the data synchronization method is asymmetric synchronization, determining the synchronized data location information of each first node in the source cluster from the global synchronization information; for each first node in the source cluster, determining the incremental data to be synchronized in the first node based on the synchronized data location information of the first node, and synchronizing the incremental data to the second node corresponding to the first node in the target cluster; after data synchronization is completed, updating the global synchronization information based on the data synchronization result.

[0082] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 5 A hardware block diagram of an electronic device for implementing a cross-cluster data synchronization method is shown. Figure 5 As shown, the electronic device 50 may include one or more processors 502 (shown as 502a, 502b, ..., 502n in the figure) 502 (processor 502 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 504 for storing data, and a transmission device 506 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 5 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, electronic device 50 may also include... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown.

[0083] It should be noted that the aforementioned one or more processors 502 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element of the electronic device 50. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0084] The memory 504 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the cross-cluster data synchronization method in this embodiment. The processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, thereby implementing the aforementioned application vulnerability detection method. The memory 504 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 504 may further include memory remotely located relative to the processor 502, and these remote memories can be connected to the electronic device 50 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] The transmission device 506 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 50. In one example, the transmission device 506 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 506 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0086] The display may be, for example, a touchscreen liquid crystal display (LCD), which allows a user to interact with the user interface of the electronic device 50.

[0087] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0088] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0092] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0093] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A cross-cluster data synchronization method, characterized in that, include: Real-time monitoring of the first distribution status of each first node in the source cluster and the second distribution status of each second node in the target cluster; Data synchronization is performed according to a preset synchronization period, or by actively triggering a new synchronization period when the first distribution state and / or the second distribution state changes, based on the following steps: Obtain global synchronization information, wherein the global synchronization information includes the data synchronization method between the source cluster and the target cluster in the previous synchronization cycle of the current synchronization cycle; When the data synchronization method is symmetric synchronization, the synchronized data location information of each first node in the source cluster is obtained from each second node in the target cluster. When the data synchronization method is asymmetric synchronization, the synchronized data location information of each first node in the source cluster is determined from the global synchronization information; For each first node in the source cluster, the incremental data to be synchronized in the first node is determined based on the synchronized data location information of the first node, and the incremental data is synchronized to the second node in the target cluster corresponding to the first node; After data synchronization is complete, the global synchronization information is updated based on the data synchronization results.

2. The method according to claim 1, characterized in that, The method further includes: Without obtaining the global synchronization information, it is determined that the source cluster and the target cluster have not performed data synchronization. For each first node in the source cluster, all data in the first node is taken as incremental data to be synchronized, and the incremental data is synchronized to the second node in the target cluster corresponding to the first node. After data synchronization is completed, the global synchronization information is generated based on the data synchronization results.

3. The method according to claim 1, characterized in that, Synchronizing the incremental data to the second node in the target cluster corresponding to the first node includes: When the first distribution state and the second distribution state are symmetrical, the incremental data is written to the second node in the target cluster corresponding to the first node in an atomic write manner, and the end position of the incremental data in the first node is written to the second node as the synchronized data position information of the first node. When the first distribution state and the second distribution state are asymmetrical, for each sub-incremental data in the incremental data, the target second node corresponding to the sub-incremental data in the target cluster is determined based on the hash value of the sub-incremental data, and the sub-incremental data is written to the target second node.

4. The method according to claim 3, characterized in that, Determining the target second node in the target cluster corresponding to the sub-incremental data based on the hash value of the sub-incremental data includes: Determine the number of second nodes in the target cluster and assign sequential numbers to each second node; Perform a modulo operation on the hash value and the number of nodes to determine the second node corresponding to the node number and the obtained remainder value as the target second node corresponding to the sub-incremental data.

5. The method according to claim 3, characterized in that, Determining the target second node in the target cluster corresponding to the sub-incremental data based on the hash value of the sub-incremental data includes: Construct a hash ring and map all second nodes in the target cluster onto the hash ring, wherein each second node corresponds to at least one virtual node on the hash ring; Determine the target position of the hash value on the hash ring, and determine the target virtual node that is closest to the target position. Then, take the second node corresponding to the target virtual node as the target second node corresponding to the sub-incremental data.

6. The method according to claim 3, characterized in that, The global synchronization information is updated based on the data synchronization results, including: When the first distribution state and the second distribution state are symmetrical, the data synchronization method in the global synchronization information is updated to symmetrical synchronization. When the first distribution state and the second distribution state are asymmetrical, the data synchronization method in the global synchronization information is updated to asymmetric synchronization. For each first node in the source cluster, the end position of the incremental data of the first node in the first node is used as the synchronized data position information of the first node and updated to the global synchronization information.

7. A cross-cluster data synchronization device, characterized in that, include: The cluster topology detection module is used to monitor the first distribution status of each first node in the source cluster and the second distribution status of each second node in the target cluster in real time. When the first distribution status and / or the second distribution status change, a new synchronization cycle is actively triggered. The acquisition module is used to acquire global synchronization information within a preset synchronization period or a synchronization period actively triggered by the cluster topology detection module. The global synchronization information includes the data synchronization method of the source cluster and the target cluster in the previous synchronization period of the current synchronization period. The first location determination module is used to obtain the synchronized data location information of each first node in the source cluster from each second node in the target cluster when the data synchronization method is symmetric synchronization. The second location determination module is used to determine the synchronized data location information of each first node in the source cluster from the global synchronization information when the data synchronization method is asymmetric synchronization. The synchronization module is used to determine the incremental data to be synchronized in each first node in the source cluster based on the synchronized data location information of the first node, and synchronize the incremental data to the second node in the target cluster corresponding to the first node. The information update module is used to update the global synchronization information based on the data synchronization results after data synchronization is completed.

8. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the cross-cluster data synchronization method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the cross-cluster data synchronization method of any one of claims 1 to 6 through the computer program.

Citation Information

Patent Citations

  • Data synchronization method and device, equipment and medium

    CN116701540A

  • Data synchronization method and device, computer equipment and storage medium

    CN117009146A