Cross-cluster data synchronization method and device

Abstract

The invention discloses a cross-cluster data synchronization method and device. The method comprises the steps that global synchronization information is acquired periodically, and the global synchronization information comprises data synchronization modes of a source cluster and a target cluster in a previous synchronization period; when the data synchronization mode is symmetric synchronization, obtaining synchronized data position information of each first node in the source cluster from each second node in the target cluster; when the data synchronization mode is asymmetric synchronization, determining the synchronized data position of each first node from the global synchronization information; for each first node, determining incremental data to be synchronized in the first node according to the synchronized data position of the first node, and synchronizing the incremental data to a second node corresponding to the first node in the target cluster; and after data synchronization is completed, global synchronization information is updated according to a synchronization result. The technical problem that in a cluster topology change scene, a traditional cross-cluster data synchronization scheme cannot perform data synchronization normally is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data synchronization, in particular to a cross-cluster data synchronization method and device. BACKGROUND

[0002] In the current Internet era, real-time and consistency of data have become one of the key indicators for measuring system performance. With the continuous expansion of business scale, a single database cluster often cannot meet the storage and access requirements of massive data. Distributed databases are widely used due to their high availability, scalability and load balancing advantages. However, the complexity of distributed database architecture also brings challenges to data synchronization, especially in the cross-cluster synchronization scenario.

[0003] Traditional distributed database synchronization methods, such as real-time synchronization tools based on Redis, such as redis-shake, xpipe, etc., usually assume that the source cluster and the target cluster have symmetrical topology structure, i.e. the number of nodes and data shard layout of both are completely consistent. This premise simplifies the synchronization process, making the transmission and playback of data logs more direct, but in actual application environment, this assumption is often difficult to meet. For example, in the cold data backup, hot standby switching or resource optimization scenario, due to cost control and resource allocation considerations, the number of nodes and structure of the target cluster may be significantly less than or different from the source cluster, causing asymmetry in data synchronization.

[0004] The main problems of asymmetric data synchronization include: data consistency is difficult to guarantee, the incremental data of the source cluster may not be accurately mapped to the target cluster; synchronization strategy adjustment is not easy when the cluster topology changes, especially when the number of nodes increases or decreases, the existing synchronization process may fail. In order to solve these problems, existing synchronization methods often rely on complex mapping tables and additional metadata management, which not only increases the complexity of the system, but also may introduce additional maintenance costs and potential failure points.

[0005] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0006] The embodiments of the present application provide a cross-cluster data synchronization method and device to at least solve the technical problem that the traditional cross-cluster data synchronization scheme cannot normally perform data synchronization in the cluster topology change scenario.

[0007] According to an aspect of the embodiments of the present application, a cross-cluster data synchronization method is provided, comprising: periodically acquiring global synchronization information, wherein the global synchronization information comprises a data synchronization mode of a source cluster and a target cluster in a last synchronization period of a current synchronization period; in a case that the data synchronization mode is symmetric synchronization, acquiring synchronized data position information of each first node in the source cluster from each second node in the target cluster; in a case that the data synchronization mode is asymmetric synchronization, determining the synchronized data position information of each first node in the source cluster from the global synchronization information; for each first node in the source cluster, determining incremental data to be synchronized in the first node according to the synchronized data position information of the first node, and synchronizing the incremental data to a second node corresponding to the first node in the target cluster; and after the data synchronization is completed, updating the global synchronization information according to a data synchronization result.

[0008] Optionally, in a case that the global synchronization information is not acquired, it is determined that the source cluster and the target cluster have not performed data synchronization; for each first node in the source cluster, all data in the first node are regarded as incremental data to be synchronized, and the incremental data is synchronized to a second node corresponding to the first node in the target cluster; and after the data synchronization is completed, the global synchronization information is generated according to a data synchronization result.

[0009] Optionally, the synchronizing of the incremental data to the second node corresponding to the first node in the target cluster comprises: acquiring a first distribution state of each first node in the source cluster and a second distribution state of each second node in the target cluster; in a case that the first distribution state and the second distribution state are symmetric, writing the incremental data into the second node corresponding to the first node in the target cluster in an atomic writing manner, and writing an end position of the incremental data in the first node as the synchronized data position information of the first node into the second node; and in a case that the first distribution state and the second distribution state are asymmetric, for each sub-incremental data in the incremental data, determining a target second node corresponding to the sub-incremental data in the target cluster based on a hash value of the sub-incremental data, and writing the sub-incremental data into the target second node.

[0010] Optionally, the determining of the target second node corresponding to the sub-incremental data in the target cluster based on the hash value of the sub-incremental data comprises: determining a node number of the second nodes in the target cluster, and sequentially numbering each second node; performing a modulo operation on the hash value and the node number, and determining that a second node corresponding to a node number and a remainder value obtained by the modulo operation is the target second node corresponding to the sub-incremental data.

[0011] Optionally, determining the target second node corresponding to the sub-increment data in the target cluster based on the hash value of the sub-increment data comprises: constructing a hash ring and mapping all second nodes in the target cluster to the hash ring, wherein each second node corresponds to at least one virtual node on the hash ring; determining a target position of the hash value on the hash ring and determining a target virtual node closest to the target position, and taking the second node corresponding to the target virtual node as the target second node corresponding to the sub-increment data.

[0012] Optionally, updating the global synchronization information according to the data synchronization result comprises: in the case that the first distribution state and the second distribution state are symmetrical, updating the data synchronization mode in the global synchronization information to be symmetrical synchronization; in the case that the first distribution state and the second distribution state are asymmetrical, updating the data synchronization mode in the global synchronization information to be asymmetrical synchronization, and for each first node in the source cluster, updating the end position of the increment data of the first node in the first node as the synchronized data position information of the first node to the global synchronization information.

[0013] Optionally, the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster are monitored in real time to determine whether the first distribution state and / or the second distribution state change; when the first distribution state and / or the second distribution state change, a new synchronization period is triggered to perform data synchronization.

[0014] According to another aspect of the embodiments of the present application, a cross-cluster data synchronization device is further provided, comprising: an acquisition module configured to periodically acquire global synchronization information, wherein the global synchronization information comprises a data synchronization mode of a source cluster and a target cluster in a last synchronization period of a current synchronization period; a first position determination module configured to, in the case that the data synchronization mode is symmetrical synchronization, acquire synchronized data position information of each first node in the source cluster from each second node in the target cluster; a second position determination module configured to, in the case that the data synchronization mode is asymmetrical synchronization, determine the synchronized data position information of each first node in the source cluster from the global synchronization information; a synchronization module configured to, for each first node in the source cluster, determine increment data to be synchronized in the first node according to the synchronized data position information of the first node, and synchronize the increment data to a second node corresponding to the first node in the target cluster; and an information update module configured to, after the data synchronization is completed, update the global synchronization information according to a data synchronization result.

[0015] According to another aspect of the embodiments of the present application, a computer program product is further provided, which comprises a computer program, wherein the computer program is executed by a processor to implement the cross-cluster data synchronization method.

[0016] According to a further aspect of the embodiments of the present application, an electronic device is also provided, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the cross-cluster data synchronization method described above by the computer program.

[0017] In the embodiments of the present application, by constructing the global synchronization information record source cluster and the data synchronization mode of the target cluster in the synchronization period, when the synchronization is symmetric, the synchronized data position information of each first node in the source cluster can be stored in the corresponding each second node in the target cluster, and when the synchronization is asymmetric, the synchronized data position information of each first node in the source cluster can be uniformly stored in the global synchronization information. Based on this mode, in the next synchronization period, whether the topology state of the target cluster changes or not, the position information of the synchronized data in the source cluster can be accurately obtained, and then the incremental data to be synchronized can be accurately determined, so as to avoid data omission or repeated synchronization. The scheme effectively solves the technical problem that the traditional cross-cluster data synchronization scheme cannot normally perform data synchronization in the cluster topology change scenario. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and its description, which serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0019] Figure 1 is a flow diagram of an optional cross-cluster data synchronization method according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of an optional data symmetric synchronization according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional data asymmetric synchronization according to an embodiment of the present application;

[0022] Figure 4 is a structural schematic diagram of an optional cross-cluster data synchronization device according to an embodiment of the present application;

[0023] Figure 5 is a structural schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the person skilled in the art better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] In order to better understand the embodiments of the present application, first, the part of the nouns or terms appearing in the process of describing the embodiments of the present application are translated and explained as follows:

[0027] Symmetric synchronization: when the nodes (data shards) of the source cluster and the target cluster are symmetric (i.e. the number of nodes of the source cluster and the target cluster is the same and the distribution area is consistent, there is a one-to-one correspondence), the data of each node in the source cluster is synchronized to the corresponding node in the target cluster.

[0028] Asymmetric synchronization: when the nodes of the source cluster and the target cluster are asymmetric, it is necessary to determine one by one which node in the target cluster each piece of data in each node in the source cluster is synchronized to. Finally, the data of the same node in the source cluster may be synchronized to different nodes in the target cluster, and the data of different nodes in the source cluster may be synchronized to the same node in the target cluster.

[0029] Atomic write: also known as atomic operation, refers to an operation in a multi-threaded environment, which is either completed entirely or not at all, and will not be interrupted by other threads. In the file system, atomic write means that the write operation on the file is indivisible, that is, even in the case of system crash, the file will not be in a partially written state.

[0030] Embodiment 1

[0031] According to the embodiment of the present application, a cross-cluster data synchronization method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0032] Figure 1 is a flowchart of a cross-cluster data synchronization method according to an embodiment of the present application, as shown in Figure 1 the method comprises the following steps:

[0033] Step S102, periodically obtaining global synchronization information, wherein the global synchronization information includes the data synchronization mode of the source cluster and the target cluster in the last synchronization period of the current synchronization period;

[0034] Step S104, in the case of symmetric synchronization, obtaining the synchronized data position information of each first node in the source cluster from each second node in the target cluster;

[0035] Step S106, in the case of asymmetric synchronization, determining the synchronized data position information of each first node in the source cluster from the global synchronization information;

[0036] Step S108, for each first node in the source cluster, determining the incremental data to be synchronized in the first node according to the synchronized data position information of the first node, and synchronizing the incremental data to the second node corresponding to the first node in the target cluster;

[0037] Step S110, after the data synchronization is completed, updating the global synchronization information according to the data synchronization result.

[0038] The present application records the data synchronization mode of the source cluster and the target cluster in the synchronization period by constructing the global synchronization information, when it is symmetric synchronization, the synchronized data position information of each first node in the source cluster is stored at the corresponding second node in the target cluster, when it is asymmetric synchronization, the synchronized data position information of each first node in the source cluster is uniformly stored in the global synchronization information, based on this mode, in the next synchronization period, whether the topology state of the target cluster changes or not, the position information of the synchronized data in the source cluster can be accurately obtained, and then the incremental data to be synchronized is accurately determined, avoiding data omission or repeated synchronization.

[0039] The steps of the cross-cluster data synchronization method will be described in detail below in combination with the specific implementation process.

[0040] As an optional implementation, first, it is detected whether the global synchronization information exists or not.

[0041] In the case where the global synchronization information is not acquired, it can be determined that the source cluster and the target cluster have not performed data synchronization, i.e., the current synchronization is the first data synchronization for the source cluster and the target cluster. In this case, for each first node in the source cluster, all data in the first node can be regarded as incremental data to be synchronized, and the incremental data is synchronized to a second node corresponding to the first node in the target cluster. After the data synchronization is completed, the global synchronization information is generated according to the data synchronization result, and the data synchronization mode of the current synchronization is recorded.

[0042] In the case where the global synchronization information is acquired, it can be determined that the source cluster and the target cluster have performed data synchronization. At this time, the data synchronization mode of the source cluster and the target cluster in the last synchronization period of the current synchronization period is directly read from the global synchronization information. If the data synchronization mode is symmetric synchronization, the synchronized data position information of each first node in the source cluster can be acquired from each second node in the target cluster. If the data synchronization mode is asymmetric synchronization, the synchronized data position information of each first node in the source cluster can be directly read from the global synchronization information. Then, for each first node in the source cluster, the incremental data to be synchronized in the first node can be determined according to the synchronized data position information of the first node, and the incremental data is synchronized to a second node corresponding to the first node in the target cluster. Finally, after the data synchronization is completed, the global synchronization information is updated according to the data synchronization result.

[0043] As an optional implementation, when the incremental data is synchronized to the second node corresponding to the first node in the target cluster, the following steps can be implemented: first, the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster are acquired; in the case where the first distribution state and the second distribution state are symmetric, the incremental data is written to the second node corresponding to the first node in the target cluster in an atomic writing manner, and the end position of the incremental data in the first node is written to the second node as the synchronized data position information of the first node; in the case where the first distribution state and the second distribution state are asymmetric, for each sub-incremental data in the incremental data, a target second node corresponding to the sub-incremental data in the target cluster is determined based on the hash value of the sub-incremental data, and the sub-incremental data is written to the target second node.

[0044] It can be understood that when data synchronization is performed, the topological structures of the source cluster and the target cluster can be compared first, i.e., whether the node distribution states are symmetric, and the subsequent data synchronization mode is determined according to the comparison result.

[0045] Optionally, when the node distribution states of the source cluster and the target cluster are symmetrical, a symmetrical synchronization mode can be adopted to synchronize data. Specifically, since the nodes in the source cluster and the target cluster are one-to-one corresponding, for each first node in the source cluster, the incremental data in the first node can be directly written into a second node corresponding thereto in the target cluster; when writing specifically, an atomic writing mode can be adopted to ensure the integrity and reliability of the synchronized data; after writing is completed, the end position of the incremental data in the first node can be written into the corresponding second node as the synchronized data position information of the first node.

[0046] Optionally, when the node distribution states of the source cluster and the target cluster are asymmetrical, an asymmetrical synchronization mode can be adopted to synchronize data. In order to ensure the balanced distribution of the synchronized data in the target cluster as much as possible, the present application does not directly establish a mapping relationship between the nodes of the source cluster and the target cluster to synchronize data with the node as the synchronization granularity, but further takes a single piece of data in a node as the synchronization granularity, determines for each piece of sub-incremental data (which can be a related data log) in the incremental data of each first node in the source cluster which second node in the target cluster to synchronize to, and specifically provides the following two node selection modes.

[0047] The first mode is mainly based on the hash modulo idea. Optionally, for each piece of sub-incremental data in the incremental data of each first node in the source cluster, the hash value corresponding to the piece of sub-incremental data can be determined first, and the number of nodes of the second nodes in the target cluster is determined, and the second nodes are sequentially numbered 0, 1, 2, …, m; the hash value of the sub-incremental data and the number of nodes of the second nodes in the target cluster are subjected to modulo operation to obtain a remainder value n = hash value % node number, and the second node corresponding to the node number corresponding to the obtained remainder value is determined as the target second node corresponding to the sub-incremental data, and the piece of sub-incremental data will be synchronized to the target second node.

[0048] The second mode is mainly based on the consistent hash idea. Optionally, a virtual hash ring can be constructed first, and all second nodes in the target cluster are mapped onto the hash ring, wherein each second node corresponds to one or more virtual nodes on the hash ring; for each piece of sub-incremental data in the incremental data of each first node in the source cluster, the hash value corresponding to the piece of sub-incremental data is determined, and the target position of the hash value on the hash ring is determined, and then the target virtual node closest to the target position is determined, and the second node corresponding to the target virtual node is the target second node corresponding to the piece of sub-incremental data.

[0049] As an optional implementation, when updating the global synchronization information according to the data synchronization result, the following method can be adopted: in the case of symmetry between the first distribution state and the second distribution state, since the synchronized data position information of each first node has been written into the corresponding second node, at this time, it is only necessary to update the data synchronization mode in the global synchronization information to symmetric synchronization; in the case of asymmetry between the first distribution state and the second distribution state, not only is it necessary to update the data synchronization mode in the global synchronization information to asymmetric synchronization, but also for each first node in the source cluster, it is necessary to update the end position of the incremental data in the first node of the first node as the synchronized data position information of the first node to the global synchronization information.

[0050] Figure 2 As an optional data symmetric synchronization diagram, as shown in Figure 2 the source cluster includes node A, node B and node C, the target cluster includes node X, node Y and node Z, and the two cluster node distribution states are symmetric. When data synchronization is performed, first, the synchronized data position information in nodes A, B and C is obtained: A = 100, B = 60, C = 40; then, based on the information, it is determined that the incremental data to be synchronized in node A is data 101 and data 102, the incremental data to be synchronized in node B is data 61 and data 62, and the incremental data to be synchronized in node C is data 41; thereafter, data 101 and data 102 are packaged as data set 102 and written into node X in an atomic writing manner, after successful writing, the synchronized data position information A = 102 corresponding to node A is added in node X, data 61 and data 62 are packaged as data set 62 and written into node Y in an atomic writing manner, after successful writing, the synchronized data position information B = 62 corresponding to node B is added in node Y, data 41 is packaged as data set 41 and written into node Z in an atomic writing manner, after successful writing, the synchronized data position information C = 41 corresponding to node C is added in node Z; after data synchronization is completed, the global synchronization information is updated to symmetric synchronization.

[0051] Figure 3 As an optional data asymmetric synchronization diagram, as shown in Figure 3As shown, the source cluster includes node A and node B, and the target cluster includes node X, node Y and node Z, and the distribution states of the two clusters are asymmetric. When data synchronization is performed, the synchronized data position information in node A and node B is first acquired: A=100, B=60; then based on the information, the incremental data to be synchronized in node A is determined as data 101, data 102 and data 103, and the incremental data to be synchronized in node B is determined as data 41, data 42 and data 43; then based on the data hash values, data 101 and data 102 are written into node X, data 103 and data 43 are written into node Y, and data 41 and data 42 are written into node Z; after the data synchronization is completed, the global synchronization information is updated as asymmetric synchronization, and the synchronized data position information A=103 corresponding to node A and the synchronized data position information B=43 corresponding to node B are added in the global synchronization information.

[0052] In order to further improve the adaptability of the data synchronization process and the cluster topology change scenario, in addition to performing data synchronization according to the preset synchronization period, the application also provides a scheme of actively triggering data synchronization based on cluster topology change.

[0053] Optionally, it can be monitored in real time whether the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster change; when the first distribution state and / or the second distribution state change, a new synchronization period is actively triggered to perform data synchronization.

[0054] In the embodiments of the application, by constructing the global synchronization information to record the data synchronization mode of the source cluster and the target cluster in the synchronization period, when the synchronization is symmetric, the synchronized data position information of each first node in the source cluster can be stored at the corresponding each second node in the target cluster, and when the synchronization is asymmetric, the synchronized data position information of each first node in the source cluster can be uniformly stored in the global synchronization information. Based on this mode, in the next synchronization period, whether the topology state of the target cluster changes or not, the position information of the synchronized data in the source cluster can be accurately acquired, and then the incremental data to be synchronized can be accurately determined, so as to avoid data omission or repeated synchronization. The scheme effectively solves the technical problem that the traditional cross-cluster data synchronization scheme cannot normally perform data synchronization in the cluster topology change scenario.

[0055] Embodiment 2

[0056] According to the embodiments of the application, a cross-cluster data synchronization device for implementing the cross-cluster data synchronization method in embodiment 1 is also provided, as shown in the figure. Figure 4 The cross-cluster data synchronization device at least includes an acquisition module 41, a first position determination module 42, a second position determination module 43, a synchronization module 44 and an information update module 45, wherein:

[0057] The acquisition module 41 is configured to periodically acquire global synchronization information, wherein the global synchronization information comprises a data synchronization mode of the source cluster and the target cluster in a last synchronization period of a current synchronization period.

[0058] The first position determination module 42 is configured to acquire, in a case where the data synchronization mode is symmetric synchronization, synchronized data position information of each first node in the source cluster from each second node in the target cluster.

[0059] The second position determination module 43 is configured to determine, in a case where the data synchronization mode is asymmetric synchronization, the synchronized data position information of each first node in the source cluster from the global synchronization information.

[0060] The synchronization module 44 is configured to determine, for each first node in the source cluster, incremental data to be synchronized in the first node according to the synchronized data position information of the first node, and synchronize the incremental data to a second node corresponding to the first node in the target cluster.

[0061] The information update module 45 is configured to update the global synchronization information according to a data synchronization result after the data synchronization is completed.

[0062] The functions of the modules of the cross-cluster data synchronization device will be described below in combination with a specific implementation process.

[0063] As an optional implementation, the acquisition module first detects whether the global synchronization information exists at present.

[0064] In a case where the global synchronization information is not acquired, it can be determined that the source cluster and the target cluster have not performed data synchronization, i.e., the present synchronization is the first data synchronization of the source cluster and the target cluster. In this case, for each first node in the source cluster, the synchronization module can synchronize all data in the first node as incremental data to be synchronized to a second node corresponding to the first node in the target cluster. After the data synchronization is completed, the information update module generates the global synchronization information according to a data synchronization result to record the data synchronization mode of the present synchronization.

[0065] In the case of obtaining the global synchronization information, it can be determined that the source cluster and the target cluster have performed data synchronization, at this time, the data synchronization mode of the source cluster and the target cluster in the last synchronization period of the current synchronization period is directly read from the global synchronization information, if it is symmetric synchronization, the first position determination module can obtain the synchronized data position information of each first node in the source cluster from each second node in the target cluster, if it is asymmetric synchronization, the second position determination module can directly read the synchronized data position information of each first node in the source cluster from the global synchronization information; then, for each first node in the source cluster, the synchronization module can determine the incremental data to be synchronized in the first node according to the synchronized data position information of the first node, and synchronize the incremental data to the second node corresponding to the first node in the target cluster; finally, after the data synchronization is completed, the information updating module updates the global synchronization information according to the data synchronization result.

[0066] As an optional implementation, when the synchronization module synchronizes the incremental data to the second node corresponding to the first node in the target cluster, it can be realized by the following steps: first, obtain the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster; in the case of symmetry of the first distribution state and the second distribution state, write the incremental data into the second node corresponding to the first node in the target cluster in an atomic write manner, and write the end position of the incremental data in the first node as the synchronized data position information of the first node into the second node; in the case of asymmetry of the first distribution state and the second distribution state, for each sub-incremental data in the incremental data, determine the target second node corresponding to the sub-incremental data in the target cluster based on the hash value of the sub-incremental data, and write the sub-incremental data into the target second node.

[0067] It can be understood that when performing data synchronization, the topological structure of the source cluster and the target cluster can be compared first, that is, whether the node distribution state of each other is symmetric, and then the subsequent data synchronization mode is determined according to the comparison result.

[0068] Optionally, when the node distribution state of the source cluster and the target cluster is symmetric, the synchronization module can perform data synchronization in a symmetric synchronization mode. Specifically, since the nodes in the source cluster and the target cluster are one-to-one corresponding, for each first node in the source cluster, the incremental data in the first node can be directly written into the second node corresponding to it in the target cluster; when writing specifically, an atomic write manner can be used to ensure the integrity and reliability of the synchronized data; after writing is completed, the end position of the incremental data in the first node can be written into the corresponding second node as the synchronized data position information of the first node.

[0069] Optionally, when the node distribution states of the source cluster and the target cluster are asymmetric, the synchronization module can perform data synchronization in an asymmetric synchronization manner. In order to ensure as much as possible that the synchronized data in the target cluster is balanced, the present application does not directly take the node as the synchronization granularity, establish the mapping relationship between the nodes of the source cluster and the target cluster to perform data synchronization, but further takes a single piece of data in the node as the synchronization granularity, determines for each piece of sub-incremental data (which can be a relevant data log) in the incremental data of each first node in the source cluster which second node in the target cluster to synchronize to, and specifically provides the following two node selection manners.

[0070] The first manner is mainly based on the hash modulo idea. Optionally, for each piece of sub-incremental data in the incremental data of each first node in the source cluster, the hash value corresponding to the piece of sub-incremental data can be determined first, and the number of nodes of the second nodes in the target cluster is determined, and the second nodes are sequentially numbered 0, 1, 2, …, m; the hash value of the sub-incremental data and the number of nodes of the second nodes in the target cluster are subjected to modulo operation to obtain a remainder value n = hash value % node number, and the second node corresponding to the node number and the obtained remainder value is determined as the target second node corresponding to the sub-incremental data, and the piece of sub-incremental data will be synchronized to the target second node.

[0071] The second manner is mainly based on the consistent hashing idea. Optionally, a virtual hash ring can be constructed first, and all the second nodes in the target cluster are mapped to the hash ring, wherein each second node corresponds to one or more virtual nodes on the hash ring; for each piece of sub-incremental data in the incremental data of each first node in the source cluster, the hash value corresponding to the piece of sub-incremental data is determined, and the target position of the hash value on the hash ring is determined, and then the target virtual node closest to the target position is determined, and the second node corresponding to the target virtual node is the target second node corresponding to the piece of sub-incremental data.

[0072] As an optional implementation, when updating the global synchronization information according to the data synchronization result, the information updating module can adopt the following manner: in the case that the first distribution state and the second distribution state are symmetric, since the synchronized data position information of each first node has been written into the corresponding second node, at this time, only the data synchronization manner in the global synchronization information needs to be updated to symmetric synchronization; in the case that the first distribution state and the second distribution state are asymmetric, not only the data synchronization manner in the global synchronization information needs to be updated to asymmetric synchronization, but also for each first node in the source cluster, the end position of the incremental data of the first node in the first node needs to be updated to the global synchronization information as the synchronized data position information of the first node.

[0073] In order to further improve the adaptability of the data synchronization process and the cluster topology change scenario, in addition to performing data synchronization according to the preset synchronization period, the application also provides a scheme for actively triggering data synchronization based on cluster topology change.

[0074] Optionally, the device further comprises a cluster topology detection module for monitoring whether the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster change in real time; when the first distribution state and / or the second distribution state changes, a new synchronization period is actively triggered to perform data synchronization.

[0075] It should be noted that the modules in the cross-cluster data synchronization device in the embodiments of the application correspond one by one to the implementation steps of the cross-cluster data synchronization method in Embodiment 1. Since Embodiment 1 has been described in detail, the details not embodied in this embodiment can be referred to Embodiment 1, and will not be described in detail here.

[0076] Embodiment 3

[0077] According to the embodiments of the application, a computer program product is also provided, which comprises a computer program, wherein the computer program is executed by a processor to implement the cross-cluster data synchronization method in Embodiment 1.

[0078] According to the embodiments of the application, a non-volatile storage medium is also provided, which comprises a stored computer program, wherein a device in which the non-volatile storage medium is located executes the cross-cluster data synchronization method in Embodiment 1 by running the computer program.

[0079] According to the embodiments of the application, a processor is also provided, which is used to run a computer program, wherein the computer program is executed to implement the cross-cluster data synchronization method in Embodiment 1 when running.

[0080] According to the embodiments of the application, an electronic device is also provided, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the cross-cluster data synchronization method in Embodiment 1 by the computer program.

[0081] Specifically, the computer program runs to implement the following steps: periodically acquiring global synchronization information, wherein the global synchronization information includes the data synchronization mode of the source cluster and the target cluster in the last synchronization period of the current synchronization period; in the case of symmetric synchronization, obtaining the synchronized data position information of each first node in the source cluster from each second node in the target cluster; in the case of asymmetric synchronization, determining the synchronized data position information of each first node in the source cluster from the global synchronization information; for each first node in the source cluster, determining the incremental data to be synchronized in the first node according to the synchronized data position information of the first node, and synchronizing the incremental data to the second node corresponding to the first node in the target cluster; after the data synchronization is completed, updating the global synchronization information according to the data synchronization result.

[0082] As an optional implementation, the electronic device can exist in the form of a mobile terminal, a computer terminal or the like. Figure 5 A hardware structure block diagram of an electronic device for implementing a cross-cluster data synchronization method is shown. As shown in the figure, Figure 5 The electronic device 50 can include one or more processors 502 (the processor 502 can include but is not limited to a microprocessor MCU or a programmable logic device FPGA processing device), a memory 504 for storing data, and a transmission device 506 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports in the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only schematic, and does not limit the structure of the above-mentioned electronic device. For example, the electronic device 50 can include more or fewer components than Figure 5 shown in the figure, or have a different configuration than Figure 5 shown in the figure.

[0083] It should be noted that the one or more processors 502 and / or other data processing circuits described above can be referred to herein as "data processing circuits" in general. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the data processing circuit can be a single independent processing module, or all or part of any one of the other elements combined into the electronic device 50. As referred to in the embodiments of the present application, the data processing circuit is a processor control (for example, selection of a variable resistance terminal path connected to an interface).

[0084] The memory 504 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the data synchronization method across clusters in the embodiments of the present application, and the processor 502 executes various functional applications and data processing, i.e., implements the vulnerability detection method of the application program, by running the software programs and modules stored in the memory 504. The memory 504 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 504 can further include memories remotely arranged with respect to the processor 502, which can be connected to the electronic device 50 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0085] The transmission device 506 is used to receive or send data via a network. The specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the electronic device 50. In one example, the transmission device 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 506 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0086] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the electronic device 50.

[0087] The above-mentioned embodiment numbers are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0088] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0089] In the several embodiments of the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the above-mentioned device embodiments are only schematic, for example, the division of units can be a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0090] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0091] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of a software functional unit.

[0092] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, the essential part or contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various program code storage media.

[0093] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for synchronizing data across clusters, the method comprising: The method comprises: periodically acquiring global synchronization information, wherein the global synchronization information comprises a data synchronization mode of a source cluster and a target cluster in a last synchronization period of a current synchronization period; in a case where the data synchronization mode is symmetric synchronization, acquiring synchronized data position information of each first node in the source cluster from each second node in the target cluster; in a case where the data synchronization mode is asymmetric synchronization, determining the synchronized data position information of each first node in the source cluster from the global synchronization information; for each first node in the source cluster, determining incremental data to be synchronized in the first node according to the synchronized data position information of the first node, and synchronizing the incremental data to a second node corresponding to the first node in the target cluster; after data synchronization is completed, updating the global synchronization information according to a data synchronization result.

2. The method of claim 1, wherein, The method further comprises: in a case where the global synchronization information is not acquired, determining that the source cluster and the target cluster have not performed data synchronization; for each first node in the source cluster, regarding all data in the first node as incremental data to be synchronized, and synchronizing the incremental data to a second node corresponding to the first node in the target cluster; after data synchronization is completed, generating the global synchronization information according to a data synchronization result.

3. The method of claim 1, wherein, Synchronizing the incremental data to the second node corresponding to the first node in the target cluster comprises: acquiring a first distribution state of each first node in the source cluster and a second distribution state of each second node in the target cluster; in a case where the first distribution state and the second distribution state are symmetric, writing the incremental data into the second node corresponding to the first node in the target cluster in an atomic writing manner, and writing an end position of the incremental data in the first node as the synchronized data position information of the first node into the second node; in a case where the first distribution state and the second distribution state are asymmetric, for each sub-incremental data in the incremental data, determining a target second node corresponding to the sub-incremental data in the target cluster based on a hash value of the sub-incremental data, and writing the sub-incremental data into the target second node.

4. The method of claim 3, wherein, Determining the target second node corresponding to the sub-incremental data in the target cluster based on the hash value of the sub-incremental data comprises: determining a number of second nodes in the target cluster, and sequentially numbering each second node; performing a modulo operation on the hash value and the number of second nodes, and determining that a second node corresponding to a node number and a remainder value obtained is the target second node corresponding to the sub-incremental data.

5. The method of claim 3, wherein, Determining the target second node corresponding to the sub-incremental data in the target cluster based on the hash value of the sub-incremental data comprises: constructing a hash ring, and mapping all second nodes in the target cluster to the hash ring, wherein each second node corresponds to at least one virtual node on the hash ring; determining a target position of the hash value on the hash ring, and determining a target virtual node closest to the target position, and taking a second node corresponding to the target virtual node as a target second node corresponding to the sub-incremental data.

6. The method of claim 3, wherein, updating the global synchronization information according to a data synchronization result, including: in a case where the first distribution state and the second distribution state are symmetrical, updating a data synchronization mode in the global synchronization information as symmetrical synchronization; in a case where the first distribution state and the second distribution state are asymmetrical, updating the data synchronization mode in the global synchronization information as asymmetrical synchronization, and for each first node in the source cluster, updating a tail position of incremental data of the first node in the first node as synchronized data position information of the first node to the global synchronization information.

7. The method of claim 1, wherein, The method further includes: monitoring whether the first distribution state of each first node in the source cluster and the second distribution state of each second node in the target cluster change in real time; when the first distribution state and / or the second distribution state change, actively triggering a new synchronization period for data synchronization.

8. A cross-cluster data synchronization device, characterized in that, including: an acquisition module configured to periodically acquire global synchronization information, wherein the global synchronization information includes a data synchronization mode of a source cluster and a target cluster in a last synchronization period of a current synchronization period; a first position determination module configured to, in a case where the data synchronization mode is symmetrical synchronization, acquire synchronized data position information of each first node in the source cluster from each second node in the target cluster; a second position determination module configured to, in a case where the data synchronization mode is asymmetrical synchronization, determine the synchronized data position information of each first node in the source cluster from the global synchronization information; a synchronization module configured to, for each first node in the source cluster, determine incremental data to be synchronized in the first node according to the synchronized data position information of the first node, and synchronize the incremental data to a second node corresponding to the first node in the target cluster; an information update module configured to, after data synchronization is completed, update the global synchronization information according to a data synchronization result.

9. A computer program product, characterised in that, including: a computer program, wherein the computer program is executed by a processor to implement the cross-cluster data synchronization method of any one of claims 1 to 7.

10. An electronic device, comprising: including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the cross-cluster data synchronization method of any one of claims 1 to 7 by the computer program.

Images