Data synchronization method and device, computer device, storage medium and program product

Abstract

This application relates to a data synchronization method, apparatus, computer device, storage medium, and program product. The method includes: receiving a data synchronization request from a target storage node, the data synchronization request carrying a first data version tree, the first data version tree being constructed by the target storage node based on the version numbers of the stored data; matching the first data version tree with a local second data version tree to obtain a data path where the first data version tree and the second data version tree differ, the second data version tree being constructed based on the version numbers of the locally stored data; determining, based on the data path, data to be synchronized with the target storage node; and performing inter-node data synchronization processing on the data to be synchronized, the first data version tree, and the second data version tree. This method can improve data synchronization efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data synchronization method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Traditional distributed system data synchronization schemes, for performance-critical read scenarios, store configuration data in external storage systems or local memory. When nodes in the distributed system need to read configuration data, they query the external database or local memory to retrieve the data. For example, in high-performance scenarios, each node in the distributed system loads all data into memory, so that when reading configuration data, only data needs to be read from memory, without accessing a remote database.

[0003] However, in high-performance data reading scenarios, when configuration data changes, a synchronization mechanism is needed to ensure that the configuration data in memory and the configuration data in the database remain consistent. However, all nodes in a distributed system need to synchronize the full amount of configuration data every time, which leads to low data synchronization efficiency between nodes. Summary of the Invention

[0004] Therefore, it is necessary to provide a data synchronization method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve synchronization efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a data synchronization method. The method includes:

[0006] Receive a data synchronization request from a target storage node, the data synchronization request carrying a first data version tree, the first data version tree being constructed by the target storage node based on the version number of the stored data;

[0007] The first data version tree is matched with the local second data version tree to obtain the data paths that differ between the first data version tree and the second data version tree, wherein the second data version tree is constructed based on the version number of the data stored locally;

[0008] Based on the data path, determine the data to be synchronized with the target storage node;

[0009] Data synchronization processing is performed between nodes on the data to be synchronized, the first data version tree, and the second data version tree.

[0010] Secondly, this application also provides a data synchronization device. The device includes:

[0011] A receiving module is used to receive a data synchronization request from a target storage node. The data synchronization request carries a first data version tree, which is constructed by the target storage node based on the version number of the stored data.

[0012] The matching module is used to match the first data version tree with the local second data version tree to obtain the data paths that differ between the first data version tree and the second data version tree, wherein the second data version tree is constructed based on the version number of the data stored locally;

[0013] The determination module is used to determine the data to be synchronized with the target storage node based on the data path;

[0014] The synchronization module is used to perform data synchronization processing between nodes on the data to be synchronized, the first data version tree, and the second data version tree.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0016] Receive a data synchronization request from a target storage node, the data synchronization request carrying a first data version tree, the first data version tree being constructed by the target storage node based on the version number of the stored data;

[0017] The first data version tree is matched with the local second data version tree to obtain the data paths that differ between the first data version tree and the second data version tree, wherein the second data version tree is constructed based on the version number of the data stored locally;

[0018] Based on the data path, determine the data to be synchronized with the target storage node;

[0019] Data synchronization processing is performed between nodes on the data to be synchronized, the first data version tree, and the second data version tree.

[0020] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0021] Receive a data synchronization request from a target storage node, the data synchronization request carrying a first data version tree, the first data version tree being constructed by the target storage node based on the version number of the stored data;

[0022] The first data version tree is matched with the local second data version tree to obtain the data paths that differ between the first data version tree and the second data version tree, wherein the second data version tree is constructed based on the version number of the data stored locally;

[0023] Based on the data path, determine the data to be synchronized with the target storage node;

[0024] Data synchronization processing is performed between nodes on the data to be synchronized, the first data version tree, and the second data version tree.

[0025] Fifthly, this application also provides a computer program product. The computer program product includes a computer program, which, when executed by a processor, performs the following steps:

[0026] Receive a data synchronization request from a target storage node, the data synchronization request carrying a first data version tree, the first data version tree being constructed by the target storage node based on the version number of the stored data;

[0027] The first data version tree is matched with the local second data version tree to obtain the data paths that differ between the first data version tree and the second data version tree, wherein the second data version tree is constructed based on the version number of the data stored locally;

[0028] Based on the data path, determine the data to be synchronized with the target storage node;

[0029] Data synchronization processing is performed between nodes on the data to be synchronized, the first data version tree, and the second data version tree.

[0030] The aforementioned data synchronization method, apparatus, computer equipment, storage medium, and computer program product receive a first data version tree carried in a data synchronization request from a target storage node. This first data version tree is constructed by the target storage node based on the version number of the stored data, while a second data version tree of the local storage node is constructed based on the version number of the locally stored data. This allows for matching the first data version tree with the local second data version tree, quickly and accurately determining whether the data on the two storage nodes is consistent, and accurately detecting data paths with differences between the first and second data version trees to obtain the data to be synchronized between the two storage nodes. By performing data synchronization processing between nodes on the data to be synchronized, the first data version tree, and the second data version tree, only a portion of the data needs to be synchronized, rather than all of it, thus reducing the amount of data synchronized and improving the efficiency of data synchronization. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the application environment of a data synchronization method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating a data synchronization method in one embodiment;

[0033] Figure 3 This is a schematic diagram of the first data version tree in one embodiment;

[0034] Figure 4 This is a schematic diagram of the global data version tree in another embodiment;

[0035] Figure 5 This is a schematic diagram of a partitioned data version tree in another embodiment;

[0036] Figure 6 This is a schematic diagram illustrating matching according to data paths in one embodiment;

[0037] Figure 7 This is a schematic diagram illustrating data synchronization between multiple storage nodes in a distributed cluster in one embodiment.

[0038] Figure 8 This is a schematic diagram illustrating the principle of a data synchronization method in one embodiment;

[0039] Figure 9 This is a schematic diagram illustrating the matching between the first data version tree and the second data version tree according to hierarchy in one embodiment;

[0040] Figure 10 This is a schematic diagram illustrating the construction of a global data version tree and a partitioned data version tree in one embodiment.

[0041] Figure 11 This is a flowchart illustrating a data synchronization method in one embodiment;

[0042] Figure 12 This is a timing diagram of a data synchronization method in one embodiment;

[0043] Figure 13 This is a structural block diagram of a data synchronization device in one embodiment;

[0044] Figure 14 This is an internal structural diagram of a computer device in one embodiment;

[0045] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] The data synchronization method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, local storage node 102 communicates with target storage node 104 via a network. The data storage system can store data that the target storage node 104 needs to process. The data storage system can be integrated onto the target storage node 104 or local storage node 102, or it can be placed in the cloud or on other storage nodes. Local storage node 102 receives a data synchronization request from target storage node 104. The data synchronization request carries a first data version tree, which is constructed by target storage node 104 based on the version number of the stored data. Local storage node 102 matches the first data version tree with its local second data version tree to obtain data paths where there are differences between the first and second data version trees. The second data version tree is constructed based on the version number of the locally stored data. Based on the data path, local storage node 102 determines the data to be synchronized with target storage node 104. Local storage node 102 performs inter-node data synchronization processing on the data to be synchronized, the first data version tree, and the second data version tree. Both local storage node 102 and target storage node 104 are computer devices, specifically terminals or servers. The terminal can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0048] In one embodiment, such as Figure 2 As shown, a data synchronization method is provided, which can be applied to... Figure 1 Taking the local storage node in the example, the following steps are included:

[0049] Step S202: Receive a data synchronization request from the target storage node. The data synchronization request carries a first data version tree, which is constructed by the target storage node based on the version number of the stored data.

[0050] Among them, the data synchronization request is the information in which the target storage node requests to synchronize data with the local storage node. Specifically, it can be to synchronize the data of the target storage node to the local storage node, or to synchronize the data of the local storage node to the target storage node.

[0051] A data synchronization request can be a single request or include multiple synchronization requests. When a data synchronization request is a single request, it carries the first data version tree of the target storage node. When a data synchronization request includes multiple synchronization requests, each synchronization request carries at least one level of tree nodes from the first data version tree.

[0052] In one embodiment, when the data change frequency in the business scenario reaches a preset change frequency, the data synchronization request includes multiple synchronization requests. If the preset change frequency is not reached, the data synchronization request is a single synchronization request.

[0053] In one embodiment, when the network quality between the target storage node and the local storage node does not reach a preset network quality, the data synchronization request includes multiple synchronization requests. When the preset network quality is reached, the data synchronization request becomes a single synchronization request.

[0054] In this embodiment, the first data version tree includes multiple levels, each level having at least one tree node. Each tree node stores a version number, and the version number stored in a tree node at one level is determined based on the version numbers stored in the associated tree nodes at the next lower level. Figure 3 The diagram shown is a schematic of the first data version tree in one embodiment. The tree nodes of multiple levels include root nodes, intermediate nodes and leaf nodes. Each leaf node stores the version numbers of data 1, data 2, data 3 and data 4 respectively.

[0055] The first data version tree is constructed by the target storage node based on the version numbers of the stored data. The first data version tree includes at least one of a global data version tree or a partitioned data version tree. The global data version tree is constructed by sorting the locally stored data and then using the version number of each data item in the sorted list. The partitioned data version tree is constructed by the local storage node by partitioning the locally stored data and then using the version number of each data item within each partition.

[0056] In the global data version tree, each leaf node stores a version number of a piece of data; that is, each leaf node stores a version number for different data. For example... Figure 4 As shown, the global data version tree has 4 leaf nodes, 2 intermediate nodes, and a root node. Data 1 has the value d1 and version number CV1; data 2 has the value d2 and version number CV2; data 3 has the value d3 and version number CV3; and data 4 has the value d4 and version number CV4. Leaf node 1 stores the version number CV1 of data d1, leaf node 2 stores the version number CV2 of data 2, leaf node 3 stores the version number CV3 of data d3, and leaf node 4 stores the version number CV4 of data d4.

[0057] like Figure 5As shown, each leaf node in the partitioned data version tree stores the version number of a type of data in a data partition. For example, data partition 1 includes three types of data: A, B, and C. The version number of data A is CV1, the version number of data B is CV1, and the version number of data C is CV2. The node corresponding to data partition 1 is partition node 1, which has three leaf nodes. These three leaf nodes store the version numbers of data A (CV1), data B (CV1), and data C (CV2), respectively.

[0058] In this embodiment, the target storage node stores a first data version tree, which is constructed by the target storage node based on the version numbers of the stored data. When data needs to be synchronized with the local storage node, the target storage node can generate a data synchronization request based on the first data version tree and send it to the local storage node. The local storage node receives the data synchronization request and extracts the first data version tree from it.

[0059] In this embodiment, the local storage node can periodically receive data synchronization requests sent by the target storage node, that is, the target storage node can periodically perform data synchronization processing with the local storage node.

[0060] In this embodiment, the target storage node can update its version number and first data version tree after the stored data changes, and then initiate a data synchronization request. The local storage node receives the data synchronization request initiated by the target storage node after the data change.

[0061] Step S204: Match the first data version tree with the local second data version tree to obtain the data paths that differ between the first data version tree and the second data version tree. The second data version tree is constructed based on the version number of the data stored locally.

[0062] The second data version tree comprises multiple levels, each with at least one tree node. It is constructed from the version numbers of the data stored locally on the local storage nodes. The second data version tree includes at least one of a global data version tree or a partitioned data version tree.

[0063] Specifically, the first data version tree is matched with the local second data version tree according to nodes or data paths to obtain the data paths that differ between the first data version tree and the second data version tree.

[0064] In this embodiment, in the first data version tree, the tree node at the upper level stores the version number of the associated tree node at the lower level, or stores the sum of the version numbers of the associated tree nodes at the lower level. In the second data version tree, the tree node at the upper level stores the version number of the associated tree node at the lower level, or stores the sum of the version numbers of the associated tree nodes at the lower level.

[0065] like Figure 6 The diagram shown illustrates matching according to data paths in one embodiment. The structures of the first and second data version trees are both as follows... Figure 6 As shown, each data path starts from the root node, passes through two intermediate nodes, and ends at a leaf node. Data path 1 in the first data version tree can be matched against each other to determine if the two data paths store differences. Matching other data paths is similar to identify data paths that differ.

[0066] In one embodiment, the first data version tree is matched with the local second data version tree according to the data path to obtain the data paths that differ between the first and second data version trees. For example, each data path of each first data version tree can be matched with each data path of the second data version tree to determine the data paths that differ between the first and second data version trees.

[0067] In one embodiment, for a certain business scenario, when the data change frequency in the business scenario reaches a preset change frequency, multiple synchronization requests from the target storage node are received. Each synchronization request carries at least one level of tree nodes in the first data version tree. Each synchronization request is obtained by feeding back the tree nodes that differ from those carried in the previous synchronization request to the target storage node. The subsequent synchronization request carries the tree nodes associated with the tree nodes that differ from those in the next level of the first data version tree. The tree nodes of at least one level carried in each received synchronization request are matched with the tree nodes of the same level in the local second data version tree until the tree nodes that differ from those in each level are obtained. The first data path includes the tree nodes that differ from those in each level of the first data version tree, and the second data path includes the tree nodes that differ from those in each level of the second data version tree.

[0068] When the data change frequency in this business scenario does not reach the preset change frequency, a synchronization request is received from the target storage node, and the synchronization request carries the first data version tree of the target storage node.

[0069] In this embodiment, the condition of whether the data change frequency in the business scenario reaches the preset change frequency can be replaced with whether the network quality between the target storage node and the local storage node reaches the preset network quality.

[0070] In one embodiment, when the second data version tree is a global data version tree, the construction steps of the second data version tree include: sorting the locally stored data according to a preset arrangement, determining the version number of each data after sorting, constructing each leaf node of the second data version tree to be generated for each data version number, with each leaf node storing the version number of a data; based on each leaf node and the stored version number, constructing multiple levels of intermediate nodes in the second data version tree to be generated to obtain the second data version tree, with the last level of intermediate nodes being the root node of the second data version tree; wherein, the intermediate nodes of the next higher level store the version numbers of a preset number of intermediate nodes at the current level, and the intermediate nodes of the level above the leaf nodes store the version numbers of a preset number of intermediate nodes. In this embodiment, the second data version tree can be a global data version tree.

[0071] In this embodiment, the locally stored data can be sorted in lexicographical order to obtain the sorted data.

[0072] Construction of the global data version tree: such as Figure 4 As shown, the data is sorted lexicographically, and the version number of each data point is used as the version of the leaf nodes in the version tree. Then, starting from the leaf nodes, parent nodes are built upwards in groups of k nodes (i.e., the degree of the version tree is k), with the parent node's value being the sum of its child node version numbers, until the root node is reached. The value of each tree node is its version number. Since the data version number is monotonically increasing, if the data changes, its version number increases. According to the tree construction rules, the root's version number is also monotonically increasing. Therefore, changes in the tree node's version number reflect the data's age. The version tree construction process is similar to that of a Merkle tree. However, unlike a Merkle tree, to support data age verification, the parent node is not obtained through child node hashes, but rather by calculating the sum of the child node version numbers. This ensures that the version number of each tree node is monotonically increasing, and the age of the child node's data can be verified by comparing the parent node.

[0073] In one embodiment, when the second data version tree is a partitioned data version tree, the construction steps of the second data version tree include: dividing the locally stored data into multiple data partitions, each data partition including at least one type of data; determining the version number of each type of data in the multiple data partitions; constructing each leaf node of the second data version tree to be generated for each type of data, each leaf node storing the version number of a type of data; constructing multiple levels of partition nodes based on each leaf node to obtain the second data version tree; wherein, the partition node at the next higher level stores the version numbers of a preset number of partition nodes at the current level, and the partition node at the next higher level of the leaf node stores the version number of each type of data in the same data partition.

[0074] Understandably, the process of building the first data version tree is the same as that of building the second data version tree.

[0075] Step S206: Based on the data path, determine the data to be synchronized with the target storage node.

[0076] Among them, the data to be synchronized refers to the data that has changed, and also the data that needs to be synchronized between nodes.

[0077] Specifically, based on the different data paths, the local storage node can determine whether the data stored on the local storage node or the data stored on the target storage node is the latest version, thereby determining the data to be synchronized between the local storage node and the target storage node.

[0078] Step S208: Perform data synchronization processing between nodes on the data to be synchronized, the first data version tree, and the second data version tree.

[0079] The data synchronization processing between nodes can include the synchronization of stored data between the local storage node and the target storage node, as well as the data synchronization between the first data version tree and the second data version tree.

[0080] Specifically, the local storage node can determine whether the data to be synchronized belongs to the target storage node or to the local storage node, in order to determine the data synchronization method, and perform data synchronization processing between nodes on the data to be synchronized, the first data version tree, and the second data version tree according to the data synchronization method.

[0081] In this embodiment, the data stored on the local storage node and the data stored on the target storage node can be synchronized according to the data to be synchronized, and the data in the first data version tree and the data in the second data version tree can also be synchronized. The data in the first data version tree includes tree nodes and the version number stored in the tree nodes.

[0082] For example, if the data to be synchronized belongs to the local storage node, the data is fed back to the target storage node, enabling the target storage node to synchronize its stored data and the first data version tree based on the data to be synchronized. If it is determined that the data to be synchronized belongs to the target storage node, the data to be synchronized is retrieved from the target storage node, and the locally stored data and the second data version tree are synchronized based on the data to be synchronized.

[0083] In one embodiment, the method further includes: generating a data synchronization request, the data synchronization request carrying an updated second data version; and sending the data synchronization request to the remaining storage nodes in the distributed cluster, respectively, to instruct each remaining storage node to perform inter-node data synchronization processing based on its local data version tree and the updated second data version tree.

[0084] like Figure 7 As shown, the distributed cluster includes multiple storage nodes, including a local storage node and a target storage node. The remaining storage nodes can be any storage nodes in the distributed cluster other than the local and target storage nodes. After synchronizing data with the target storage node, the local storage node can proactively synchronize the version tree with the other nodes to synchronize data with the remaining storage nodes in the distributed cluster, thereby improving the data synchronization progress.

[0085] like Figure 8 As shown, the distributed cluster consists of four storage nodes. Configuration data changes are executed on a designated storage node. After a client modifies data and initiates a configuration change to a specific storage node, that storage node updates the corresponding data, increments the version number of the modified data, and regenerates the version tree based on the new version. Version tree reconstruction only requires rebuilding the path from the modified data to the root node. Storage nodes periodically synchronize their version trees to ensure data consistency. For example, after a configuration change is sent to storage node N1, storage node N1 modifies the data and regenerates its version tree. When storage node N4 sends its version tree to N1 to request data synchronization, if the N1 and N4 version trees are identical, the data is consistent and synchronization is unnecessary. If they are inconsistent, the two version trees are compared, the data requiring synchronization is calculated, and the result is returned to N4. Furthermore, to accelerate data synchronization, storage nodes can proactively synchronize their version trees with other storage nodes.

[0086] When synchronizing data between storage nodes, the nodes first synchronize and verify each other's version trees to determine if the data is consistent. If they are consistent, then the data is also consistent; if not, starting from the root, a breadth-first traversal is performed on both version trees simultaneously, comparing the versions at each level of the tree until a leaf node is reached. By comparing the version trees, inconsistent data can be identified, and data synchronization can be initiated with the remaining nodes. For example, ... Figure 8 As shown, the source storage node N4 requests the target storage node N2 to synchronize N4's version tree to the target storage node N2. After receiving N4's version tree, N2 compares it with its local version tree. If N2's version is updated, it synchronizes the latest data to N4. If N2's version is outdated, it reports the difference in data path to N4. N4 then obtains the data based on the difference in data path and synchronizes it to node N2.

[0087] In this embodiment, data updates can be initiated not only on a designated storage node but also on any node, with the receiving node forwarding the configuration change to the designated node for execution. Furthermore, configuration data can also reside in an external storage system and be synchronized using the data synchronization method of this embodiment, for example... Figure 4 In this context, N1 can be an external database.

[0088] In the aforementioned data synchronization method, a first data version tree carried in the data synchronization request from the target storage node is received. This first data version tree is constructed by the target storage node based on the version number of the stored data, while the second data version tree of the local storage node is constructed based on the version number of the locally stored data. This allows for matching the first data version tree with the local second data version tree, quickly and accurately determining whether the data on the two storage nodes are consistent. Furthermore, it can accurately detect data paths with discrepancies between the first and second data version trees, thereby obtaining the data to be synchronized between the two storage nodes. By performing data synchronization processing between nodes on the data to be synchronized, the first data version tree, and the second data version tree, only a portion of the data needs to be synchronized, rather than all of it. This reduces the amount of data synchronized and the network resource consumption during the data synchronization process, and improves the efficiency of data synchronization.

[0089] In one embodiment, matching a first data version tree with a local second data version tree to obtain data paths that differ between the first data version tree and the second data version tree includes: matching the first data version tree with a local second data version tree to obtain a first data path in the first data version tree and a second data path in the second data version tree, wherein the first data path and the second data path differ.

[0090] Based on the data path, determine the data to be synchronized with the target storage node, including: determining the first data version number corresponding to the first data path and the second data version number corresponding to the second data path; and determining the data to be synchronized with the target storage node based on the first data version number and the second data version number.

[0091] The data paths that differ include the first data path in the first data version tree and the second data path in the second data version tree. The difference between the first data path and the second data path means that the version number stored in the tree node in the first data path is different from the version number stored in the tree node in the second data path.

[0092] Specifically, the local storage node matches the first data version tree with the local second data version tree according to the node or data path to obtain the first data path in the first data version tree and the second data path in the second data version tree. The first data path and the second data path are different.

[0093] The local storage node determines the version number of the data stored in the leaf nodes of the first data path, i.e., the first data version number. The local storage node then determines the version number of the data stored in the leaf nodes of the second data path, i.e., the second data version number. The local storage node compares the first and second data version numbers to determine whether the data stored on the local storage node or the data stored on the target storage node is the latest version. This determines the data to be synchronized between the local and target storage nodes; this data is the latest version.

[0094] In this embodiment, the first data version tree is matched with the local second data version tree to obtain the first data path and the second data path that differ between the two data versions. Based on the first data version number corresponding to the first data path and the second data version number corresponding to the second data path, it is possible to accurately determine which node of the local storage node and the target storage node stores the latest version of the data, thereby accurately determining the latest version of the data to be synchronized.

[0095] In one embodiment, determining the data to be synchronized with the target storage node based on the first data version number and the second data version number includes: when the first data version number is higher than the second data version number, feeding back the first data path to the target storage node so that the target storage node can obtain the data to be synchronized corresponding to the first data path;

[0096] The process of synchronizing data, the first data version tree, and the second data version tree between nodes includes: receiving the data to be synchronized from the target storage node, and synchronizing the locally stored data and the second data version tree based on the data to be synchronized.

[0097] Specifically, the local storage node can determine the first data version number corresponding to the first data path and the second data version number corresponding to the second data path. When the first data version number is higher than the second data version number, it indicates that the data on the target storage node is the latest version, meaning that the data to be synchronized needs to be obtained from the target storage node. The local storage node sends the first data path back to the target storage node, which then retrieves the data to be synchronized corresponding to the first data path from its stored data and sends it back to the local storage node.

[0098] The local storage node receives the data to be synchronized from the target storage node, updates the data stored locally based on the data to be synchronized, and updates the second data version tree. The updated local storage data is the same as the data stored on the target storage node, and the updated second data version tree is the same as the first data version tree of the target storage node.

[0099] In this embodiment, by comparing the version numbers of the differing paths in the data version tree, it is possible to accurately determine which storage node has the latest data version. When the first data version number is higher than the second data version number, the first data path is fed back to the target storage node to obtain the latest data from the target storage node. The local storage data and version number are then updated based on the latest data to quickly achieve data synchronization. Furthermore, obtaining the data to be synchronized based on the differing path only requires adjusting the data with differences, rather than overwriting the entire amount, which reduces the amount of data to be synchronized and helps improve synchronization efficiency.

[0100] In one embodiment, data synchronization is performed on locally stored data and a second data version tree based on the data to be synchronized, including:

[0101] The locally stored data is updated based on the data to be synchronized, and the version number of the locally stored data is also updated. Based on the updated version number, the second data path of the second data version tree is updated, and the updated second data path is the same as the first data path of the first data version tree.

[0102] Specifically, the local storage node updates the data it stores based on the data to be synchronized, and also updates the version number of the locally stored data. Based on the updated version number, the local storage node updates the version number stored in the leaf nodes of the second data path in the second data version tree, obtaining the updated second data path, and thus the updated second data version. The updated second data path is the same as the first data path in the first data version tree, and the updated second data version is the same as the first data version tree.

[0103] In this embodiment, the locally stored data is updated based on the data to be synchronized, and the version number of the locally stored data is updated. The second data path of the second data version tree is updated based on the updated version number, so that only the path from the updated data version number to the root node needs to be rebuilt, without rebuilding the entire version tree, reducing the amount of data to be synchronized and improving the speed of data synchronization.

[0104] In one embodiment, determining the data to be synchronized with the target storage node based on the first data version number and the second data version number includes: when the second data version number is higher than the first data version number, obtaining the data to be synchronized corresponding to the second data path from the local storage data;

[0105] The process of synchronizing data, the first data version tree, and the second data version tree between nodes includes: feeding back the data to be synchronized to the target storage node, so that the target storage node can synchronize the stored data and the first data version tree based on the data to be synchronized.

[0106] Specifically, when the local storage node determines that the second data version number is higher than the first data version number, it retrieves the data corresponding to the second data path from its local storage as the data to be synchronized. The local storage node then sends the data to be synchronized back to the target storage node. Upon receiving the data to be synchronized, the target storage node updates its stored data based on the data to be synchronized and updates the version number of the stored data. Based on the updated version number, the target storage node updates the first data path of the first data version tree, making the updated first data path the same as the second data path of the second data version tree, thereby making the updated first data version the same as the second data version tree.

[0107] In this embodiment, by comparing the version numbers of the different paths in the data version tree, it is possible to accurately determine which storage node has the latest data version. When the second data version number is higher than the first data version number, it indicates that the version number of the locally stored data is the latest. Then, the data to be synchronized corresponding to the second data path is obtained from the local storage data, and the data to be synchronized is fed back to the target storage node. This enables the target storage node to quickly and accurately synchronize the stored data and the first data version tree based on the data to be synchronized, effectively realizing the synchronization of stored data and version tree.

[0108] In one embodiment, a data synchronization request includes multiple synchronization requests, each carrying at least one level of tree nodes in a first data version tree.

[0109] Match the first data version tree with the local second data version tree to obtain the first data path in the first data version tree and the second data path in the second data version tree, including:

[0110] The tree node of at least one level carried in each received synchronization request is matched with the tree node of the same level in the local second data version tree until the tree node with differences at each level is obtained.

[0111] Each synchronization request is obtained by sending feedback to the target storage node of the tree nodes that differ from those carried in the previous synchronization request. The first data path includes the tree nodes that differ at each level of the first data version tree, and the second data path includes the tree nodes that differ at each level of the second data version tree.

[0112] In this embodiment, the data synchronization method is used to synchronize data between storage nodes. The initiator of the data synchronization request is the target storage node, and the recipient of the data through the request is the local storage node.

[0113] Data synchronization requests consist of multiple synchronization requests. Each synchronization request carries at least one level of tree nodes from the first data version tree. Each synchronization request is obtained by the local storage node after it reports the differing tree nodes to the target storage node based on the tree nodes from the previous synchronization request. In other words, each synchronization request is obtained by the local storage node determining the differing tree nodes based on the tree nodes from the previous synchronization request and the tree nodes at the same level in its local second data version tree, and then reporting these differing tree nodes to the target storage node.

[0114] Each synchronization request carries the associated tree nodes in the next level of the tree node to which the difference exists. Specifically, if a tree node A in one level of the data version tree points to tree nodes B and C in the next level, then tree node A is associated with tree nodes B and C.

[0115] like Figure 9 As shown, the first synchronization request carries the first-level tree node A. After receiving the difference in the first-level tree node A from the local storage node, the target storage node determines the associated tree nodes B and C in the next level (i.e., the second level) of tree node A. The second synchronization request carries the second-level tree nodes B and C. After receiving the difference in the second-level tree node C, the target storage node determines the associated tree nodes D and E in the third level of tree node C. The third synchronization request carries the third-level tree nodes D and E. After receiving the difference in the third-level tree node D, the target storage node determines the associated tree nodes F and G in the fourth level of tree node D. The local storage node determines that the difference in the fourth level tree node is G. Therefore, the data path with differences is ACDG, and the data to be synchronized is the data corresponding to G and its version number.

[0116] Specifically, the target storage node obtains at least one level of tree nodes from the first data version tree, generates a synchronization request based on the at least one level of tree nodes, and sends the synchronization request to the local storage node.

[0117] The local storage node receives the synchronization request, extracts at least one level of tree nodes carried in the synchronization request, and matches the at least one level of tree nodes with the same level of tree nodes in the local second data version tree to determine the tree nodes that differ in the at least one level.

[0118] The local storage node sends the tree nodes that differ from the synchronization request back to the target storage node. After receiving the tree nodes that differ from the request, the target storage node determines the next-level tree node associated with the tree node in the first data version tree, and uses the next-level tree node to generate a new synchronization request.

[0119] The new synchronization request is sent again to the local storage node. The local storage node repeatedly processes the process of receiving the synchronization request until it stops looping when it reports back to the target storage node that there is a difference in the tree node at the last level. At this point, the local storage node can determine the tree nodes that differ at each level in the first data version tree, as well as the tree nodes that differ at each level in the second data version tree.

[0120] In the first data version tree, tree nodes that differ at each level can form the first data path; in the second data version tree, tree nodes that differ at each level can form the second data path.

[0121] In the first data version tree, each tree node stores the version number of the data stored on the target storage node. In the second data version tree, each tree node stores the version number of the locally stored data. The local storage node determines the first data version number corresponding to the first data path based on the version numbers stored in each tree node of the first data path. The local storage node determines the second data version number corresponding to the second data path based on the version numbers stored in each tree node of the second data path.

[0122] The local storage node compares the first data version number and the second data version number to determine the data to be synchronized with the target storage node.

[0123] The data to be synchronized can be either data from the local storage node or data from the target storage node. If the first data version number is higher than the second data version number, the data to be synchronized is from the target storage node. If the second data version number is higher than the first data version number, the data to be synchronized is from the local storage node.

[0124] In one embodiment, the local storage node may feed back a first data path to the target storage node. Alternatively, the local storage node may choose not to feed back the first data path, and the target storage node may determine the first data path in the first data version tree based on the differing tree nodes fed back by the local storage node each time.

[0125] In one embodiment, the target storage node can select tree nodes to generate synchronization requests according to the hierarchy in the first data version tree from high to low, so as to realize the generation and sending of multiple synchronization requests.

[0126] For scenarios where data changes infrequently, version tree synchronization can be simplified to synchronizing the root of the version tree. Nodes periodically synchronize the root version tree to ensure data consistency. When inconsistencies are found in the root version tree, the complete version tree is then synchronized, comparing and synchronizing the data that needs to be synchronized. Compared to directly synchronizing the version tree, this involves an additional communication process, but for scenarios with infrequent data changes, synchronizing the root version tree still reduces the overall communication overhead between nodes.

[0127] Furthermore, this invention proposes a hierarchical verification method for version trees, determining the data requiring synchronization with minimal network communication. Tree nodes first synchronize the root; if inconsistencies are found, the next level is compared. Each subsequent level of version tree node is compared, identifying inconsistent nodes, and this process is repeated recursively until inconsistent data is found. Figure 9 As shown, only tree nodes connected by solid lines (i.e., data paths ACDG) need to be transmitted between nodes in the distributed cluster, avoiding the synchronization of the entire version tree between storage nodes in the distributed cluster, and further reducing the amount of data transmission in network communication.

[0128] In this embodiment, the target storage node communicates with the local storage node in multiple rounds. Each round of communication sends a synchronization request carrying at least one level tree node from the first data version tree. The local storage node then matches the at least one level tree node carried in the synchronization request with tree nodes of the same level in its local second data version tree to identify the tree nodes that differ at each level and feeds them back to the target storage node. This ensures that the multiple rounds of communication between the two storage nodes only carry a small portion of the version tree data, effectively reducing communication overhead between nodes. Through these multiple rounds of communication, the differences in the data version trees between the two storage nodes can be determined. Therefore, based on the data path in the data version tree, the data that needs to be synchronized between the two storage nodes can be accurately determined. Furthermore, only incremental data needs to be synchronized, not all data, avoiding waste of computer resources and improving data synchronization efficiency.

[0129] Furthermore, when the communication network quality is poor, using multi-round communication to carry a small portion of the version tree data can reduce the network requirements, enabling effective data synchronization even under poor network conditions. This makes the data synchronization method adaptable to various network usage scenarios, thereby improving the efficiency of data synchronization.

[0130] In one embodiment, both the first data version tree and the second data version tree include multiple levels of tree nodes. Matching the first data version tree with the local second data version tree to obtain the first data path in the first data version tree and the second data path in the second data version tree includes:

[0131] The first data version tree and the local second data version tree are matched by tree nodes according to their levels to obtain the tree nodes that differ between the first data version tree and the second data version tree at each level; based on the tree nodes that differ between each level in the first data version tree, the first data path in the first data version tree is obtained; based on the tree nodes that differ between each level in the second data version tree, the second data path in the second data version tree is obtained.

[0132] Specifically, the first data version tree includes tree nodes at multiple levels, and the second data version tree includes tree nodes at multiple levels. The first data version tree and the second data version tree have the same number of levels.

[0133] For each level of the first data version tree and the local second data version tree, the local storage node matches the tree nodes of the first data version tree at the target level with the tree nodes of the second data version tree at the target level to obtain the tree nodes that differ at the target level, thereby obtaining the tree nodes that differ at each level of the first data version tree and the tree nodes that differ at each level of the second data version tree.

[0134] Based on the tree nodes that differ at each level in the first data version tree, the first data path in the first data version tree can be obtained. Based on the tree nodes that differ at each level in the second data version tree, the second data path in the second data version tree can be obtained. The first data path and the second data path are the data paths that differ from each other.

[0135] In this embodiment, the first data version tree and the local second data version tree are matched hierarchically to accurately filter out the tree nodes that differ between the first and second data version trees at each level. This accurately determines the first data path in the first data version tree and the second data path in the second data version tree, so that only the data paths that differ and their corresponding data need to be synchronized, instead of synchronizing all data and the entire version tree, reducing the amount of data to be synchronized and thus improving data synchronization efficiency.

[0136] In one embodiment, the first data version tree is constructed by the target storage node after partitioning the stored data, based on the version number of the data in each data partition; the second data version tree is constructed by partitioning the locally stored data, based on the version number of the data in each data partition.

[0137] In this embodiment, both the first data version tree and the second data version tree are partitioned data version trees. The first data version tree is constructed by the target storage node after partitioning the stored data, based on the version number of the data in each data partition, and each data partition includes at least one type of data.

[0138] Specifically, the target storage node divides the stored data into multiple data partitions, each containing at least one type of data, and determines a version number for each type of data within the multiple data partitions. For each type of data, a leaf node is constructed to generate a first data version tree, storing the version number of one type of data. Multiple levels of partition nodes are constructed based on each leaf node to obtain the first data version tree. Specifically, the partition node at the next higher level in the first data version tree stores the version numbers of a predetermined number of partition nodes at the current level, and the partition node at the next higher level of a leaf node stores the version number of each type of data within the same data partition.

[0139] In this embodiment, the second data version tree is constructed by the local storage node after partitioning the locally stored data, based on the version number of the data in each data partition. Each data partition includes at least one type of data. Each leaf node in the second data version tree stores the version number of one type of data in a data partition. The partition node at the next higher level of the leaf node stores the version number of each type of data in the same data partition, and the partition node at the next higher level stores the version numbers of a preset number of partition nodes at the current level.

[0140] In this embodiment, when a type of data in the locally stored data is deleted or a new type of data is added, it is only necessary to determine the data partition to which the new data belongs and the partition node of that partition. A leaf node is then added under that partition node to store the version number of the new data, or the leaf node of that data in the second data version tree is deleted. Then, the version number stored in the partition node of that data partition is changed. This only involves changes to the leaf node and partition node of one data partition, while other partition nodes and their leaf nodes remain unaffected, reducing the amount of data changes and data synchronization.

[0141] In this embodiment, the first data version tree is constructed by the target storage node after partitioning the stored data, based on the version number of the data in each data partition. The second data version tree is constructed by partitioning the locally stored data, based on the version number of the data in each data partition. This allows for partitioning the stored data before constructing the version tree, ensuring that data of the same version is divided into different data partitions. Subsequent data changes and synchronization only require modifications to the data in the corresponding data partitions and the corresponding nodes in the version tree, reducing the amount of modification during data synchronization and improving data synchronization efficiency. Furthermore, constructing the version tree according to data partitions can mitigate the slow response speed caused by frequent data changes and synchronizations in scenarios with frequent data changes and synchronizations.

[0142] In one embodiment, the method further includes constructing a local second data version tree, the steps of which include:

[0143] The locally stored data is divided into multiple data partitions, and each data partition includes at least one type of data;

[0144] Determine the version number of each type of data in multiple data partitions, and construct each leaf node of the second data version tree to be generated for each type of data. Each leaf node stores the version number of a type of data. Based on each leaf node, construct multiple levels of partition nodes to obtain the second data version tree. The partition node at the next higher level stores the version numbers of a preset number of partition nodes at the current level, and the partition node at the next higher level of the leaf node stores the version number of each type of data in the same data partition.

[0145] A single data partition may contain multiple versions of data.

[0146] Data partitions can be divided according to a preset data range. One data partition corresponds to one preset data range, and a preset data range may include at least one type of data.

[0147] Data partitions can also be divided according to data categories, with one data partition corresponding to one category of data.

[0148] The local storage node divides the locally stored data into multiple data partitions, each containing at least one type of data. The local storage node determines the version number of each type of data in the multiple data partitions, and constructs each leaf node of the second data version tree to be generated for each type of data. The number of each type of data is equal to the number of leaf nodes, and each leaf node stores the version number of one type of data.

[0149] For each data partition, a partition node is constructed. This partition node serves as the parent partition node of the leaf nodes for each type of data within that partition. This process constructs the parent partition nodes of each leaf node in the second data version tree to be generated. Furthermore, the partition node stores the version number of each type of data within that partition.

[0150] Next, the constructed partition node hierarchy can be used as the current hierarchy. The partition node of the next level can be constructed according to a preset number of partition nodes of the current hierarchy, thereby constructing the partition node of the next level for each partition node of the current hierarchy. Furthermore, the partition node of the next level stores the version number of the preset number of partition nodes of the current hierarchy.

[0151] This process continues until the last level of partition nodes is constructed, resulting in the second data version tree. The last level of partition nodes is the root node of the second data version tree.

[0152] In this embodiment, when constructing the partition node of the next higher level, the number of partition nodes used in the current level can be the same or different.

[0153] For example, starting from the leaf nodes, construct a partition node for each leaf node of the same data partition. For each partition node, construct the next level partition node for every 6 partition nodes, until the root node is constructed. Alternatively, construct a partition node for each leaf node of the same data partition, construct the next level partition node for every 4 partition nodes, then construct the next level partition node for every 3 next level partition nodes, and so on, until the root node is constructed.

[0154] In one embodiment, the upper-level partition node stores the sum of the version numbers of a preset number of partition nodes at the current level, and the upper-level partition node of the leaf node stores the version number of each type of data in the same data partition.

[0155] like Figure 5 As shown, in this invention, after data modification, the versions of the corresponding leaf nodes and ancestor nodes in the version tree will change. The path from the root node to the modified data reflects the changes in the version tree caused by the data change. When new data is added, according to the global version tree construction rules, the new data is inserted into the ordered list of existing data. At this time, the ancestor nodes of the version trees of all data to the right of the new data will change. Therefore, the addition and deletion of data will lead to a large-scale reconstruction of the version tree, and the amount of data to be synchronized obtained by comparing the version tree will also increase, and the synchronization data will be amplified, but at this time only one data item has been added or deleted.

[0156] To avoid amplification of synchronized data, this invention further proposes a partitioned version tree to reduce the impact of synchronized data amplification caused by data additions and deletions. For example... Figure 5 As shown, unlike version trees which build parent nodes upwards in groups of k nodes, partitioned version trees first divide data into different partitions according to corresponding rules. Leaf nodes then build partition nodes upwards in units of partitions, and partition nodes then build parent nodes upwards according to the global version tree's building rules, until the root node is completed. Partition rules can be based on fixed data ranges or business logic, such as configuring data categories. When data items are added or deleted, only the partition node containing that data item is affected. Compared to the previous method of affecting the parent node of all subsequent data items, this reduces the impact of synchronous data amplification caused by data item additions and deletions.

[0157] Because data modification operations such as adding and deleting data can easily lead to large-scale reconstruction of the data version tree, the amount of data to be synchronized in the resulting data version tree will also increase, even though only one piece of data may have been added or deleted. To avoid a large increase in the amount of data to be synchronized, this embodiment constructs a partitioned version tree. By dividing the locally stored data into multiple data partitions, each data partition includes at least one type of data, and determining the version number of each type of data in the multiple data partitions, each leaf node of the second data version tree to be generated is constructed for each type of data, and each leaf node stores the version number of one type of data. Based on each leaf node, multiple levels of partition nodes are constructed to obtain the second data version tree. In the second data version tree, the partition node at the next higher level stores the version numbers of a preset number of partition nodes at the current level, while the partition node at the next higher level of the leaf node stores the version number of each type of data in the same data partition. In this way, when data is added or deleted, only the partition node containing the changed data and the leaf nodes in that partition node are affected, without affecting other partition nodes and the leaf nodes in other partition nodes, thereby reducing the impact of the ever-increasing amount of data to be synchronized caused by data additions and deletions.

[0158] In one embodiment, the method further includes:

[0159] Determine the business scenario to which the locally stored data belongs, and determine the frequency of data changes within that business scenario;

[0160] When the data change frequency reaches the preset change frequency, a partitioned data version tree is constructed based on the version number of the locally stored data. The partitioned data version tree is constructed by partitioning the locally stored data and then constructing it based on the version number of each type of data in each data partition. When the data change frequency does not reach the preset change frequency, a global data version tree is constructed based on the version number of the locally stored data. The global data version tree is constructed by sorting the locally stored data and then constructing it based on the version number of each sorted data.

[0161] Here, "business scenario" refers to scenarios related to business processing, such as configuration scenarios for various applications, application session scenarios, transaction scenarios, live streaming scenarios, etc., but not limited to these. Data within business scenarios includes, but is not limited to, configuration data for various applications, user operation data for various applications, and interaction data between users.

[0162] Data change frequency indicates how frequently data changes. The preset change frequency is a pre-set empirical value.

[0163] like Figure 10As shown, the local storage node determines the business scenario to which the locally stored data belongs and the data change frequency within that business scenario. The local storage node obtains a preset change frequency. When the data change frequency reaches the preset change frequency, it indicates that the locally stored data is changing frequently. In this case, the locally stored data is partitioned, and a partition data version tree is constructed based on the version number of each type of data in each data partition.

[0164] If the data change frequency does not reach the preset change frequency, it means that the locally stored data has changed less, so a global data version tree is built based on the version number of the locally stored data.

[0165] In this embodiment, the data change frequency in this business scenario can be determined based on the version number of the locally stored data. The more version numbers there are, the higher the data change frequency.

[0166] In one embodiment, the frequency of data changes within a business scenario can be determined based on the category of the business scenario. For example, different business scenarios can be divided into categories with frequent data changes and categories with infrequent changes. The business scenario to which the locally stored data belongs is determined, and the category to which the business scenario belongs is also determined. If the business scenario belongs to the category with frequent data changes, a partitioned data version tree is constructed; if the business scenario belongs to the category with infrequent data changes, a global data version tree is constructed.

[0167] In this embodiment, the business scenario to which the locally stored data belongs is determined, and the data change frequency within that business scenario is also determined to assess the frequency of changes to the locally stored data. When the data change frequency reaches a preset frequency, indicating frequent changes to the locally stored data, a partitioned data version tree is constructed based on the version number of the locally stored data. This partitioned data version tree is built by partitioning the locally stored data and then using the version number of each type of data within each partition. This allows subsequent data changes to be identified by the partitioned data version tree, enabling the synchronization of only the changed data within the same version, rather than synchronizing the entire version, thus improving data synchronization efficiency. When the data change frequency does not reach the preset frequency, indicating infrequent changes to the locally stored data, a global data version tree is constructed based on the version number of the locally stored data. This results in a smaller data volume in the constructed version tree, allowing for faster identification of changed data paths during subsequent synchronization, thereby improving data synchronization efficiency by accelerating version tree comparison.

[0168] In one embodiment, the method further includes:

[0169] The data synchronization request is forwarded to the other storage nodes in the distributed cluster to which the target storage node belongs. The other storage nodes match the data based on the first data version tree and their local data version trees. When a data path with a difference is received from the other storage nodes after matching, and the data path is the same as the data path with a difference between the first data version tree and the second data version tree, the data to be synchronized is forwarded to the other storage nodes, so that the other storage nodes can perform data synchronization processing between nodes based on the data to be synchronized.

[0170] Specifically, during data synchronization between storage nodes in a distributed cluster, when a local storage node receives a data synchronization request from the target storage node, it can forward the request to the remaining storage nodes in the distributed cluster. The remaining storage nodes refer to all storage nodes in the distributed cluster other than the local and target storage nodes.

[0171] The data synchronization request carries a first data version tree of the target storage node. The local storage node stores a second data version tree. The first data version tree is matched with the local second data version tree to obtain the data paths where there are differences between the first and second data version trees. Based on the data paths, the local storage node determines the data to be synchronized with the target storage node.

[0172] Upon receiving a data synchronization request, the remaining storage nodes can match the first data version tree with their own local data version tree to identify any discrepancies in the data path. These discrepancies are then reported back to the local storage nodes.

[0173] When a local storage node receives data paths from other storage nodes, it can compare the data paths that differ between the first and second data version trees with the data paths reported by the other storage nodes to determine whether the two data paths are the same.

[0174] If the two data paths are the same, the data to be synchronized is forwarded to the other storage nodes, enabling them to perform inter-node data synchronization based on this data. If the two data paths are different, the data is discarded.

[0175] In this embodiment, the data synchronization of other storage nodes in the distributed cluster can be performed after the data synchronization between the local storage node and the target storage node is completed, or it can be performed in parallel during the data synchronization process between the local storage node and the target storage node, so that the data synchronization between nodes in the distributed cluster can be completed quickly.

[0176] In this embodiment, the data synchronization request is forwarded to the other storage nodes in the distributed cluster to which the target storage node belongs. The other storage nodes match the data version tree based on the first data version tree and the local data version tree of the other storage nodes. When the other storage nodes receive the data path that differs after matching, and the data path returned is the same as the data path that differs between the first data version tree and the second data version tree, the data to be synchronized is forwarded to the other storage nodes, so that the other storage nodes can perform data synchronization processing between nodes based on the data to be synchronized. This allows each storage node to achieve data synchronization with other storage nodes, thereby enabling the rapid completion of data synchronization between nodes in the distributed cluster.

[0177] In one embodiment, such as Figure 11 As shown, a data synchronization method is provided, applicable to any storage node in a distributed cluster, which includes multiple storage nodes. These multiple storage nodes can be categorized as a local storage node, a target storage node, and other storage nodes besides the local and target storage nodes. The data synchronization method includes:

[0178] Step S1102: The local storage node determines the business scenario to which the data stored locally belongs and determines the data change frequency in the business scenario.

[0179] Step S1104: When the data change frequency reaches a preset change frequency, the local storage node constructs a partitioned data version tree based on the version number of the locally stored data: the locally stored data is divided into multiple data partitions, each data partition including at least one type of data; the version number of each type of data in the multiple data partitions is determined, and each leaf node of the second data version tree to be generated is constructed for each type of data, with each leaf node storing the version number of one type of data; multiple levels of partition nodes are constructed based on each leaf node to obtain the second data version tree; wherein, the partition node at the next higher level stores the version numbers of a preset number of partition nodes at the current level, and the partition node at the next higher level of the leaf node stores the version number of each type of data in the same data partition.

[0180] Step S1106: When the data change frequency does not reach the preset change frequency, the local storage node constructs a global data version tree based on the version number of the locally stored data: the locally stored data is sorted according to a preset arrangement, the version number of each data after sorting is determined, and each leaf node of the second data version tree to be generated is constructed for each data version number, with each leaf node storing the version number of a data; based on each leaf node and the stored version number, multiple levels of intermediate nodes in the second data version tree to be generated are constructed to obtain the second data version tree, with the last level of intermediate nodes being the root node of the second data version tree; wherein, the intermediate node of the previous level stores the version numbers of a preset number of intermediate nodes at the current level, and the intermediate node of the previous level of the leaf node stores the version numbers of a preset number of intermediate nodes.

[0181] In step S1108, the local storage node receives a data synchronization request from the target storage node. The data synchronization request carries a first data version tree, which can be a global data version tree or a partition data version tree.

[0182] In step S1110, the local storage node matches the first data version tree with the local second data version tree to obtain the first data path in the first data version tree and the second data path in the second data version tree. The first data path and the second data path are different. The first data version tree can be a global data version tree or a partition data version tree.

[0183] In step S1112, the local storage node determines the first data version number corresponding to the first data path and the second data version number corresponding to the second data path.

[0184] When the first data version number is higher than the second data version number, execute steps S1114 to S1116: Step S1114, the first data version number is higher than the second data version number, the local storage node feeds back the first data path to the target storage node, so that the target storage node can obtain the data to be synchronized corresponding to the first data path.

[0185] In step S1116, the local storage node receives the data to be synchronized from the target storage node, updates the data stored locally based on the data to be synchronized, updates the version number of the data stored locally, and updates the second data path of the second data version tree based on the updated version number. The updated second data path is the same as the first data path of the first data version tree.

[0186] When the second data version number is higher than the first data version number, execute steps S1118 to S1120: Step S1118, the local storage node obtains the data to be synchronized corresponding to the second data path from the local storage data.

[0187] In step S1120, the local storage node sends the data to be synchronized back to the target storage node, enabling the target storage node to synchronize the stored data and the first data version tree based on the data to be synchronized.

[0188] In step S1122, the local storage node forwards the data synchronization request to the other storage nodes in the distributed cluster to which the target storage node belongs. The other storage nodes match the data version tree based on the first data version tree and the local data version tree of the other storage nodes.

[0189] Step S1124: When the local storage node receives the data path that differs from the data path returned by the other storage nodes after matching, and the returned data path is the same as the data path that differs between the first data version tree and the second data version tree, the local storage node forwards the data to be synchronized to the other storage nodes, so that the other storage nodes can perform data synchronization processing between nodes based on the data to be synchronized.

[0190] In one embodiment, such as Figure 12 As shown, an application scenario for a data synchronization method is provided, specifically applicable to the synchronization of configuration data. The distributed cluster in this application scenario includes multiple storage nodes, which can be categorized as storage node 1, storage node 2, and other storage nodes besides storage node 1 and storage node 2. Storage node 1 can be the storage node 1 in the aforementioned embodiments, and storage node 2 can be the storage node 2 in the aforementioned embodiments. The synchronized data can be configuration data. The specific data synchronization process in this application scenario is as follows:

[0191] 1) Users modify configuration data on the client side.

[0192] 2) The client generates a data change request based on the user's modification operation and sends the data change request to storage node 1.

[0193] 3) Storage node 1 modifies the stored configuration data based on the configuration data in the data change request, and modifies the first data version tree based on the modified data. The first data version tree can be a global data version tree or a partition data version tree.

[0194] 4) Storage node 1 detects the network quality between storage node 1 and storage node 2.

[0195] 5) When the network quality is poor, obtain the root node of the first data version tree, generate a synchronization request for configuration data, and send it to storage node 2.

[0196] 6) Storage node 2 matches the root node of the first data version tree with the root node of the second data version tree.

[0197] 7) When the root node is different, storage node 2 will send the root node back to storage node 1.

[0198] 8) Storage node 1 obtains the next level tree node from the first data version tree, generates a synchronization request including the next level tree node, and sends it to storage node 2.

[0199] 9) Storage node 2 will match the tree node in the synchronization request with the tree node at the same level in the second data version tree.

[0200] 10) When there are differences between tree nodes in this level, storage node 2 will send the differences to storage node 1.

[0201] 11) Repeat steps 8)-10) until storage node 2 determines the leaf node with the difference.

[0202] 12) For the leaf nodes in the first data version tree that differ from each other, compare the first data version number stored in the leaf node in the first data version tree with the second data version number stored in the leaf node in the second data version tree.

[0203] 13) When the first data version number is higher than the second data version number, storage node 2 generates a data acquisition request. The data acquisition request includes the leaf nodes that differ in the first data version tree, or the first data path composed of nodes that differ at each level in the first data version tree.

[0204] 14) Storage node 1 receives a data retrieval request and, based on the request, retrieves the configuration data corresponding to the leaf nodes with discrepancies as the data to be synchronized. This data to be synchronized is the configuration data modified by the user.

[0205] 15) Storage node 2 receives the data to be synchronized from storage node 1, updates the configuration data stored locally based on the data to be synchronized, and updates the version number of the local configuration data.

[0206] 16) Storage node 2 updates the second data version tree based on the updated version number.

[0207] 17) After storage node 2 is updated, it can initiate data synchronization with the other storage nodes. The data synchronization process between storage node 2 and the other storage nodes is the same as steps 4) to 16) above, thereby completing the data synchronization between storage nodes in the distributed cluster.

[0208] This embodiment provides a data synchronization method for configuration data based on a verifiable version tree, applicable to data synchronization between storage nodes in a distributed system. By constructing a version tree from the version numbers of the configuration data, storage nodes verify whether the data on the current storage node is outdated by synchronizing the version tree with each other, and determine the data that needs to be synchronized based on the comparison of the version trees. To further reduce the data transmission volume of each communication, a hierarchical verification of the version tree is proposed. Only one level of the version tree node needs to be synchronized at a time, and the amount of data synchronized each time is only related to the degree of the version tree, effectively reducing the network communication overhead caused by data synchronization. The version tree can be a global version tree or a partitioned version tree, enabling low-communication-overhead data synchronization in scenarios with large data volumes. The communication overhead does not increase with the data volume, demonstrating scalability.

[0209] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0210] Based on the same inventive concept, this application also provides a data synchronization device for implementing the data synchronization method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more data synchronization device embodiments provided below can be found in the limitations of the data synchronization method described above, and will not be repeated here.

[0211] In one embodiment, such as Figure 13 As shown, a data synchronization device 1300 is provided, comprising:

[0212] The receiving module 1302 is used to receive a data synchronization request from the target storage node. The data synchronization request carries a first data version tree, which is constructed by the target storage node based on the version number of the stored data.

[0213] The matching module 1304 is used to match the first data version tree with the local second data version tree to obtain the data paths that differ between the first data version tree and the second data version tree. The second data version tree is constructed based on the version number of the data stored locally.

[0214] The determination module 1306 is used to determine the data to be synchronized with the target storage node based on the data path.

[0215] The synchronization module 1308 is used to perform data synchronization processing between nodes for the data to be synchronized, the first data version tree, and the second data version tree.

[0216] In this embodiment, by receiving the first data version tree carried in the data synchronization request from the target storage node (this first data version tree is constructed by the target storage node based on the version number of the stored data), and the second data version tree of the local storage node is constructed based on the version number of the locally stored data, the system can quickly and accurately determine whether the data of the two storage nodes is consistent by matching the first data version tree with the local second data version tree. It can also accurately detect data paths with differences between the first and second data version trees, thereby obtaining the data to be synchronized between the two storage nodes. By performing data synchronization processing between nodes on the data to be synchronized, the first data version tree, and the second data version tree, only a portion of the data needs to be synchronized, rather than all of it. This reduces the amount of data synchronized and the network resource consumption during the data synchronization process, and improves the efficiency of data synchronization.

[0217] In one embodiment, the matching module 1304 is further configured to match the first data version tree with the local second data version tree to obtain the first data path in the first data version tree and the second data path in the second data version tree, wherein the first data path and the second data path are different.

[0218] The determination module 1306 is also used to determine the first data version number corresponding to the first data path and the second data version number corresponding to the second data path; and to determine the data to be synchronized with the target storage node based on the first data version number and the second data version number.

[0219] In one embodiment, the determining module 1306 is further configured to, when the first data version number is higher than the second data version number, feed back the first data path to the target storage node, so that the target storage node obtains the data to be synchronized corresponding to the first data path;

[0220] The synchronization module 1308 is also used to receive the data to be synchronized from the target storage node, and to synchronize the locally stored data and the second data version tree based on the data to be synchronized.

[0221] In one embodiment, the synchronization module 1308 is further configured to update the locally stored data based on the data to be synchronized, and update the version number of the locally stored data; based on the updated version number, update the second data path of the second data version tree, wherein the updated second data path is the same as the first data path of the first data version tree.

[0222] In one embodiment, the determining module 1306 is further configured to obtain the data to be synchronized corresponding to the second data path from the local storage data when the second data version number is higher than the first data version number;

[0223] The synchronization module 1308 is also used to feed back the data to be synchronized to the target storage node, so that the target storage node can synchronize the stored data and the first data version tree based on the data to be synchronized.

[0224] In one embodiment, a data synchronization request includes multiple synchronization requests, each carrying at least one level of tree nodes in a first data version tree.

[0225] The matching module 1304 is further configured to match the tree nodes of at least one level carried in each received synchronization request with the tree nodes of the same level in the local second data version tree until tree nodes with differences at each level are obtained; wherein, each synchronization request is obtained by feeding back the tree nodes with differences in the tree nodes carried in the previous synchronization request to the target storage node, the first data path includes tree nodes with differences at each level in the first data version tree, and the second data path includes tree nodes with differences at each level in the second data version tree.

[0226] In one embodiment, both the first data version tree and the second data version tree include tree nodes at multiple levels. The matching module 1304 is further configured to match the first data version tree with the local second data version tree according to the levels to obtain tree nodes that differ between the first data version tree and the second data version tree at each level; obtain a first data path in the first data version tree based on the tree nodes that differ between each level in the first data version tree; and obtain a second data path in the second data version tree based on the tree nodes that differ between each level in the second data version tree.

[0227] In one embodiment, the first data version tree is constructed by the target storage node after partitioning the stored data, based on the version number of the data in each data partition; the second data version tree is constructed by partitioning the locally stored data, based on the version number of the data in each data partition.

[0228] In one embodiment, the apparatus further includes the construction of a local second data version tree, and the apparatus further includes:

[0229] The module is used to divide locally stored data into multiple data partitions, each of which includes at least one type of data; determine the version number of each type of data in the multiple data partitions; construct each leaf node of the second data version tree to be generated for each type of data, with each leaf node storing the version number of a type of data; construct multiple levels of partition nodes based on each leaf node to obtain the second data version tree; wherein, the partition node at the next higher level stores the version numbers of a preset number of partition nodes at the current level, and the partition node at the next higher level of the leaf node stores the version number of each type of data in the same data partition.

[0230] In one embodiment, the device further includes:

[0231] The construction module is used to determine the business scenario to which the locally stored data belongs and the data change frequency in the business scenario. When the data change frequency reaches the preset change frequency, a partitioned data version tree is constructed based on the version number of the locally stored data. The partitioned data version tree is constructed based on the version number of each type of data in each data partition after the locally stored data is partitioned. When the data change frequency does not reach the preset change frequency, a global data version tree is constructed based on the version number of the locally stored data. The global data version tree is constructed based on the version number of each sorted data after the locally stored data is sorted.

[0232] In one embodiment, the device further includes:

[0233] The forwarding module is used to forward data synchronization requests to other storage nodes in the distributed cluster to which the target storage node belongs. The other storage nodes match the data based on the first data version tree and their local data version trees. When a data path with a difference is received from the other storage nodes after matching, and the returned data path is the same as the data path with a difference between the first data version tree and the second data version tree, the data to be synchronized is forwarded to the other storage nodes, so that the other storage nodes can perform data synchronization processing between nodes based on the data to be synchronized.

[0234] In one embodiment, the device further includes:

[0235] The forwarding module is used to generate data synchronization requests, which carry the updated second data version. The data synchronization requests are sent to the remaining nodes in the storage node cluster to instruct each remaining node to perform inter-node data synchronization processing based on its local data version tree and the updated second data version tree.

[0236] Each module in the aforementioned data synchronization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0237] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores synchronization data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data synchronization method.

[0238] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. The computer program, when executed by the processor, implements a synchronization method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0239] Those skilled in the art will understand that Figure 14 and Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0240] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0241] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0242] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0243] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, modified data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0244] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0245] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0246] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data synchronization method, characterized by, The method comprises: receiving a data synchronization request of a target storage node, the data synchronization request carrying a first data version tree, the first data version tree being constructed by the target storage node based on version numbers of stored data; matching the first data version tree with a second data version tree locally, obtaining a data path with differences between the first data version tree and the second data version tree, the second data version tree being constructed based on version numbers of locally stored data; determining to-be-synchronized data between the target storage node based on the data path; performing data synchronization processing between nodes on the to-be-synchronized data, the first data version tree and the second data version tree.

2. The method of claim 1, wherein, The matching of the first data version tree with the second data version tree locally to obtain a data path with differences between the first data version tree and the second data version tree comprises: matching the first data version tree with the second data version tree locally to obtain a first data path in the first data version tree and a second data path in the second data version tree, the first data path and the second data path having differences; The determination of to-be-synchronized data between the target storage node based on the data path comprises: determining a first data version number corresponding to the first data path and a second data version number corresponding to the second data path; determining to-be-synchronized data between the target storage node according to the first data version number and the second data version number.

3. The method of claim 2, wherein, The determination of to-be-synchronized data between the target storage node according to the first data version number and the second data version number comprises: when the first data version number is higher than the second data version number, feeding back the first data path to the target storage node, so that the target storage node obtains to-be-synchronized data corresponding to the first data path; The data synchronization processing between nodes on the to-be-synchronized data, the first data version tree and the second data version tree comprises: receiving the to-be-synchronized data fed back by the target storage node, and performing data synchronization on locally stored data and the second data version tree based on the to-be-synchronized data.

4. The method of claim 3, wherein, The data synchronization based on the to-be-synchronized data on locally stored data and the second data version tree comprises: updating locally stored data based on the to-be-synchronized data, and updating version numbers of locally stored data; updating a second data path of the second data version tree based on the updated version numbers, the updated second data path being the same as a first data path of the first data version tree.

5. The method of claim 2, wherein, The determination of to-be-synchronized data between the target storage node according to the first data version number and the second data version number comprises: when the second data version number is higher than the first data version number, obtaining to-be-synchronized data corresponding to the second data path from locally stored data; The data synchronization processing between nodes on the to-be-synchronized data, the first data version tree and the second data version tree comprises: feedback the to-be-synchronized data to the target storage node, so that the target storage node performs data synchronization on the stored data and the first data version tree based on the to-be-synchronized data.

6. The method of claim 2, wherein, The data synchronization request comprises multiple synchronization requests, and each synchronization request carries at least one level of tree nodes in the first data version tree. The matching of the first data version tree with the local second data version tree comprises: matching at least one level of tree nodes carried by each received synchronization request with tree nodes of the same level in the local second data version tree until tree nodes of each level that have differences are obtained; wherein each synchronization request is obtained after feedback of tree nodes having differences in tree nodes carried by a previous synchronization request to the target storage node, the first data path comprises tree nodes of each level having differences in the first data version tree, and the second data path comprises tree nodes of each level having differences in the second data version tree.

7. The method of claim 2, wherein, The first data version tree and the second data version tree each comprise multiple levels of tree nodes, and the matching of the first data version tree with the local second data version tree comprises: matching tree nodes of the first data version tree with tree nodes of the local second data version tree according to levels to obtain tree nodes of each level having differences in the first data version tree and the second data version tree; obtaining a first data path in the first data version tree based on tree nodes of each level having differences in the first data version tree; obtaining a second data path in the second data version tree based on tree nodes of each level having differences in the second data version tree.

8. The method of claim 1, wherein, The first data version tree is constructed based on version numbers of data in each data partition after the target storage node performs data partitioning on stored data, and the second data version tree is constructed based on version numbers of data in each data partition after local stored data is partitioned.

9. The method of claim 7, wherein, The method further comprises construction of a local second data version tree, and the construction of the second data version tree comprises: dividing local stored data into multiple data partitions, each data partition comprising at least one type of data; determining version numbers of each type of data in the multiple data partitions, and constructing each leaf node of a to-be-generated second data version tree for each type of data, each leaf node storing a version number of one type of data; constructing multiple levels of partition nodes based on the each leaf node to obtain a second data version tree; wherein a partition node of a previous level stores version numbers of a preset number of partition nodes of a current level, and a partition node of a previous level of the leaf node stores version numbers of each type of data in a same data partition.

10. The method of claim 1, wherein, The method further comprises: determining a business scenario to which local stored data belongs, and determining a data change frequency in the business scenario; When the data change frequency reaches a preset change frequency, a partitioned data version tree is constructed based on the version number of the locally stored data. The partitioned data version tree is constructed by partitioning the locally stored data and then constructing it based on the version number of each type of data in each data partition. When the data change frequency does not reach the preset change frequency, a global data version tree is constructed based on the version number of the locally stored data. The global data version tree is constructed by sorting the locally stored data and then constructing it based on the version number of each sorted data.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: The data synchronization request is forwarded to the other storage nodes in the distributed cluster to which the target storage node belongs, and the other storage nodes match the data version tree based on the first data version tree and the local data version tree of the other storage nodes; When the other storage nodes receive data paths that differ from each other after matching, and the data paths returned are the same as the data paths that differ between the first data version tree and the second data version tree, the data to be synchronized is forwarded to the other storage nodes, so that the other storage nodes can perform data synchronization processing between nodes based on the data to be synchronized.

12. A data synchronization apparatus, characterized by comprising: The device includes: A receiving module is used to receive a data synchronization request from a target storage node. The data synchronization request carries a first data version tree, which is constructed by the target storage node based on the version number of the stored data. The matching module is used to match the first data version tree with the local second data version tree to obtain the data paths that differ between the first data version tree and the second data version tree, wherein the second data version tree is constructed based on the version number of the data stored locally; The determination module is used to determine the data to be synchronized with the target storage node based on the data path; The synchronization module is used to perform data synchronization processing between nodes on the data to be synchronized, the first data version tree, and the second data version tree.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images