Data synchronization method, electronic device, storage medium and computer program product

Abstract

The application discloses a data synchronization method, an electronic device, a storage medium and a computer program product, relates to the technical field of distributed databases, and comprises the following steps: a branch blood relationship graph is constructed between a source data table and a target data table; a nearest common ancestor node between the source data table and the target data table is determined based on the branch blood relationship graph; first data change information from the nearest common ancestor node to the source data table and second data change information from the nearest common ancestor node to the target data table are collected; difference information between the source data table and the target data table is determined based on the first data change information and the second data change information, and the target data table is updated based on the difference information, so that the updated target data table is synchronized with the source data table. The application solves the technical problem of poor synchronization effect.

Description

Technical Field

[0001] This application relates to the field of distributed database technology, and in particular to a data synchronization method, electronic device, storage medium and computer program product. Background Technology

[0002] With the widespread application of cloud computing and big data technologies, distributed database systems have become the core infrastructure for enterprise data management. In such systems, data is typically deployed across multiple physical or logical clusters to achieve key objectives such as load balancing, disaster recovery backup, and multi-tenant isolation. Therefore, efficient, reliable, and consistent data synchronization between different computing clusters has become a fundamental technical requirement to support business continuity and data flow.

[0003] In current technological practices, cross-cluster data synchronization primarily relies on methods based on full data comparison (such as full table scan comparison), data export-based synchronization methods, log replication mechanisms, or snapshot comparison techniques. However, when dealing with massive amounts of data tables, these methods require loading and comparing all data in memory, resulting in enormous computational resource consumption and lengthy processing times. Furthermore, existing methods typically depend on direct network transmission of full data or logs, which incurs high bandwidth costs and significant transmission latency in cloud environments. This leads to severe challenges for cross-cluster data synchronization, including low efficiency, high cost, and insufficient reliability. Therefore, current cross-cluster data synchronization technologies for distributed databases suffer from poor synchronization performance.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a data synchronization method, electronic device, storage medium, and computer program product, which aims to solve the technical problem of poor synchronization performance.

[0006] To achieve the above objectives, this application proposes a data synchronization method, which includes:

[0007] Construct a branch lineage graph between the source data table and the target data table;

[0008] Based on the branch lineage graph, determine the nearest common ancestor node between the source data table and the target data table;

[0009] Collect the first data change information from the nearest common ancestor node to the source data table, and the second data change information from the nearest common ancestor node to the target data table;

[0010] Based on the first data change information and the second data change information, the difference information between the source data table and the target data table is determined, and the target data table is updated based on the difference information so that the updated target data table is synchronized with the source data table.

[0011] In one embodiment, the step of constructing the branch lineage map between the source data table and the target data table includes:

[0012] Obtain historical cloning relationship records between the source data table, the target data table, and the associated data table, and extract the parent-child relationship between the source data table, the target data table, and the associated data table from the historical cloning relationship records, wherein the associated data table is a data table that has an association relationship with the source data table and the target data table;

[0013] Based on the parent-child relationship, directed connections are established between the source data table, the target data table, and the associated data table to obtain a directed acyclic graph;

[0014] Traverse the directed acyclic graph (DAG), record the depth values ​​between each node in the DAG and the root node in the DAG, and use the DAG with the depth values ​​as a branch lineage graph.

[0015] In one embodiment, the step of determining the nearest common ancestor node between the source data table and the target data table based on the branch lineage graph includes:

[0016] Locate the source node corresponding to the source data table and the target node corresponding to the target data table in the branch lineage map, and determine the first depth value corresponding to the source node and the second depth value corresponding to the target node;

[0017] If the first depth value and the second depth value are different, the node with the larger depth value among the source node and the target node is taken as the backtracking node, and the backtracking node is moved along the backtracking path until the first depth value and the second depth value are the same. The backtracking path is the connection path from the backtracking node to the corresponding parent node of the backtracking node.

[0018] When the first depth value and the second depth value are the same, the source node is moved back layer by layer along the connection path of the source node's corresponding parent node, and the target node is moved back layer by layer along the connection path of the target node's corresponding parent node, until the source node and the target node move to the same node, and the same node is taken as the nearest common ancestor node.

[0019] In one embodiment, the step of collecting first data change information from the nearest common ancestor node to the source data table, and second data change information from the nearest common ancestor node to the target data table includes:

[0020] Obtain the first timestamp corresponding to the nearest common ancestor node, the second timestamp corresponding to the source data table, and the third timestamp corresponding to the target data table;

[0021] The data row insertion record, data row content update record, and data row deletion record between the first timestamp and the second timestamp are used as the first data change information;

[0022] The data row insertion record, data row content update record, and data row deletion record between the first timestamp and the third timestamp are used as the second data change information.

[0023] In one embodiment, the difference information includes information to be inserted, information to be updated, and information to be deleted;

[0024] The step of determining the difference information between the source data table and the target data table based on the first data change information and the second data change information includes:

[0025] A first data hash table for the source data table is determined based on the first data change information, and a second data hash table for the target data table is determined based on the second data change information.

[0026] For any second hash index in the second data hash table, query the first hash index corresponding to the second hash index in the first data hash table;

[0027] If no first hash index corresponding to the second hash index is found in the first data hash table, then the data row in the target data table associated with the second hash index is taken as the information to be inserted into the target data table.

[0028] If a first hash index corresponding to the second hash index is found in the first data hash table, and the content of the data row corresponding to the first hash index is inconsistent with the content of the data row corresponding to the second hash index, then the data row in the target data table associated with the second hash index is taken as the information to be updated in the target data table.

[0029] After traversing the second data hash table, for any first hash index in the first data hash table, query the second data hash table for the second hash index corresponding to the first hash index;

[0030] If no second hash index corresponding to the first hash index is found in the second data hash table, then the data row in the source data table associated with the first hash index is taken as the information to be deleted from the target data table.

[0031] In one embodiment, the steps of determining a first data hash table of the source data table based on the first data change information and determining a second data hash table of the target data table based on the second data change information include:

[0032] Based on the first data change information, a first candidate hash table and a first tombstone hash table are constructed for the source data table. The first candidate hash table is used to store the first hash index of all data rows in the source data table, and the first tombstone hash table is used to store the deletion records of the source data table.

[0033] Based on the second data change information, a second candidate hash table and a second tombstone hash table are constructed for the target data table. The second candidate hash table is used to store the second hash index of all data rows in the target data table, and the second tombstone hash table is used to store the deleted records of the target data table.

[0034] The first candidate hash table is updated based on the deleted records in the first tombstone hash table to obtain the first data hash table, and the second candidate hash table is updated based on the deleted records in the second tombstone hash table to obtain the second data hash table.

[0035] In one embodiment, the step of updating the target data table based on the difference information includes:

[0036] The difference information is converted into a data operation instruction sequence, and the data operation instruction sequence is transmitted to the target cluster where the target data table is located;

[0037] In the target cluster, each data operation instruction in the data operation instruction sequence is executed sequentially to update the target data table.

[0038] Furthermore, to achieve the above objectives, this application also proposes a data synchronization system, which includes:

[0039] The graph construction module is used to construct the branch lineage graph between the source data table and the target data table;

[0040] A node lookup module is used to determine the nearest common ancestor node between the source data table and the target data table based on the branch lineage graph.

[0041] The incremental acquisition module is used to acquire the first data change information from the nearest common ancestor node to the source data table, and the second data change information from the nearest common ancestor node to the target data table;

[0042] The data synchronization module is used to determine the difference information between the source data table and the target data table based on the first data change information and the second data change information, and to update the target data table based on the difference information so that the updated target data table is synchronized with the source data table.

[0043] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data synchronization method as described above.

[0044] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the data synchronization method described above.

[0045] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data synchronization method described above.

[0046] This application provides a data synchronization method, which includes: constructing a branch lineage graph between a source data table and a target data table; determining the nearest common ancestor node between the source data table and the target data table based on the branch lineage graph; collecting first data change information from the nearest common ancestor node to the source data table, and second data change information from the nearest common ancestor node to the target data table; determining difference information between the source data table and the target data table based on the first data change information and the second data change information, and updating the target data table based on the difference information, so that the updated target data table is synchronized with the source data table.

[0047] This application constructs a branch lineage graph and identifies the nearest common ancestor node, automatically recognizing the historical derivation relationships between data tables to be synchronized. This allows for precise location of the data time range to be compared, avoiding the enormous computational overhead caused by repeated comparisons of common ancestor data in traditional full-scan or full-export methods. Secondly, by collecting only incremental data change information (first and second data change information) after the nearest common ancestor node, the amount of data involved in comparison and transmission is limited to the truly changed parts. This solves the problems of high bandwidth consumption, long transmission latency, and high cloud service costs caused by full-data transmission in traditional methods. Finally, the target data table is updated based on this incremental difference information, ensuring eventual data consistency while achieving precise and efficient version control synchronization. Compared to related solutions that rely on full-data comparison, export, or log copying, requiring processing the entire dataset in memory, resulting in huge computational resource consumption and lengthy processing latency, this application, by introducing branch lineage relationship recognition, achieves incremental comparison and synchronization similar to version control, processing only data changes after the branch. This significantly reduces the amount of computation and data transmission, significantly improves synchronization efficiency, and reduces resource consumption and operating costs. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating an embodiment of the data synchronization method of this application.

[0051] Figure 2 This is a flowchart illustrating Embodiment 2 of the data synchronization method of this application;

[0052] Figure 3 This is a flowchart illustrating Embodiment 3 of the data synchronization method of this application;

[0053] Figure 4 This is a schematic diagram of the module structure of the data synchronization system according to an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the data synchronization method in this application embodiment.

[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be understood that the first embodiment described herein is merely used to explain the technical solution of this application and is not intended to limit this application.

[0057] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The main solution of the first embodiment of this application is: constructing a branch lineage graph between the source data table and the target data table; determining the nearest common ancestor node between the source data table and the target data table based on the branch lineage graph; collecting first data change information from the nearest common ancestor node to the source data table, and second data change information from the nearest common ancestor node to the target data table; determining the difference information between the source data table and the target data table based on the first data change information and the second data change information, and updating the target data table based on the difference information, so that the updated target data table is synchronized with the source data table.

[0059] In the first embodiment, for ease of description, the following description uses a data synchronization system as the execution subject.

[0060] With the development of cloud computing and big data technologies, hyperconverged database systems are widely used in enterprise applications. Hyperconverged databases integrate OLTP, OLAP, and stream processing capabilities, adopting a storage-compute-transaction separation architecture, capable of simultaneously supporting transaction processing and analytical query workloads. In modern distributed database architectures, data is often distributed across multiple computing clusters, requiring data synchronization between different clusters to achieve functions such as load balancing, disaster recovery, data migration, and multi-tenant isolation.

[0061] Traditional data discrepancy analysis and synchronization methods typically employ the following approaches:

[0062] 1. Full table scan or row-by-row comparison: Identifying data differences by performing a full table scan or row-by-row comparison requires loading all data into memory for comparison.

[0063] 2. MySQL full dump and apply: The traditional MySQL full dump method is used to export the data of the entire table to an SQL file, and then the apply operation is executed in the target environment.

[0064] 3. Master-slave replication or log transmission: Data synchronization is achieved through master-slave replication or log transmission, which requires the transmission of complete change logs.

[0065] 4. Snapshot comparison: Create table snapshots and then compare the differences between different snapshots, but usually it is necessary to compare all the data.

[0066] The following defects exist:

[0067] 1. Computational Efficiency Issues: For tables with a lineage (such as multiple branch tables cloned from the same table), traditional methods repeatedly compare data from their common ancestor, resulting in enormous computational overhead. For example, when comparing two branch tables cloned from the same table, traditional methods require comparing all the data, whereas in reality, only the incremental portion after the branch needs to be compared. This repeated comparison causes a large amount of unnecessary computational overhead, especially in large-scale data table scenarios, where computation time increases exponentially.

[0068] 2. Memory Consumption Issues: Full comparisons of large-scale tables require loading a large amount of data into memory, leading to excessive memory consumption, sometimes exceeding system capacity. Traditional methods cannot effectively utilize streaming processing mechanisms, requiring the loading of all data at once, which limits the system's ability to process large-scale tables.

[0069] 3. Inaccurate handling of deletion operations: In a Multi-Version Concurrency Control (MVCC) environment, deletion operations are implemented through tombstone records. Traditional methods struggle to properly handle these deletion markers, failing to accurately identify deleted records and leading to inaccurate difference analysis results, potentially missing deletion operations or misidentifying data states.

[0070] 4. Output format limitations: Traditional methods typically only support a single output format, which cannot meet the needs of different application scenarios. For example, they cannot directly generate executable SQL scripts, CSV files, or directly output to object storage, thus limiting the scope of application of the technology.

[0071] 5. Lack of conflict resolution mechanisms: In data merging scenarios, when two branches make different modifications to the same data, traditional methods lack effective conflict detection and resolution mechanisms. The inability to identify conflicts and provide flexible conflict resolution strategies may lead to data inconsistency or operation failure.

[0072] 6. Insufficient adaptation to cloud-native environments: In cloud environments, traditional methods cannot fully utilize the characteristics of object storage, such as directly referencing persistent data objects. All data must be transferred, resulting in high transmission costs and long processing times, making it unsuitable for cloud-native deployment scenarios.

[0073] 7. Low efficiency in cross-cluster synchronization:

[0074] ① High cost of full data transfer: In cloud deployment environments, cross-cluster full data transfer requires extremely high bandwidth and is very expensive, especially when transferring data across regions. Traditional methods must transfer all data and cannot utilize the lineage relationships between tables to achieve incremental transfer.

[0075] ②Long transmission time: Large data volumes lead to long transmission times, affecting business continuity and real-time requirements. For TB-level data tables, a full transmission may take several hours or even days.

[0076] ③ Lack of incremental synchronization mechanism: It cannot identify the lineage relationship between tables and cannot implement an incremental synchronization mechanism similar to the Git version control system (`git diff` only compares the changed parts). All data must be transferred, resulting in serious waste of cross-cluster network bandwidth.

[0077] ④ Difficulty in ensuring consistency: Data may continue to change during the full transmission process, leading to inconsistencies after synchronization. There is a lack of effective snapshot mechanisms and transaction guarantees.

[0078] ⑤ Cross-cluster network dependency: Full transmission requires high network stability; network interruptions necessitate retransmission, and there is a lack of breakpoint resumption mechanisms. In scenarios involving cross-regional or cross-availability zones, network instability can lead to synchronization failures.

[0079] ⑥ Inability to optimize based on lineage: Similar to how Git version control systems compare only the changed parts of files instead of the entire file using `git diff`, database table data difference analysis should also only compare the changed parts. However, current technologies lack an incremental comparison mechanism like Git, making it impossible to identify the lineage relationships between tables, resulting in the transmission of a large amount of redundant data during cross-cluster synchronization.

[0080] Compared to the traditional MySQL full dump and apply method, existing technologies still face the following problems in cross-cluster synchronization scenarios:

[0081] 1. Huge bandwidth consumption: Cross-cluster full data transmission has extremely high bandwidth requirements and is very expensive;

[0082] 2. Excessive transmission time: Large data volume leads to long transmission time, which cannot meet real-time requirements;

[0083] 3. Lack of incremental mechanism: It cannot utilize the lineage relationships between tables to optimize calculation and transmission;

[0084] 4. High complexity: Full data transmission is highly complex, and ensuring consistency is difficult;

[0085] 5. Poor reliability: It lacks a mechanism for resuming interrupted transmissions, and retransmission is required when the network is interrupted.

[0086] This application provides a solution that, by constructing a branch lineage graph and determining the nearest common ancestor node, can automatically identify the historical derivation relationships between data tables to be synchronized, thereby accurately locating the time range of data to be compared. This avoids the huge computational overhead caused by repeated comparisons of common ancestor data in traditional full-scan or full-export methods. Secondly, by collecting only incremental data change information (first data change information and second data change information) after the nearest common ancestor node, the amount of data involved in comparison and transmission is limited to the part that has actually changed. This solves the problems of high bandwidth consumption, long transmission latency, and high cloud service costs caused by full data transmission in traditional methods. Finally, the target data table is updated based on this incremental difference information, ensuring eventual data consistency while achieving precise and efficient version control synchronization. Compared to related solutions that rely on full data comparison, export, or log copying, which require processing the entire dataset in memory, resulting in huge computational resource consumption and lengthy processing delays, this application introduces branch lineage recognition to achieve incremental comparison and synchronization similar to version control. It only processes data changes after the branch, thereby significantly reducing the amount of computation and data transmission, significantly improving synchronization efficiency, and reducing resource consumption and operating costs.

[0087] It should be noted that the executing entity in the first embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, or other electronic device, or a system, application, or program capable of implementing the above functions. The first embodiment and the following embodiments will be described using a data synchronization system as an example.

[0088] All actions involving the acquisition of signals, information, or data in this application are carried out in accordance with the relevant data protection laws and policies of the country where the application is located, and with the authorization of the owner of the relevant device.

[0089] Based on this, embodiments of this application provide a data synchronization method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data synchronization method of this application.

[0090] In this embodiment, the data synchronization method includes steps S01 to S04:

[0091] Step S01: Construct a branch lineage graph between the source data table and the target data table;

[0092] It's important to note that the source table refers to the database table that serves as the origin and benchmark for data changes in a given synchronization operation. In cross-cluster synchronization scenarios, the source table is the cluster that initiated the synchronization request. The target table refers to the table that needs to be updated in a given synchronization operation to ensure its data state is consistent with the source table. A branch lineage graph is a directed acyclic graph data structure used to formally record the cloning and derivation relationships between all data tables. Each node in the graph represents a data table (source or target), and node attributes include at least a table identifier and a depth value. Each directed edge points from a parent table node to a child table node, representing a cloning operation, and is associated with a cloning timestamp.

[0093] Understandably, step S01 establishes the lineage topology between data tables by extracting clone records and constructing a directed acyclic graph with depth information. This provides a cognitive basis for all subsequent processing, enabling the system to distinguish between common ancestor data and branched data, avoiding the huge computational overhead caused by repeatedly processing the common ancestor part, and creating a prerequisite for incremental synchronization.

[0094] Step S02: Determine the nearest common ancestor node between the source data table and the target data table based on the branch lineage graph;

[0095] It should be noted that the nearest common ancestor node refers to the common ancestor node found in the branch lineage graph that is closest to the source data table node and the target data table node.

[0096] Understandably, step S02 addresses the shortcomings of existing full comparison methods, which suffer from excessive computational scope and low efficiency, by quickly locating the nearest common ancestor node and its branch time points of the two tables in the lineage graph. It intelligently limits the time range of data comparison to the incremental data generated after each table separates from its common ancestor, avoiding the repeated processing of a large amount of unchanged common historical data. This optimizes the computational complexity from being related to the total amount of data in the table to being related only to the incremental change amount, thus significantly improving computational efficiency.

[0097] Step S03: Collect the first data change information from the nearest common ancestor node to the source data table, and the second data change information from the nearest common ancestor node to the target data table;

[0098] It should be noted that the first data change information and the second data change information are collections of all data row change records that occurred between the first timestamp and their respective corresponding timestamps for the source and target data tables, respectively. The first data change information includes records of all insert, update, and delete operations in the source data table within a preset time range. The second data change information includes records of all insert, update, and delete operations in the target data table within a preset time range.

[0099] Understandably, step S03, based on the determined branch time point, only collects incremental data change information afterward, which solves the problem of excessive memory usage caused by the traditional method of having to load all data into memory, and reduces memory pressure through incremental collection.

[0100] Step S04: Determine the difference information between the source data table and the target data table based on the first data change information and the second data change information, and update the target data table based on the difference information so that the updated target data table is synchronized with the source data table.

[0101] It should be noted that the difference information is derived by comparing the incremental data change information (i.e., the first data change information and the second data change information) of the source data table and the target data table. It is the minimum set of differences that needs to be executed to make the target data table consistent with the source data table. During data synchronization, only this set of differences is transmitted and operated on, rather than the entire table data.

[0102] Understandably, step S04 determines the final difference information between the source table and the target table based on the collected incremental data change information, and updates the target data table accordingly to complete the synchronization. This overcomes the problems of rigid output format and high bandwidth cost, large latency and poor reliability caused by the reliance on full cross-cluster network transmission in traditional methods. It avoids the direct network transmission of a large amount of redundant data, and significantly improves the overall efficiency and reliability of cross-cluster synchronization while ensuring the atomicity of the operation.

[0103] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S01 includes steps S11 to S13:

[0104] Step S11: Obtain the historical cloning relationship records between the source data table, the target data table, and the associated data table, and extract the parent-child relationship between the source data table, the target data table, and the associated data table from the historical cloning relationship records. The associated data table is a data table that has an association relationship with the source data table and the target data table.

[0105] It's important to clarify that related data tables are all relevant tables that must be included in the calculation when constructing a complete lineage network encompassing the source and target data tables. These tables are all connected to the source and target data tables through cloning derivation paths. For example, if source data table A is cloned from ancestor data table O, and target data table B is cloned from another intermediate data table M, and M is also cloned from ancestor data table O, then both table O and table M are related data tables. A relationship refers to the parent-child derivation chain directly generated by the cloning operation, as well as the indirect lineage relationships formed through this chain. It describes the genealogical connection between tables in terms of data source, rather than general business foreign key associations or query dependencies.

[0106] Additionally, it should be noted that historical cloning relationship records are structured data entries in the database system metadata that record each table cloning operation. Each record includes at least: a child table identifier, a parent table identifier, and a timestamp of the cloning operation. The parent-child relationship is a direct abstraction of historical cloning relationship records. If table Y is cloned from table X, then X is defined as the parent table of Y, and Y is the child table of X. This is a directional derivation relationship.

[0107] Step S12: Based on the parent-child relationship, establish directed connections between the source data table, the target data table, and the related data table to obtain a directed acyclic graph;

[0108] It's important to note that a directed connection is a directed edge established between nodes representing parent and child tables when constructing a graph model. The direction of this edge points from the parent node to the child node, indicating that data flows from the parent table to (derives from) the child table. A directed acyclic graph (DAG) is a graph data structure composed of nodes (tables) and directed connections (parent-child relationships). Its directionality is reflected in the direction of the edges, while acyclicity ensures that a table cannot become its own ancestor through a series of cloning operations.

[0109] Step S13: Traverse the directed acyclic graph, record the depth values ​​between each node in the directed acyclic graph and the root node in the directed acyclic graph, and use the directed acyclic graph with each depth value as a branch lineage graph.

[0110] It's important to note that a graph node is an entity representing a specific data table in a directed acyclic graph (DAG). Each node carries its identifier (such as the table name) and attributes (such as the depth value). The root node is a node with an in-degree of zero in a DAG, meaning it has no parent node; it is the initially created base table. A graph may have multiple root nodes (multiple independent lineage trees). The depth value is an integer attribute calculated for each graph node, representing the minimum number of directed connections (edges) required to reach the nearest root node in its lineage tree. The root node has a depth of 0, its direct children have a depth of 1, and so on.

[0111] For example, a traversal algorithm (such as depth-first search) is performed on the constructed directed acyclic graph. Starting from all root nodes with an in-degree of zero, the depth value of each node in the directed acyclic graph from its nearest root node is calculated and recorded. Finally, the directed acyclic graph with the depth value is persistently stored as a branch lineage graph.

[0112] In this embodiment, by acquiring and parsing historical cloning relationship records, the system is provided with the original data of the derived relationships between data tables. This solves the problem that traditional methods cannot identify the derived associations between tables, leading to a lack of optimization basis for subsequent calculations. A directed acyclic graph structure is established based on the clear parent-child relationship, transforming discrete metadata into a computable lineage topology model. This lays the foundation for efficient querying and path analysis. By traversing the graph and calculating the depth of each node, a lineage graph with depth information is generated, which significantly reduces the algorithm complexity for determining the nearest common ancestor. This solves the problem of redundant and inefficient synchronization calculations caused by the inability to utilize lineage relationships in the full comparison method, providing an accurate and efficient global relationship view for implementing incremental difference synchronization.

[0113] In one feasible implementation, step S02 includes steps A01 to A03:

[0114] Step A01: Locate the source node corresponding to the source data table and the target node corresponding to the target data table in the branch lineage map, and determine the first depth value corresponding to the source node and the second depth value corresponding to the target node.

[0115] It should be noted that the source node and the target node are two vertices in the branch lineage graph that correspond to the source and target data tables to be synchronized, respectively. The first depth value and the second depth value refer to the attribute values ​​of the source node and the target node in the directed acyclic graph, respectively. They are used to characterize the number of edges traversed by the longest path from any root node in the branch lineage graph to the corresponding node (source node or target node). The larger the depth value, the farther the node is from the root in the lineage derivation chain.

[0116] Step A02: If the first depth value and the second depth value are different, take the node with the larger depth value between the source node and the target node as the backtracking node, and move the backtracking node along the backtracking path until the first depth value and the second depth value are the same. The backtracking path is the connection path from the backtracking node to the corresponding parent node of the backtracking node.

[0117] It should be noted that a backtracking node is a temporarily designated operand reference when the first depth value differs from the second depth value; it points to the node with the larger depth value between the source and target nodes. A backtracking path is a chain-like path formed in the branch lineage graph by a backtracking node moving upwards along its own path, continuously following the connection paths of its parent nodes. A connection path is a single directed edge from a node to its parent node in the branch lineage graph.

[0118] Step A03: If the first depth value and the second depth value are the same, move the source node back layer by layer along the connection path of the source node's corresponding parent node, and move the target node back layer by layer along the connection path of the target node's corresponding parent node, until the source node and the target node move to the same node, and take the same node as the nearest common ancestor node.

[0119] It should be noted that identical nodes occur when the source and target nodes, after their depth values ​​are aligned, move synchronously upwards along their respective parent edges. At a certain point, these two moving nodes point to the same vertex in the graph. This vertex is the lowest common ancestor node determined when the algorithm terminates.

[0120] For example, in the branch lineage graph, the source and target nodes corresponding to the source and target data tables are located, and their depth values ​​are read. If the depths are different, the node with the larger depth value is used as the backtracking node, and it moves up layer by layer along the connection path pointing to its parent node until the two nodes have the same depth, thus achieving depth alignment. Subsequently, the two nodes are synchronously backtracked layer by layer along their respective parent node paths until they reach the same node, which is then determined as the nearest common ancestor node. For example, if table t1 (depth 2) and table t2 (depth 3) both originate from table t0, the system first backtracks the node with depth 3 to depth 2, and then synchronously backtracks to the common node t0.

[0121] In this implementation, by quickly locating the source and target nodes and obtaining their depth values, the problem of lacking precise starting point positioning in traditional methods for difference analysis is solved. This establishes an accurate coordinate benchmark for subsequent efficient searching. Through depth alignment, the hierarchical differences between the two nodes caused by different branch histories are eliminated, ensuring the fairness and accuracy of the search path and avoiding invalid traversal branches. Through the synchronous backtracking movement mechanism, the system efficiently converges to the common ancestor from the aligned node, directly and accurately locating the nearest common ancestor node. By accurately locating the branch point, the scope of subsequent data calculation and transmission is strictly limited to the necessary incremental part, thus providing a core algorithm guarantee for significantly reducing computational resource consumption and improving synchronization efficiency.

[0122] In one feasible implementation, the incremental data change information includes first data change information and second data change information;

[0123] Step S03 includes steps A11 to A13:

[0124] Step A11: Obtain the first timestamp corresponding to the nearest common ancestor node, the second timestamp corresponding to the source data table, and the third timestamp corresponding to the target data table;

[0125] It should be noted that the first timestamp refers to the creation timestamp of the data table corresponding to the nearest common ancestor node in the branch lineage graph. This time point marks the last moment of consistency in the common data state of the source and target data tables. The second and third timestamps refer to the comparison benchmark time points selected for the source and target data tables for difference analysis, respectively. These can be the timestamp of a data snapshot or the current time when the analysis operation is initiated. For example, a snapshot Snapshot_A can be created for the source table Table_A, and its timestamp TS_A is the second timestamp; a snapshot Snapshot_B can be created for the target table Table_B, and its timestamp TS_B is the third timestamp.

[0126] Step A12: The data row insertion record, data row content update record, and data row deletion record between the first timestamp and the second timestamp are taken as the first data change information;

[0127] It should be noted that data row insertion records, data row content update records, and data row deletion records are metadata units used by the database system to track and record data changes within a preset time range (e.g., one hour). Specifically, a data row insertion record records information about the creation of a new data row, including its primary key value and the initial values ​​of all columns. A data row content update record records information about the modification of the content of an existing data row, including its primary key value and the new values ​​of the modified columns. A data row deletion record, in a Multi-Version Concurrency Control (MVCC) environment, records the point in time when the primary key of a data row was marked for logical deletion, rather than immediately physically removing the data.

[0128] Step A13: The data row insertion record, data row content update record, and data row deletion record between the first timestamp and the third timestamp are used as the second data change information.

[0129] It should be noted that if the nearest common ancestor node cannot be found in the branch lineage graph, then all data row insertion records, data row content update records, and data row deletion records from the creation time to the current time point in the source data table are used as the first data change information. At the same time, all data row insertion records, data row content update records, and data row deletion records from the creation time to the current time point in the target data table are used as the second data change information. This ensures that even in the absence of a lineage optimization path, the differences between the two can still be accurately calculated through a complete full data baseline comparison, providing a complete and consistent data foundation for subsequent synchronization operations.

[0130] In this implementation, by acquiring key timestamps, the problem of traditional methods being unable to determine the precise comparison time range is solved, providing time boundaries for targeted collection of changed data. Based on these time boundaries, all data operation records of the source table and the target table within the corresponding time period are collected respectively. This solves the problem of excessive memory consumption and large processing latency caused by the full synchronization method having to load and process all historical data. By precisely limiting the data collection scope to incremental changes after the branch, and combining it with the special collection of deleted records, the accuracy of data status identification in a multi-version concurrency control environment is ensured. This lays a reliable data foundation for subsequent efficient and accurate difference calculation, directly improving the efficiency and accuracy of cross-cluster synchronization.

[0131] Based on the first and second embodiments of this application, in this third embodiment, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. In addition, the difference information includes information to be inserted, information to be updated, and information to be deleted.

[0132] Please refer to Figure 3 In step S04, the step of determining the difference information between the source data table and the target data table based on the first data change information and the second data change information includes steps S21 to S26:

[0133] Step S21: Determine the first data hash table of the source data table based on the first data change information, and determine the second data hash table of the target data table based on the second data change information;

[0134] It should be noted that the first data hash table is an in-memory data structure built upon the first data change information. It supports fast lookup of data rows that logically exist in the source data table at the time of comparison. It uses the unique identifier of the data row (e.g., a primary key column value) as the first hash index and stores the complete content of that data row as the associated value. The second data hash table is symmetrical to the first data hash table, the only difference being that the second data hash table is built upon the second data change information for the target data table, and is used to quickly find valid data rows in the target data table.

[0135] In one feasible implementation, step S21 includes steps B01 to B03:

[0136] Step B01: Based on the first data change information, construct a first candidate hash table and a first tombstone hash table for the source data table. The first candidate hash table is used to store the first hash index of all data rows in the source data table, and the first tombstone hash table is used to store the deleted records of the source data table.

[0137] Step B02: Based on the second data change information, construct a second candidate hash table and a second tombstone hash table for the target data table. The second candidate hash table is used to store the second hash index of all data rows in the target data table, and the second tombstone hash table is used to store the deleted records of the target data table.

[0138] It should be noted that the first tombstone hash table is an in-memory data structure built based on the first data change information to record logical deletion operations that occur in the table under an MVCC (Multi-Version Concurrency Control) environment. It uses the unique identifier of the deleted data row (also encoded as a hash key) as the key and the timestamp of the deletion operation as the associated value for storage, i.e., the deleted record. The second tombstone hash table is symmetrical to the first, except that it is built based on the second data change information for the target data table and is used to record deletion operations in the target data table. The first and second candidate hash tables are intermediate data index structures before the application of logical deletion filtering. These candidate hash tables are directly built based on their respective data change information (first / second data change information) and store the complete content of all inserted or updated data rows collected within a specified time range (after the branch from the nearest common ancestor node). When the candidate hash tables are built, they have not yet been cleaned according to the corresponding tombstone hash table (which stores deleted records). Therefore, they may contain some historical data rows that are logically marked as deleted but still physically exist in the change log.

[0139] Additionally, it should be noted that the first hash index and the second hash index refer to the encoded values ​​that uniquely identify the data row in the data hash table, serving as the lookup key. These can be the primary key of the data table (supporting single-column and composite primary keys). For data tables without a primary key, all visible columns are used as pseudo-primary keys. The first and second indexes respectively refer to the source and target data tables. Deleted records are entries stored in the tombstone hash table, representing the event that a row of data was marked as logically deleted at a certain point in time. These entries include the key of the deleted data row and the deletion timestamp, serving as the basis for subsequently filtering out invalid data from the data hash table and ensuring accurate data status comparisons.

[0140] Additionally, it should be noted that a sharded parallel processing mechanism can be employed during the construction of the first data hash table, the first tombstone hash table, and the second data hash table and the second tombstone hash table: the hash table is logically pre-divided into multiple shards. When data streams are injected, the system routes data rows to different shards based on hash keys. Each shard is handled by an independent processing unit (such as a thread or coroutine) responsible for constructing its local hash mapping. All processing units work in parallel, and the results are finally aggregated through a message channel, thereby fully utilizing multi-core CPU resources and significantly improving the throughput of large-scale data hash table construction.

[0141] Step B03: Update the first candidate hash table based on the deleted records in the first tombstone hash table to obtain the first data hash table; update the second candidate hash table based on the deleted records in the second tombstone hash table to obtain the second data hash table.

[0142] For example, incremental data change information (including first data change information and second data change information) is parsed. For the source data table, a first data hash table and a first tombstone hash table are constructed using its first data change information (i.e., the change records after the branch). The first data hash table is indexed by the hash value of the unique identifier key (such as the primary key) of each data row, storing the complete content of that row; the first tombstone hash table is indexed by the hash value of the identifier key of the deleted row, recording the timestamp of its deletion operation. Similarly, a second data hash table and a second tombstone hash table can be constructed using the second data change information of the target data table, as described above, without further elaboration here. During the construction process, the association between the hash index and the specific data row content is established. The first tombstone hash table is traversed, and based on the deletion records therein, all logically deleted data row entries are removed from the first data hash table. The same operation is performed on the second data hash table to ensure that subsequent comparisons are only performed between data that logically exist on both sides, accurately handling the deletion status under multi-version concurrency control.

[0143] Step S22: For any second hash index in the second data hash table, query the first hash index corresponding to the second hash index in the first data hash table;

[0144] Step S23: If no first hash index corresponding to the second hash index is found in the first data hash table, then the data row in the target data table associated with the second hash index is taken as the information to be inserted into the target data table.

[0145] It's important to note that the data to be inserted (INSERT) is a row that exists in the data hash table of the target table but not in the data hash table of the source table. This means that the data row is extra in the target table compared to the source table. In the process of synchronizing from the source table to the target table, this means that the target table is missing this row. Therefore, the operation required is to insert this row into the target table to make them consistent.

[0146] Step S24: If a first hash index corresponding to the second hash index is found in the first data hash table, and the content of the data row corresponding to the first hash index is inconsistent with the content of the data row corresponding to the second hash index, then the data row in the target data table associated with the second hash index is taken as the information to be updated in the target data table.

[0147] It's important to note that the information to be updated (UPDATE) refers to data rows whose hash indexes exist in both the source and target data tables (i.e., the first hash index corresponding to the second hash index can be found in the first hash table), but the complete data row content (i.e., the values ​​of non-primary key columns) associated with that identifier is inconsistent between the two hash tables. This indicates that the same row of data has been modified to different states on both sides. During synchronization, an update operation needs to be performed on the target table according to a strategy (usually based on the source table) to modify the content of that row in the target table to be consistent with that in the source table.

[0148] Additionally, it should be noted that during conflict detection, the update and delete operation batches generated after the nearest common ancestor node in the source data table, as well as the update and delete operation batches corresponding to the target data table, need to be sorted according to the primary key of the data row. A two-pointer algorithm is used to synchronously traverse the sorted source and target data table operation batches. By comparing the primary keys, the system can efficiently locate records that have the same primary key but different operation types or data contents. These records are marked as conflicts.

[0149] Additionally, it should be noted that each detected conflict is handled according to a preset conflict resolution strategy, including: failure mode (CONFLICT_FAIL), acceptance mode (CONFLICT_ACCEPT), and skip mode (CONFLICT_SKIP). These modes can be customized according to user needs. Failure mode (CONFLICT_FAIL) immediately halts the current difference calculation and synchronization process upon detecting a conflict and returns an error message containing the conflicting primary key value and operation type, suitable for business scenarios requiring strict consistency. Acceptance mode (CONFLICT_ACCEPT) prioritizes modifications in the target table, ignoring conflicting modifications to the same primary key in the source table, suitable for scenarios where the target branch data has higher priority. Skip mode (CONFLICT_SKIP) retains the conflict record but does not apply modifications to either the source or target table, meaning subsequent manual intervention is required, suitable for cautious merge operations.

[0150] Step S25: After traversing the second data hash table, for any first hash index in the first data hash table, query the second hash index corresponding to the first hash index in the second data hash table;

[0151] Step S26: If no second hash index corresponding to the first hash index is found in the second data hash table, then the data row in the source data table associated with the first hash index is taken as the information to be deleted from the target data table.

[0152] It's important to note that the "DELETE" information refers to the data row identifier that exists in the data hash table of the source table but not in the data hash table of the target table. This indicates that the data row is extra in the source table compared to the target table. In the synchronization process, this means that the target table should not contain this row, so the operation to be performed is to delete the data row with this identifier from the target table.

[0153] Additionally, it should be noted that for data tables without primary keys, differences need to be determined by comparing the values ​​of all columns in the table.

[0154] For example, each second hash index in the second data hash table (representing a valid row of data in the target table) is traversed. For each second hash index, a query is performed in the first data hash table. If no matching index is found, the data row is determined to exist only in the target data table and not in the source data table, thus marking it as information to be inserted. This means that during synchronization, this row of data needs to be operated on in reverse from the target table (depending on the synchronization direction, but logically, it's a difference that exists in the target table but not in the source table). If a matching index is found in the first data hash table, the content of the data row in the source data table associated with that index is further compared with the content of the data row in the target data table. If the content is inconsistent, it is determined that the data row has undergone different modifications in the source and target data tables, and the data row in the target table is marked as information to be updated. After completing the traversal of the target data table, each first hash index in the first data hash table is traversed. For each first hash index, a query is performed in the second data hash table. If no matching index is found, the data row is determined to exist only in the source data table and not in the target data table (or has been deleted), thus marking it as information to be deleted. This means that during synchronization, this row needs to be deleted from the target table.

[0155] In this implementation, by constructing a data hash table and a tombstone hash table in parallel and updating the data table based on the tombstone records, the high memory consumption and inaccurate handling of MVCC deletions in traditional methods are solved. This achieves accurate purification and efficient organization of effective data. By traversing the target table hash index and performing fast lookups and content comparisons in the source table hash table, the data rows that need to be inserted and updated are efficiently identified, solving the problem of low computational efficiency caused by full row-by-row comparison. By traversing the source table hash index in reverse, the data rows that need to be deleted are accurately identified, ensuring the integrity of the difference information. These steps work together to achieve fast and accurate difference calculation based on incremental change information, laying the foundation for subsequent synchronization of incremental differences only. This fundamentally solves the problems of bandwidth waste and high latency caused by full synchronization across clusters.

[0156] In one feasible implementation, step S04, the step of updating the target data table based on the difference information, includes steps B11-B12:

[0157] Step B11: Convert the difference information into a data operation instruction sequence and transmit the data operation instruction sequence to the target cluster where the target data table is located;

[0158] It's important to note that data manipulation commands are basic operation commands that can be directly parsed and executed by the database engine to modify the state of data tables. These commands correspond to specific categories in the difference information. For example, the INSERT or REPLACE INTO command, corresponding to the information to be inserted, is used to add new data rows to the table; the UPDATE command, corresponding to the information to be updated, is used to modify the content of existing data rows in the table; and the DELETE command, corresponding to the information to be deleted, is used to remove specified data rows from the table. Each command contains a clear target table identifier, operation type, and precise data location information (such as primary keys) and / or specific data values. A data manipulation command sequence is an ordered list of multiple independent data manipulation commands organized according to logical dependencies and execution order. The target cluster is a collection of cooperating computing nodes, storage resources, and database service processes that run and manage the target data table in a distributed database architecture. It is a logically or physically independent operating environment, separate from the source cluster running the source data table.

[0159] Additionally, it should be noted that, to adapt to different business scenarios and performance requirements, multiple output formats are supported when converting difference information into a sequence of data operation instructions. The appropriate format can be selected based on user commands or automatic judgment, such as result set output, count output, SQL file output, CSV file output, and object storage output. Result set output directly returns the difference information (including rows affected by INSERT, UPDATE, and DELETE information) in the form of a standard database result set. Each record includes the table name, operation type (INSERT / UPDATE / DELETE), and values ​​for all columns. This is suitable for interactive queries, data auditing, or scenarios requiring secondary analysis of the difference data. Count output, on the other hand, does not return specific difference information but only a count result, listing the number of rows affected by INSERT, UPDATE, and DELETE information. This is suitable for scenarios requiring rapid assessment of the scale of differences between two data tables and impact analysis before synchronization, as it does not require... Serializing and transmitting actual data is highly efficient; SQL file output refers to serializing the difference information into an independently executable Structured Query Language (SQL) file, which uses a database transaction format to ensure the atomicity of the application and is used for data synchronization and migration; CSV file output refers to automatically selecting to output all data of the target table in comma-separated value (CSV) format when the source data table is detected to be empty (i.e., the difference is actually the full data of the target table). Compared with SQL files, CSV format usually has higher parsing and loading efficiency in batch data import scenarios, and the file size is smaller; Object storage output supports writing the difference data file directly to cloud object storage services (such as Amazon S3, Azure BlobStorage, etc.), which can be resolved by parsing the path to obtain the storage configuration and select the writing strategy according to the type of cloud object storage service.

[0160] Specifically, path resolution includes two formats: local file path and stage path. Local path resolution refers to directly resolving the file system path and verifying the path's validity and permissions; stage path resolution involves resolving paths in the format `stage: / / stage_name / path`, retrieving stage configuration information from system metadata, including URL, access credentials, and region information, and converting it into the actual file service path, supporting various object storage backends such as S3, Azure Blob, and GCS.

[0161] Specifically, when selecting a write strategy, for file systems that support append-only writing (MutableFS), the append interface is used directly, supporting incremental writing of file content. An empty file is created first, and then content is written step by step through the append interface. This is suitable for local file systems and object storage that supports appending. For immutable file systems (ImmutableFS, such as standard S3), a piped streaming write mechanism is adopted. The main processing flow writes data through the pipe, and an independent write thread reads data from the pipe and writes it to the target storage. Data is transmitted asynchronously through the pipe, eliminating the need to cache the entire file in memory, which greatly reduces memory usage and is suitable for large-scale data output.

[0162] Specifically, for CSV file output scenarios, the system first automatically detects whether the source data table is empty (0 rows). When the source data table is detected to be empty, it automatically switches to CSV mode and directly uses the SELECT query interface to export the target table data, avoiding the overhead of building a hash table. It adopts a streaming write mechanism, querying and writing at the same time, reducing memory usage, making it suitable for batch data import. It is more efficient than SQL format and has a smaller file size.

[0163] Additionally, it's worth noting that during batch operations, for batch SQL generation, the batch VALUES syntax can be used to combine multiple values ​​into a single INSERT statement, reducing the number of SQL statements. For batch DELETE operations, the IN clause can be used to batch delete, combining multiple primary key values ​​into a single DELETE statement. For transaction optimization, multiple operations can be combined into a single transaction, reducing transaction overhead and network round trips. Furthermore, the batch size can be dynamically adjusted based on data size and system configuration to balance memory usage and processing efficiency.

[0164] Additionally, it should be noted that to optimize the performance, cost, and reliability of cross-cluster transmission, the following transmission optimization mechanisms can be selected when transmitting data operation command sequences to the target cluster where the target data table resides: object storage sharing, streaming transmission, and breakpoint resumption. Object storage sharing supports directly outputting difference files to object storage (e.g., S3), accessible to both the source cluster (the cluster containing the source data table) and the target cluster, avoiding direct cross-cluster network transmission, making it suitable for cross-regional and cross-availability zone scenarios. Streaming transmission supports simultaneous computation and transmission, reducing memory usage; data is transmitted asynchronously through pipelines, eliminating the need to cache the entire file in memory. Breakpoint resumption allows transmission to resume from the point of interruption after network interruption, improving the reliability of cross-cluster synchronization.

[0165] It should be noted that, in order to ensure the efficiency, stability and controllability of the data synchronization method when dealing with massive amounts of data, a series of performance optimization mechanisms can be selected: asynchronous processing and producer-consumer pattern, parallel processing mechanism, memory management optimization mechanism and context cancellation mechanism.The core of the asynchronous processing and producer-consumer model adopts an asynchronous processing architecture, decoupling the two main stages of difference calculation and result output / processing. The difference calculation unit acts as a "producer," placing batches of calculated difference information into a buffered message channel; the independent output processing unit acts as a "consumer," retrieving and processing these batches from the channel (such as format conversion and writing to files). This achieves pipelined processing and streaming data processing, effectively avoiding the accumulation of a large number of intermediate results in memory due to waiting for I / O or subsequent processing, thus significantly reducing the system's peak memory usage. In the parallel processing mechanism, the construction of data hash tables and tombstone hash tables, as well as the subsequent traversal of the hash tables for difference calculation... During heterogeneous detection, a sharding strategy is employed. The hash table is logically divided into multiple shards, each operated by an independent processing thread or coroutine. All shards process their assigned data in parallel, and the results are aggregated via a message channel. The shard size can be dynamically adjusted based on data distribution to achieve load balancing. This mechanism fully utilizes the computing resources of modern multi-core CPUs, transforming large-scale data comparison tasks from serial to parallel processing, significantly improving overall throughput and computation speed. Furthermore, since each processing unit operates only on its exclusive shard data and communicates with other threads via a message channel, lock contention can be avoided on critical data comparison paths, further enhancing parallel efficiency. Memory management optimization is also implemented. In this system, a memory pool is maintained for frequently created and destroyed data batch objects. By reusing allocated memory blocks, the overhead and latency caused by dynamic memory allocation and garbage collection are significantly reduced, improving processing performance. After completing the difference calculation, large memory data structures such as hash tables used to store source and target table data are immediately released, rather than waiting for the entire process to end, accelerating memory resource turnover. This enables more efficient handling of continuous or concurrent synchronous tasks. From data collection and difference calculation to result output, the entire process is designed as a streaming process, with data flowing in batches. This avoids the need to load the entire table data into memory at any stage, ensuring the ability to handle ultra-large-scale tables and allowing real-time monitoring of internal processes. The system stores memory usage and implements corresponding optimization strategies when memory is insufficient. The context cancellation mechanism provides a context management mechanism for long-running difference analysis operations. Cancellation checkpoints are set in critical loops and long-running operations. Users can request termination of ongoing synchronization tasks by sending a cancellation signal (e.g., interrupting the operation on the client side). Upon receiving the signal, the system stops receiving new data, completes the current batch processing, cleans up allocated resources, and returns to a cancelled state. This mechanism gives users control over long-running tasks, avoiding resource waste and unnecessary waiting due to misoperation or changes in requirements, thus improving system operability and user experience.

[0166] Step B12: In the target cluster, execute each data operation instruction in the data operation instruction sequence in sequence to update the target data table.

[0167] In this embodiment, by serializing the difference information into a compact sequence of data operation instructions for transmission, the problem of excessive network bandwidth consumption and large transmission delay caused by transmitting full data or original difference records in traditional methods is solved. This achieves efficient data flow that only synchronizes necessary incremental changes. By atomically executing the received instruction sequence in sequence in the target cluster, the problem of data inconsistency caused by partial execution of synchronization operations in cross-cluster network interaction environments is solved, ensuring the reliability and eventual consistency of synchronization operations and significantly improving the overall efficiency and success rate of cross-cluster data synchronization.

[0168] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data synchronization method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0169] This application also provides a data synchronization system; please refer to [link / reference]. Figure 4 The data synchronization system includes:

[0170] Graph construction module 10 is used to construct a branch lineage graph between the source data table and the target data table;

[0171] Node lookup module 20 is used to determine the nearest common ancestor node between the source data table and the target data table based on the branch lineage graph;

[0172] The incremental acquisition module 30 is used to acquire the first data change information from the nearest common ancestor node to the source data table, and the second data change information from the nearest common ancestor node to the target data table;

[0173] The data synchronization module 40 is used to determine the difference information between the source data table and the target data table based on the first data change information and the second data change information, and to update the target data table based on the difference information so that the updated target data table is synchronized with the source data table.

[0174] The data synchronization system provided in this application, employing the data synchronization method described in the above embodiments, can solve the technical problem of poor synchronization performance. Compared with the prior art, the beneficial effects of the data synchronization system provided in this application are the same as those of the data synchronization method described in the above embodiments, and other technical features of the data synchronization system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0175] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data synchronization method in Embodiment 1 above.

[0176] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0177] like Figure 5 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0178] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0179] The electronic device provided in this application, employing the data synchronization method described in the above embodiments, can solve the technical problem of poor synchronization performance. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the data synchronization method provided in the above embodiments, and other technical features of the electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0180] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0181] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0182] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data synchronization method described in the above embodiments.

[0183] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0184] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0185] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the data synchronization device to: construct a branch lineage graph between a source data table and a target data table; determine the nearest common ancestor node between the source data table and the target data table based on the branch lineage graph; collect first data change information from the nearest common ancestor node to the source data table, and second data change information from the nearest common ancestor node to the target data table; determine the difference information between the source data table and the target data table based on the first data change information and the second data change information, and update the target data table based on the difference information, so that the updated target data table is synchronized with the source data table.

[0186] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0188] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0189] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data synchronization method, thereby solving the technical problem of poor synchronization performance. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data synchronization method provided in the above embodiments, and will not be repeated here.

[0190] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data synchronization method described above.

[0191] The computer program product provided in this application can solve the technical problem of poor synchronization effect. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the data synchronization method provided in the above embodiments, and will not be repeated here.

[0192] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method of data synchronization, the method comprising: The data synchronization method includes: Construct a branch lineage graph between the source data table and the target data table. The branch lineage graph is determined based on the historical cloning relationship records between the source data table, the target data table, and the associated data table. The historical cloning relationship records are structured data entries for each table cloning operation. The branch lineage graph is used to record the cloning derivation relationships between all data tables. Each directed edge represents a cloning operation. Based on the branch lineage graph, the nearest common ancestor node between the source data table and the target data table is determined. The nearest common ancestor node is the common ancestor node that is closest to the source node corresponding to the source data table and the target node corresponding to the target data table in the branch lineage graph. It is obtained by aligning the source node and the target node with depth values ​​and then moving them up layer by layer along the path of the corresponding parent node in the branch lineage graph to the same node. Collect first data change information from the nearest common ancestor node to the source data table, and second data change information from the nearest common ancestor node to the target data table. The first data change information is a set of data row change records between the first timestamp corresponding to the nearest common ancestor node and the corresponding timestamp of the source data table, and the second data change information is a set of data row change records between the first timestamp corresponding to the nearest common ancestor node and the corresponding timestamp of the target data table. Based on the first data change information and the second data change information, the difference information between the source data table and the target data table is determined, and the target data table is updated based on the difference information so that the updated target data table is synchronized with the source data table.

2. The data synchronization method of claim 1, wherein, The steps for constructing the branch lineage graph between the source data table and the target data table include: Obtain the historical clone relationship record, and extract the parent-child relationship of the source data table, the target data table, and the associated data table from the historical clone relationship record, wherein the associated data table is a data table that has an association relationship with the source data table and the target data table; Based on the parent-child relationship, directed connections are established between the source data table, the target data table, and the associated data table to obtain a directed acyclic graph; Traverse the directed acyclic graph (DAG), record the depth values ​​between each node in the DAG and the root node in the DAG, and use the DAG with the depth values ​​as a branch lineage graph.

3. The data synchronization method of claim 1, wherein, The step of determining the nearest common ancestor node between the source data table and the target data table based on the branch lineage graph includes: Locate the source node corresponding to the source data table and the target node corresponding to the target data table in the branch lineage map, and determine the first depth value corresponding to the source node and the second depth value corresponding to the target node; If the first depth value and the second depth value are different, the node with the larger depth value among the source node and the target node is taken as the backtracking node, and the backtracking node is moved along the backtracking path until the first depth value and the second depth value are the same. The backtracking path is the connection path from the backtracking node to the corresponding parent node of the backtracking node. When the first depth value and the second depth value are the same, the source node is moved back layer by layer along the connection path of the source node's corresponding parent node, and the target node is moved back layer by layer along the connection path of the target node's corresponding parent node, until the source node and the target node move to the same node, and the same node is taken as the nearest common ancestor node.

4. The data synchronization method of claim 1, wherein, The steps of collecting the first data change information from the nearest common ancestor node to the source data table, and the second data change information from the nearest common ancestor node to the target data table, include: Obtain the first timestamp, the second timestamp corresponding to the source data table, and the third timestamp corresponding to the target data table; The data row insertion record, data row content update record, and data row deletion record between the first timestamp and the second timestamp are used as the first data change information; The data row insertion record, data row content update record, and data row deletion record between the first timestamp and the third timestamp are used as the second data change information.

5. The data synchronization method as described in claim 1, characterized in that, The difference information includes information to be inserted, information to be updated, and information to be deleted; The step of determining the difference information between the source data table and the target data table based on the first data change information and the second data change information includes: A first data hash table for the source data table is determined based on the first data change information, and a second data hash table for the target data table is determined based on the second data change information. For any second hash index in the second data hash table, query the first hash index corresponding to the second hash index in the first data hash table; If no first hash index corresponding to the second hash index is found in the first data hash table, then the data row in the target data table associated with the second hash index is taken as the information to be inserted into the target data table. If a first hash index corresponding to the second hash index is found in the first data hash table, and the content of the data row corresponding to the first hash index is inconsistent with the content of the data row corresponding to the second hash index, then the data row in the target data table associated with the second hash index is taken as the information to be updated in the target data table. After traversing the second data hash table, for any first hash index in the first data hash table, query the second data hash table for the second hash index corresponding to the first hash index; If no second hash index corresponding to the first hash index is found in the second data hash table, then the data row in the source data table associated with the first hash index is taken as the information to be deleted from the target data table.

6. The data synchronization method as described in claim 5, characterized in that, The steps of determining the first data hash table of the source data table based on the first data change information and determining the second data hash table of the target data table based on the second data change information include: Based on the first data change information, a first candidate hash table and a first tombstone hash table are constructed for the source data table. The first candidate hash table is used to store the first hash index of all data rows in the source data table, and the first tombstone hash table is used to store the deletion records of the source data table. Based on the second data change information, a second candidate hash table and a second tombstone hash table are constructed for the target data table. The second candidate hash table is used to store the second hash index of all data rows in the target data table, and the second tombstone hash table is used to store the deleted records of the target data table. The first candidate hash table is updated based on the deleted records in the first tombstone hash table to obtain the first data hash table, and the second candidate hash table is updated based on the deleted records in the second tombstone hash table to obtain the second data hash table.

7. The data synchronization method as described in claim 1, characterized in that, The step of updating the target data table based on the difference information includes: The difference information is converted into a data operation instruction sequence, and the data operation instruction sequence is transmitted to the target cluster where the target data table is located; In the target cluster, each data operation instruction in the data operation instruction sequence is executed sequentially to update the target data table.

8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data synchronization method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data synchronization method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the data synchronization method as described in any one of claims 1 to 7.

Images