A method and product for data synchronization

Abstract

The embodiment of the present application provides a kind of data synchronization method and product, the method comprises: reading the data of the i-th partition from target table;The record of the i-th partition read from the target table is loaded into the hash table built in memory;All records of the corresponding partition of source table and the i-th partition are traversed, and are compared with the record stored in the hash table one by one, and the record matched with each traversal record is dynamically removed from the hash table;Confirm that there is un-deleted record in the hash table, then mark the un-deleted record as "deletion";At least according to all deletion marks, the synchronization operation between the data of the i-th partition and the corresponding partition of the source table is executed;Repeat the above steps until all partitions are processed.The embodiment of the present application solves the problems of permission dependence, performance bottleneck and poor adaptability in traditional data synchronization by comparing in-memory hash table and reverse deletion marking method.

Description

Technical Field

[0001] This application relates to the field of big data, and specifically, the embodiments of this application relate to a data synchronization method and product. Background Technology

[0002] Building or acquiring high-quality datasets is crucial for the success of machine learning, artificial intelligence, and data analytics projects. High-quality datasets accurately reflect reality, are error-free or distortion-free, have no missing key fields, cover sufficient sample size and diversity, maintain consistent data format and units, and ensure data characteristics are strongly correlated with the problem, avoiding redundant information. Only by building high-quality datasets can data applications have a positive impact. However, in the process of building big data datasets, data consistency is the core principle for ensuring that data maintains logical correctness, integrity, and synchronization throughout distributed systems, databases, or business processes; therefore, it is particularly important for building high-quality datasets.

[0003] Obtaining incremental database data is crucial and frequently used in production environments. Common methods include triggers, timestamps, full table comparisons, and log comparisons. However, each method has limitations. Triggers require intrusion into the database's internal trigger components, impacting the business system, and the conditions for creating triggers may not be available in production environments. Timestamps are fast, but require the source business system database to have robust timestamp support; not all database designers add effective timestamps. Full table comparisons are non-intrusive, but involve comparing a large amount of data each time, making them slow. Log comparisons require the database to have audit logging enabled and log access permissions granted.

[0004] In the current system operating environment, data security requirements are high. In production environments, database administrator privileges are generally unavailable, making triggers and logging methods difficult to apply universally. Therefore, improving the speed of incremental data identification and achieving non-intrusive incremental data identification and synchronization have become urgent technical challenges. Summary of the Invention

[0005] The purpose of this application is to provide a data synchronization method and product. Embodiments of this invention can process multiple partitions in parallel through partitioning, construct a hash table to find differences between the target table and the source table, and identify records that need to be added or updated by traversing and comparing the data in the source table. Simultaneously, matching records are deleted from the hash table, leaving only those records in the hash table that exist in the target table but not in the source table (i.e., records that need to be deleted). Embodiments of this application solve the problems of permission dependency, performance bottlenecks, and poor adaptability in traditional data synchronization through in-memory hash table comparison and reverse deletion marking.

[0006] In a first aspect, embodiments of this application provide a data synchronization method, the method comprising: reading data from a target table of the i-th partition, wherein i is an integer greater than or equal to 1 and less than the total number of partitions, the i-th partition being obtained by partitioning the target table using a consistent hash function; loading the records of the i-th partition read from the target table into a hash table constructed in memory, wherein the key of the hash table is used to store record keys, the record keys including primary keys or composite primary keys, and the value of the hash table is used to store field values; traversing all records of the source table and the corresponding partition of the i-th partition, and comparing each record with the records stored in the hash table, dynamically removing records matching each traversed record from the hash table, wherein the matching records include: matching record keys, or matching both record keys and field values; confirming that there are records in the hash table that have not been deleted, and marking the records that have not been deleted as "deleted"; performing a synchronization operation between the data of the i-th partition and the data of the corresponding partition of the source table based on at least all deletion marks; repeating the above steps until all partitions have been processed.

[0007] The embodiments of this application establish an in-memory mirror of all records in each partition of the target table using a hash table. During the unidirectional traversal of the source table data, the state of this mirror is dynamically maintained, ultimately calculating the data that needs to be "deleted" from the target table accurately and efficiently. Partitioning allows for parallel processing of multiple partitions, constructing a hash table to find records showing differences between the target and source tables. By traversing and comparing the source table data, newly added and updated records are deleted from the hash table. This leaves the records in the hash table that exist in the target table but not in the source table (i.e., the records that need to be deleted). In other words, the embodiments of this application utilize small datasets within partitions for fast in-memory comparisons and use hash tables to mark the operation status, cleverly using a combination of "partitioning + in-memory hash tables" to circumvent the performance bottleneck of database sorting. Compared to existing technologies, the advantages of the embodiments of this application include: no special database permissions or modifications are required (unlike trigger and log schemes), it does not rely on timestamp fields, and it supports composite primary key scenarios. In other words, the embodiments of this application overcome the problems of permission dependence, strong invasiveness, or low performance of traditional incremental data synchronization schemes (triggers, timestamps, logs, full table comparisons).

[0008] In some embodiments, traversing all records of the source table and the partition corresponding to the i-th partition, and comparing them one by one with the records stored in the hash table, and dynamically removing records that match each traversed record from the hash table, includes: for the j-th traversed record: confirming that the record key of the j-th record is the same as the record key of the k-th record in the hash table; retrieving the k-th record from the hash table, comparing the field values ​​of the k-th record and the j-th record, if the field values ​​are confirmed to be the same, then marking the j-th record as "duplicate", if the field values ​​are confirmed to be different, then marking the j-th record as "updated", and deleting the k-th record from the hash table.

[0009] The embodiments of this application significantly improve data processing speed by deleting matching records in a hash table that have the same record key as the traversed record (i.e., are related to the primary key or have the same composite primary key), or whose record key and the field value corresponding to the record key are the same, thereby quickly identifying the data to be deleted in the i-th partition of the target table.

[0010] In some embodiments, traversing all records of the source table and the partition corresponding to the i-th partition, and comparing them one by one with the records stored in the hash table, and dynamically removing records matching each traversed record from the hash table, includes: traversing all records of the source table and the partition corresponding to the i-th partition; for the j-th traversed record: confirming that the record key of the j-th record is the same as the record key of the k-th record in the hash table; retrieving the k-th record from the hash table, comparing the field values ​​of the k-th record and the j-th record; if the field values ​​are confirmed to be the same, marking the j-th record as "duplicate"; if the field values ​​are confirmed to be different, marking the j-th record as "updated" and deleting the k-th record from the hash table; repeating the above comparison process for each traversed record of the corresponding partition; confirming that there are records in the hash table that have not been deleted, and marking the records that have not been deleted as "deleted", includes: confirming that after traversing all records of the source table and the partition corresponding to the i-th partition, all remaining records in the hash table are marked as "deleted".

[0011] The embodiments of this application provide a method for identifying records that exist in the target table but not in the original table, thereby improving data processing speed.

[0012] In some embodiments, the method further includes: traversing all records of the source table and the partition corresponding to the i-th partition; for the m-th record being traversed: if the record key of the m-th record does not exist in the hash table, then the m-th record is marked as "new"; the step of performing a synchronization operation between the data of the i-th partition and the data of the partition corresponding to the source table based at least on all deletion markers includes: performing a synchronization operation between the data of the i-th partition and the data of the partition corresponding to the source table based at least on all deletion markers and all new markers.

[0013] Embodiments of this application provide a method for identifying newly added and deleted records in a partition, thereby improving the speed and accuracy of data synchronization processing.

[0014] In some embodiments, the method further includes: calculating a consistent hash value based on the target table and the source table; and partitioning the target table and the source table based on the consistent hash value.

[0015] Some embodiments of this application provide a method for partitioning a source table and a target table based on a consistent hash value, thereby increasing the probability that similar data will be assigned to the same partition.

[0016] In some embodiments, calculating a consistent hash value based on the target table and the source table includes: obtaining the maximum and minimum values ​​of the primary key IDs in the target table to obtain the range of primary key IDs in the target table; querying the source table for primary key IDs that are not within the range of the maximum and minimum values ​​based on the range of primary key IDs in the target table to obtain records to be marked, and marking the records to be marked as "new"; querying the source table for primary key IDs that are within the range of the maximum and minimum values ​​based on the range of primary key IDs in the target table to obtain records to be partitioned; and calculating a consistent hash value corresponding to the source table based on the records to be partitioned.

[0017] Some embodiments of this application use the range of primary key IDs to initially filter out new data, which can further reduce the number of subsequent partition data comparisons and improve the speed of data processing.

[0018] In some embodiments, calculating a consistent hash value based on the target table and the source table includes: calculating an MD5 hash value based on the primary key ID corresponding to the record to be partitioned in the target table or the source table, or calculating an MD5 hash value based on the composite primary key corresponding to the record to be partitioned in the target table or the source table; partitioning the target table and the source table based on the consistent hash value includes: partitioning the record to be partitioned in the target table and the source table based on the MD5 hash value.

[0019] Some embodiments of this application perform partitioning by calculating MD5 hash values, thereby improving the accuracy of the partitioning results.

[0020] Secondly, some embodiments of this application provide a data synchronization apparatus, the apparatus comprising: a target table data reading module configured to read data from the i-th partition of the target table, wherein i is an integer greater than or equal to 1 and less than the total number of partitions, and the i-th partition is obtained by partitioning the target table using a consistent hash function; a hash table construction module configured to load the records of the i-th partition read from the target table into a hash table constructed in memory, wherein the key of the hash table is used to store record keys, the record keys including primary keys or composite primary keys, and the value of the hash table is used to store field values; and a hash table record deletion module configured to traverse the source table and... The process involves processing all records in the i-th partition and comparing them one by one with the records stored in the hash table. Records matching each traversed record are dynamically removed from the hash table. These matching records include: matching record keys, or matching both the record key and the field value. A deletion marker module is configured to mark any undeleted records in the hash table as "deleted." A synchronization module is configured to perform synchronization operations between the data in the i-th partition and the data in the corresponding partition of the source table, based on at least all deletion markers. A loop execution determination module is configured to terminate the current loop upon confirming that all partitions have been processed.

[0021] Thirdly, some embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, can implement the method described in any of the embodiments included in the first aspect.

[0022] Fourthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can implement the method described in any of the embodiments included in the first aspect. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram illustrating the composition of a data synchronization system provided in this application embodiment;

[0025] Figure 2 One of the flowcharts for the data synchronization method provided in the embodiments of this application;

[0026] Figure 3 A second flowchart illustrating the data synchronization method provided in this application embodiment;

[0027] Figure 4 A block diagram illustrating the composition of a data synchronization apparatus provided in the embodiments of this application;

[0028] Figure 5 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] Glossary

[0032] A database is a logical container, a complete and organized collection of data. It contains all the system resources needed to manage the data (such as tables, views, indexes, stored procedures, user permissions, etc.).

[0033] A table is like a specific folder within the "file cabinet" of a database. It's a structured collection of data, consisting of rows and columns. Each table has a unique name.

[0034] The source database contains one or more tables, and the specific table that needs to be synchronized is the source table.

[0035] The target database contains one or more tables, and the specific table that receives the synchronized data is the target table.

[0036] The source database is the provider (Source) of the data, and is the authoritative source of the data.

[0037] The target database is the recipient of the data and needs to be consistent with the data in the source database.

[0038] The source table and the target are the specific entity objects that perform the data synchronization operation.

[0039] The meanings of some terms in the embodiments of this application can be found in Table 1 below.

[0040] Table 1 Summary of related vocabulary information

[0041]

[0042] This invention provides a data synchronization method based on a hash-based improved data consistency synchronization algorithm. It achieves non-intrusive incremental data identification by introducing an in-memory hash table to quickly compare data between the source and target tables. Compared with existing data comparison technologies, the synchronization method provided in this application does not require database administrator privileges, reduces queries and the amount of comparison data, thereby improving incremental identification efficiency. The structure of the source database table (hereinafter referred to as the source table) does not need to be adjusted, and logging is not required.

[0043] Please refer to Figure 1 , Figure 1 A system for performing a data synchronization method is provided in some embodiments of this application. The system includes: an external API gateway, a synchronization service area including a synchronization task scheduler and a data synchronizer cluster, a target database, a source database, and a management console.

[0044] Source database: This is the data source being synchronized, such as Oracle, MySQL, or SQL Server in a production environment. The source database is typically located in the internal production network area, with the highest security level. This source database can have a read-only account (SELECT privileges).

[0045] Target database: This is the database that receives incremental data, such as a data warehouse, reporting database, or backup database. The source database is located in the data analysis or backup area and is isolated from the production network. Permission requirements for the source database include read and write permissions (INSERT, UPDATE, DELETE).

[0046] The Synchronization Service Cluster is the core of the architecture and it is recommended to deploy it in a separate Synchronization Service Zone (DMZ) to decouple the network and functions.

[0047] The synchronization task scheduler is responsible for starting and stopping synchronization tasks, scheduling them on a timer, monitoring and alerting, and load balancing. It receives instructions from the management console and distributes them to the synchronizer nodes. The synchronization task scheduler can be deployed as a standalone machine or in a master-slave configuration to ensure high availability of the scheduling function.

[0048] A data synchronizer (i.e., a synchronizer in a data synchronizer cluster) is a computing node and an entity that executes the data synchronization methods of some embodiments of this application. Each node is responsible for synchronizing data in one or more partitions. For example, each data synchronizer can be configured to perform: SELECT data from the source database, perform hash calculations and partition comparisons in memory; generate incremental SQL statements; and execute INSERT / UPDATE / DELETE statements to the target database.

[0049] It should be noted that the data synchronizer in this embodiment adopts a clustered deployment, and multiple nodes can process different partitions in parallel, thereby achieving horizontal scaling processing capacity that increases linearly with the number of nodes.

[0050] The Management Console provides a web UI or API for configuring data sources, mapping relationships, synchronization strategies (full / incremental, scheduled / manual), and monitoring task status.

[0051] An API Gateway (optional) provides the following functions: if the algorithm is provided as a microservice, the gateway is responsible for routing, authentication, rate limiting, and logging.

[0052] In some embodiments of this application, the system also includes security and network infrastructure, particularly in the government and financial sectors. For example, these network elements include: firewalls, database passwords, API keys, and other sensitive information that should be managed using tools such as Vault or KMS, rather than stored in plaintext in configuration files.

[0053] For example, using Figure 1 The methods for data synchronization performed by the system include:

[0054] Initialization configuration: Administrators configure the connection information for the source and target databases through the management console.

[0055] Configure a synchronization task (e.g., start synchronizing at 1 AM every day).

[0056] Task triggered:

[0057] The scheduler triggers synchronization tasks according to a predetermined time.

[0058] The scheduler assigns tasks to one or more available synchronizer nodes based on the current cluster load.

[0059] Data synchronization is executed, meaning the data synchronizer node executes the algorithm according to the flowchart:

[0060] Pull: The synchronizer establishes a connection with the source database, executes a SELECT query, and pulls the data into local memory in batches.

[0061] Calculation: Consistent hash calculation, partitioning, hash table construction, and difference comparison are performed in the data synchronizer's memory. This process does not consume database resources.

[0062] Push: The data synchronizer establishes a connection with the target database and commits the calculated incremental data (INSERT, UPDATE, DELETE statements) to the database in a transactional manner.

[0063] Partition parallel processing: Multiple partitions can be processed in parallel by different data synchronizer nodes, greatly improving speed.

[0064] Completion and Reporting: The data synchronizer node reports the task status (success / failure) to the scheduler. The scheduler updates the task status, logs it, and sends notifications. The management console interface updates to display the latest synchronization status and performance metrics.

[0065] Figure 1 The synchronous task scheduler and the data synchronizer cluster can exchange management commands. The data synchronizer cluster writes incremental data to the target database and exchanges data with the source database, for example, by pulling data from the source.

[0066] It should be noted that in scenarios with high security requirements, the synchronization service cluster can be directly deployed within a trusted network at the source or target end, or even a one-way network gateway technology can be used to achieve one-way synchronization of data from a high-security area to a low-security area. In some embodiments of this application, synchronization operations can be logged in detail and connected to a unified auditing platform. For example, a high-speed network should be deployed between the data synchronizer and the database. The synchronizer's local memory and CPU configuration should be high to ensure computing speed and reduce network connection time. In some embodiments of this application, the synchronizer acts as an independent intermediate layer, enabling "reading" from the production database and "writing" to the target database with minimal permissions; the calculation process is completed in the synchronizer's memory, resulting in the highest performance; and the components are modular, making it easy to expand and maintain.

[0067] The following is combined Figure 2 An exemplary method for data synchronization performed by a data synchronizer is illustrated.

[0068] like Figure 2 As shown, some embodiments of this application provide a data synchronization method, the method comprising:

[0069] S110, Read the data of the i-th partition from the target table, where i is an integer greater than or equal to 1 and less than the total number of partitions, and the i-th partition is obtained by partitioning the target table using a consistent hash function.

[0070] It should be noted that, in some embodiments of this application, data from multiple partitions can be read simultaneously, and all records read from each partition can be stored in a hash table constructed in memory for that corresponding partition.

[0071] S120, the records of the i-th partition read from the target table are loaded into a hash table constructed in memory, wherein the key of the hash table is used to store record keys, the record keys including primary keys or composite primary keys, and the value of the hash table is used to store field values ​​(i.e., the specific content of the record).

[0072] In other words, in some embodiments of this application, the key of the hash table is the primary key ID (for cases where a single-column primary key exists); in other embodiments of this application, the key of the hash table is a composite primary key (for cases where a single-column primary key does not exist). It should be noted that in some embodiments of this application, the composite primary key and the primary key ID are collectively referred to as the record key.

[0073] S130, traverse all records of the source table and the corresponding partition of the i-th partition, and compare them one by one with the records stored in the hash table. Dynamically remove records that match each traversed record from the hash table. The matching records include: record key matching, or record key and field value matching respectively.

[0074] In other words, in some embodiments of this application, records with different field values ​​from the source table but the same record key, and records with the same field values ​​and record key values, are all deleted from the hash table as deletion entries. This identifies both duplicate records in the source table relative to the target table and updated records in the source table relative to the target table. Then, both duplicate and updated records are deleted from the hash table. It is understood that some embodiments of this application also require marking the corresponding records in the source table with duplicate and updated tags for these deleted records, and then using these tags to adjust the field values ​​of the corresponding entries in the target table during the subsequent synchronization phase.

[0075] S140, if it is confirmed that there are records in the hash table that have not been deleted, then mark the records that have not been deleted as "deleted".

[0076] It is understood that in some embodiments of this application, there are no records in the hash table that have not been deleted. In this case, it can be assumed that there is no data in the target table that is not in the source table. Therefore, it is not necessary to delete any records in the target table during the synchronization phase. In some embodiments of this application, after deleting duplicate records and updated records from the hash table, there are still remaining records. These remaining records are records that exist only in the target table and not in the source table. They are also records that should be deleted during the synchronization phase. Therefore, these records are marked for deletion.

[0077] S150, perform a synchronization operation between the data of the i-th partition and the data of the corresponding partition of the source table, based at least on all deletion markers.

[0078] It should be noted that some embodiments of this application can also identify newly added records (i.e., records that exist in the source table but do not exist in the corresponding partition of the target table) and updated records (i.e., records that exist in the source table but exist in the target table and the field values ​​corresponding to the record keys are different in the source table and the target table) by using record keys. Then, it is necessary to synchronize the data in the target table and the source table based on the newly added and updated records.

[0079] Repeat the above steps until all partitions have been processed. It should be noted that, in some embodiments of this application, the above operations can be performed on each partition simultaneously until all partitions have been processed, or, in some embodiments of this application, the above operations can be performed on some partitions simultaneously, and then the above operations can be repeated multiple times or once on the remaining partitions until all partitions have been processed.

[0080] The embodiments of this application establish an in-memory mirror of all records in each partition of the target table using a hash table. During the unidirectional traversal of the source table data, the state of this mirror is dynamically maintained, ultimately calculating the data that needs to be "deleted" from the target table accurately and efficiently. Partitioning allows for parallel processing of multiple partitions, constructing a hash table to find records showing differences between the target and source tables. By traversing and comparing the source table data, duplicate and updated records are deleted from the hash table. This leaves only the records in the hash table that exist in the target table but not in the source table (i.e., the records that need to be deleted). In other words, the embodiments of this application utilize small datasets within partitions for fast in-memory comparisons and use hash tables to mark the operation status, cleverly using a combination of "partitioning + in-memory hash tables" to circumvent the performance bottleneck of database sorting. Compared to existing technologies, its advantages are obvious: it does not require any special database permissions or modifications (unlike trigger and log schemes), does not rely on timestamp fields, and supports composite primary key scenarios. The embodiments of this application overcome the problems of permission dependencies, strong invasiveness, or low performance of traditional incremental data synchronization schemes (triggers, timestamps, logs, full table comparisons).

[0081] The following example illustrates... Figure 2 The implementation process of the relevant steps.

[0082] In some embodiments of this application, step S130 involves traversing all records of the source table and the corresponding partition of the i-th partition, comparing each record with the records stored in the hash table, and dynamically removing records matching each traversed record from the hash table. This includes: for the j-th traversed record: confirming that the record key of the j-th record is the same as the record key of the k-th record in the hash table (i.e., if the record has a primary key ID, then the record keys are the same, meaning the primary key ID of the j-th record is the same as the primary key ID of the k-th record; if the record does not have a primary key ID, then the record keys are the same, meaning the composite primary key of the j-th record is the same as the composite primary key of the k-th record); retrieving the k-th record from the hash table, comparing the field values ​​of the k-th record and the j-th record; if the field values ​​are confirmed to be the same, then marking the j-th record as "duplicate"; if the field values ​​are confirmed to be different, then marking the j-th record as "updated" and deleting the k-th record from the hash table. Where j is an integer greater than or equal to 1, and the maximum value of j is the total number of records in the corresponding partition, and k is an integer greater than or equal to 1, and the maximum value of k is the total number of records in the i-th partition.

[0083] The embodiments of this application delete data from a hash table that has the same record key as the traversed record (i.e., related to the primary key or the same composite primary key), or where the record key and the field value corresponding to the record key are the same. This allows for the filtering of data to be deleted from the i-th partition, enabling rapid identification of data to be deleted in the target table and significantly improving data processing speed.

[0084] In some embodiments of this application, S130 exemplarily includes: traversing all records of the source table and the corresponding partition of the i-th partition; for the j-th record being traversed: confirming that the record key of the j-th record is the same as the record key of the k-th record in the hash table; retrieving the k-th record from the hash table, comparing the field values ​​of the k-th record and the j-th record; if the field values ​​are confirmed to be the same, marking the j-th record as "duplicate"; if the field values ​​are confirmed to be different, marking the j-th record as "updated" and deleting the k-th record from the hash table; confirming that there are records in the hash table that have not been deleted, and marking the records that have not been deleted as "deleted", includes: repeatedly executing the matching process of each traversed record; after traversal is completed, marking all remaining records in the hash table as "deleted". Wherein, j is an integer greater than or equal to 1, and the maximum value of j is the total number of records in the corresponding partition; k is an integer greater than or equal to 1, and the maximum value of k is the total number of records included in the i-th partition.

[0085] The embodiments of this application provide a method for identifying records that exist in the target table but not in the original table, thereby improving data processing speed.

[0086] It should be noted that some embodiments of this application also provide a method for identifying newly added records in the source table relative to the target table using a hash table.

[0087] For example, in some embodiments of this application, the method further includes: traversing all records of the source table and the corresponding partition of the i-th partition; for the m-th record traversed: if the record key of the m-th record does not exist in the hash table, then the m-th record is marked as "new"; the step of performing a synchronization operation between the data of the i-th partition and the data of the corresponding partition of the source table based at least on all deletion markers includes: performing a synchronization operation between the data of the i-th partition and the data of the corresponding partition of the source table based at least on all deletion markers and all addition markers. Wherein, m is an integer greater than or equal to 1, and m is less than the total number of records included in the corresponding partition.

[0088] Embodiments of this application provide a method for identifying newly added and deleted records in a partition, thereby improving the speed and accuracy of data synchronization processing.

[0089] It should be noted that in some embodiments of this application, consistent hash values ​​are used to partition the target table and the source table.

[0090] For example, in some embodiments of this application, the method further includes:

[0091] The first step is to calculate a consistent hash value based on the target table and the source table.

[0092] For example, in some embodiments of this application, the first step of calculating a consistent hash value based on the target table and the source table includes: obtaining the maximum and minimum values ​​of the primary key IDs in the target table to obtain the range of primary key IDs in the target table; querying the source table for primary key IDs that are not within the range of the maximum and minimum values ​​based on the range of primary key IDs in the target table to obtain records to be marked, and marking the records to be marked as "new" (it should be noted that these new markings are also needed as a synchronization basis when performing the synchronization operation in S150); querying the source table for primary key IDs that are within the range of the maximum and minimum values ​​based on the range of primary key IDs in the target table to obtain records to be partitioned; and calculating a consistent hash value corresponding to the source table based on the records to be partitioned. Some embodiments of this application initially filter out new data by the range of primary key IDs, which can further reduce the number of subsequent partition data comparisons and improve the speed of data processing.

[0093] For example, in some embodiments of this application, the first step of calculating a consistent hash value based on the target table and the source table includes: calculating an MD5 hash value based on the primary key ID corresponding to the record to be partitioned in the target table or the source table, or calculating an MD5 hash value based on the composite primary key corresponding to the record to be partitioned in the target table or the source table; the step of partitioning the target table and the source table based on the consistent hash value includes: partitioning the records to be partitioned in the target table and the source table based on the MD5 hash value. Some embodiments of this application improve the accuracy of the partitioning results by calculating MD5 hash values ​​for partitioning.

[0094] The second step is to partition the target table and the source table based on the consistent hash value.

[0095] For example, in some embodiments of this application, the second step typically includes:

[0096] Step 1: Determine partition parameters, i.e. initialize partition system parameters. Examples of corresponding operations include:

[0097] 1. Set the number of physical partitions `PHYSICAL_PARTITIONS` (e.g., 1024).

[0098] 2. Set the number of virtual nodes per physical partition `VIRTUAL_NODES_PER_PARTITION` (e.g., 200).

[0099] 3. Calculate the total number of virtual nodes: `TOTAL_VIRTUAL_NODES = PHYSICAL_PARTITIONS *VIRTUAL_NODES_PER_PARTITION`

[0100] 4. Define the hash ring space size `RING_SIZE` (recommended `2^64` or `2^32`, use 64-bit unless otherwise specified).

[0101] Step 2: Construct a virtual node ring, which is to create a mapping ring from virtual nodes to physical partitions. Examples of the corresponding operations include:

[0102] 1. Generate `VIRTUAL_NODES_PER_PARTITION` virtual nodes for each physical partition.

[0103] 2. For each virtual node:

[0104] - Generate a unique identifier: `node_id = f"part{physical partition number}_vnode{virtual node serial number}"`

[0105] - Calculate the hash value of the identifier using a high-performance hash algorithm (such as xxHash3): `hash_val = xxhash.xxh3_64(node_id).intdigest()`

[0106] - Map the hash value to a position on the ring: `ring_position = hash_val % RING_SIZE`

[0107] - Stored as a tuple: `(ring_position, physical partition number)`

[0108] 3. Sort all virtual nodes according to `ring_position` to form an ordered ring.

[0109] Step 3: Data primary key standardization, for example, handling the heterogeneity of government data. This operation includes, for example, the following:

[0110] 1. Receive the original primary key (primary key ID or composite primary key)

[0111] 2. Standardization process:

[0112] Single-column primary key (i.e., primary key ID): Convert to string

[0113] Standard processing for composite primary keys includes: sorting fields (in a fixed order); removing leading and trailing spaces from each field; converting all fields to uppercase; handling special business rules (such as mapping administrative division changes); and concatenating the keys using a delimiter: `std_key = "|".join(field1, field2, ...)`

[0114] 3. Output the normalized string.

[0115] Step 4: Calculate the primary key hash value, that is, map the primary key to the hash ring space. Examples of the corresponding operations include:

[0116] 1. Apply a hash function to the normalized primary key: `key_hash = xxhash.xxh3_64(std_key).intdigest() % RING_SIZE`

[0117] 2. Output a 64-bit integer hash value.

[0118] Step 5: Locate the virtual node, that is, find the nearest virtual node on the ring. Examples of corresponding operations include:

[0119] 1. In the sorted virtual node ring, use binary search to find the first node whose `ring_position >= key_hash`.

[0120] 2. If not found, select the first node of the loop (for loop structure processing).

[0121] Step 6: Determine the physical partition. The purpose is to return the physical partition to which the data belongs. This operation includes, for example:

[0122] 1. Obtain the physical partition number from the located virtual node: `physical_part_id = virtual_node[1]`

[0123] 2. Returns the physical partition ID (0 ~ `PHYSICAL_PARTITIONS-1`).

[0124] Step 7: Data storage / processing, the purpose of which is to route data to the corresponding partition. Examples of corresponding operations include:

[0125] 1. Select storage location or processing node based on physical partition ID.

[0126] 2. Perform subsequent operations (such as data synchronization, calculation, etc.).

[0127] Some embodiments of this application provide a method for partitioning a source table and a target table based on a consistent hash value, thereby increasing the probability that similar data will be assigned to the same partition.

[0128] As an example, some embodiments of this application provide a data synchronization method including:

[0129] Step (1): Obtain the maximum value (maxId) and minimum value (minId) of the target table id.

[0130] The purpose of step (1) is to obtain the range (minimum and maximum values) of the primary key ID in the target table when using a single-column primary key (i.e., primary key ID). This is to narrow down the range of subsequent data comparisons.

[0131] It is easy to understand that this step is only performed when a single-column primary key is used. If a composite primary key is used, this step, as well as the range filtering in subsequent steps (2) and (3), are skipped.

[0132] Step (2): Query the source table using maxId and minId, and mark data outside the range as "new".

[0133] The purpose of performing this step (2) is to query the source table for records whose primary key IDs are not within the target table's ID range (i.e., less than minId or greater than maxId) based on the ID range obtained in the previous step. These records do not exist in the target table and are therefore marked as "new".

[0134] It is understandable that step (2) utilizes the range of IDs to quickly identify some newly added data, avoiding the need to process this data in subsequent steps, thereby reducing the amount of data that needs to be compared. Similarly, this only applies to single-column primary keys.

[0135] Step (3): Read the data within the ID range from the source table and the target table by partition (no range condition if a composite primary key is used).

[0136] The purpose of step (3) is to read data belonging to each partition (divided by a consistent hashing algorithm) from both the source and target tables. If a single-column primary key is used, the data read must also be within the range of minId and maxId (because step (2) has already processed data outside the range). If a composite primary key is used, there is no range condition, and data is read directly by partition.

[0137] It is easy to understand that step (3) is the beginning of partitioning, which divides the data into partitions so that the amount of data in each partition is small, making it easier to process in memory. At the same time, partitioning allows multiple partitions to be processed in parallel, improving efficiency.

[0138] Step (4): Construct an in-memory hash table for the target table data.

[0139] The goal of step (4) is to load the target table data of the current partition into a hash table (dictionary) in memory, where the key is the primary key (ID or composite primary key) of the record and the value is the entire record.

[0140] It's easy to understand that hash tables are built to quickly look up records in subsequent steps. The lookup time complexity of a hash table is O(1), which can greatly improve the comparison speed.

[0141] Steps (5) and (6): Traverse the source table data and perform comparisons.

[0142] The purpose of steps (5) and (6) is to traverse the source table data of the current partition, for each record:

[0143] If its primary key (or composite primary key) does not exist in the hash table of the target table, it is marked as "added".

[0144] If it exists, retrieve the corresponding target table record from the hash table and compare the field values ​​of the two records (you can compare some fields, or all fields except the primary key):

[0145] If the field values ​​are the same, mark them as "duplicate" (i.e., no update is needed).

[0146] If the field values ​​are different, mark it as "updated".

[0147] Then, delete the record from the hash table (meaning that the record exists in the source table, so it should not be deleted from the target table).

[0148] It is easy to understand that by traversing and comparing the data in the source table, the embodiments of this application can identify the records that need to be added and updated. At the same time, matching records are deleted from the hash table, so that the records remaining in the hash table are those that exist in the target table but not in the source table (i.e., the records that need to be deleted).

[0149] Step (7): Mark the remaining records in the hash table as "delete".

[0150] The purpose of step (7) is to mark the remaining records in the hash table as "delete" after the source table data has been traversed.

[0151] It's easy to understand that this step identifies the records that need to be deleted, ensuring that there is no redundant data in the target table.

[0152] Step (8): Perform synchronization operation on the current partition.

[0153] The purpose of step (8) is to perform the corresponding SQL operations (INSERT, UPDATE, DELETE) on the target database based on the previously marked results (add, update, delete) and mark the partition as processed.

[0154] It's easy to understand that performing operations by partition can break down large transactions into multiple smaller transactions, reducing database lock contention and transaction log size, improving execution efficiency, and reducing the impact on the database.

[0155] Step (9): Process all partitions in a loop

[0156] Repeat steps (3) to (8) until all partitions have been processed.

[0157] It is easy to understand that the data synchronization method in this application embodiment uses partitioning. The algorithm can process each partition one by one, or even process multiple partitions in parallel, thereby efficiently completing the entire data synchronization process.

[0158] Compared with related technologies, the technical advantages of the data synchronization method provided in this application include: 1. Quickly filtering new data using ID ranges (single-column primary keys only): reducing the amount of data for subsequent comparisons. 2. Consistent hash partitioning: dividing large datasets into small partitions, facilitating memory processing and supporting parallelism. 3. Memory hash table accelerating lookups: quickly locating target table records. 4. Clever way of marking deletion: by deleting matching records from the hash table, the remaining records are the records to be deleted. 5. Partition commit: each partition commits transactions independently, reducing database pressure. This design enables the algorithm to efficiently handle large amounts of data, adapting to scenarios with single-column primary keys and composite primary keys, and without requiring intrusive modifications to the source database (such as creating triggers or enabling logging).

[0159] The following is combined Figure 3 The embodiments illustrate data synchronization methods of some embodiments of this application.

[0160] First step, let's begin.

[0161] Phase 1: Initialization and Preprocessing

[0162] To obtain the target table ID range: Query the target table in the target database to find the maximum (maxId) and minimum (minId) value of the primary key ID (usually an auto-incrementing ID). That is, the corresponding... Figure 3 The query targets the maximum and minimum values ​​of the ID in the target table, maxId and minId.

[0163] Quickly identify newly added data (optional optimization path): Based on the maxId and minId obtained in the previous step, query the source table in the source database. Corresponding to... Figure 3 The maximum value maxId and the minimum value minId are used as query conditions to retrieve data from the source table.

[0164] Mark all records whose primary key IDs are not within the range [minId, maxId] as "new" data. Figure 3 Add a flag to the data design for data outside the ranges of maxId and minId. This is a quick pre-screening process designed to narrow down the data range for subsequent fine-grained comparisons.

[0165] Partitioned data loading: Data is read from the source and target tables by partition, based on the consistent hash value (partition field) pre-calculated for each data record.

[0166] correspond Figure 3 The consistent hash of the source table ID (i.e., primary key) / composite primary key is calculated. In embodiments of this application, the data table is divided into several partitions on an even scale using consistent hashing to narrow the scope of data comparison. Then, execution... Figure 3It retrieves data from the same partition (with the same node value) of the source and target tables.

[0167] If the optimization from the previous step was used, the query here will include the condition `ID BETWEEN minId AND maxId`; if not (for example, when dealing with a composite primary key), the full data will be read directly by partition.

[0168] Phase 2: Partition Loop Processing. In this phase, the following steps (steps 4-8) will be executed sequentially for each partition until all partitions have been processed.

[0169] Constructing an in-memory hash table: Loads all records from the target table in the current partition into memory. Figure 3 The composite primary key of the target table is used as the rowkey and stored in the hash table.

[0170] Use the primary key (or composite primary key) of each record as the key and the entire record data as the value to construct an in-memory hash table (such as a HashMap).

[0171] Traverse and compare source table records: Traverse the source table records of the current partition one by one.

[0172] For each record in the source table, a query is performed in the in-memory hash table constructed in step 4 based on its primary key (corresponding to...). Figure 3 (Does the source table rowkey exist in the hash table?)

[0173] Case A: The primary key does not exist in the hash table.

[0174] This indicates that the target table does not contain this record, so mark the source table record as "new".

[0175] Scenario B: The primary key exists in a hash table.

[0176] This indicates that the record exists in the target table; retrieve the corresponding record from the target table using the hash table.

[0177] Detailed comparison of the values ​​of each field in the two records (configurable to compare all fields or some key fields):

[0178] Field values ​​are exactly the same: Mark the current source table record as "duplicate" (i.e., no change is needed).

[0179] The field values ​​are different: mark the current source table record as "updated".

[0180] Operation: Regardless of whether it is marked as "duplicate" or "update", remove (Delete) the target table record that was just retrieved from the memory hash table.

[0181] This process is executed Figure 3 If the comparison results are identical (i.e., both the record key and field value are the same), a "same" flag is set; otherwise, an "update" flag is added. Records corresponding to the "same" and "update" flags are then removed from the hash table. The next record from the source table is retrieved. Once it's confirmed that all records for the current partition have been processed, execution proceeds. Figure 3 The remaining records in the hash are marked for deletion, and then the add, update, and delete operations are performed based on the marks.

[0182] Identify deleted data: After all source table records in the current partition have been traversed and compared, check the in-memory hash table.

[0183] At this point, all the remaining records in the hash table represent records that exist in the target table but whose corresponding primary keys were not found in the source table.

[0184] Mark these remaining target table records as "delete".

[0185] Perform synchronization operations: Based on the "Insert", "Update", and "Delete" markers generated within this partition, execute the corresponding SQL operations (INSERT, UPDATE, DELETE) on the target database to ensure that the target table is consistent with the source table.

[0186] Mark this partition as having finished processing data.

[0187] Loop check: Check if all partitions have been processed. If not, jump back to step 3 and process the next partition; if yes, the entire synchronization process ends.

[0188] It is easy to understand that the embodiments of this application provide a partitioning strategy that breaks down large datasets into smaller tasks through consistent hashing, reducing complexity. The embodiments of this application provide an in-memory computing strategy that utilizes an in-memory hash table for fast comparison, avoiding time-consuming database sorting and full table scans. The embodiments of this application also employ a reverse deletion derivation: by dynamically removing matched records from the hash table, the remaining records are ultimately defined as "deleted." This step is ingenious, accurately identifying the data to be deleted without secondary traversal or complex queries. The entire process of the embodiments of this application only requires SELECT permissions on the source database and DML permissions on the target database, without the need for triggers, logs, or advanced permissions, achieving low-privilege, non-intrusive synchronization.

[0189] Please refer to Figure 4 , Figure 4 This application illustrates a data synchronization apparatus according to an embodiment of the present application. It should be understood that this apparatus is similar to the one described above. Figure 2 or Figure 3Corresponding to the method embodiments, it is capable of executing the various steps involved in the above method embodiments. The specific functions of the device can be found in the description above; to avoid repetition, detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in a memory or embedded in the device's operating system in the form of software or firmware. The data synchronization device includes:

[0190] The target table data reading module 210 is configured to read data from the i-th partition of the target table, where i is an integer greater than or equal to 1 and less than the total number of partitions, and the i-th partition is obtained by partitioning the target table using a consistent hash function.

[0191] Hash table construction module 220 is configured to load records of the i-th partition read from the target table into a hash table constructed in memory, wherein the key of the hash table is used to store record keys, the record keys including primary keys or composite primary keys, and the value of the hash table is used to store field values.

[0192] The hash table record deletion module 230 is configured to traverse all records of the source table and the corresponding partition of the i-th partition, and compare them one by one with the records stored in the hash table, and dynamically remove the records that match each traversed record from the hash table. The matching records include: record key matching, or record key and field value matching respectively.

[0193] The deletion marker module 240 is configured to mark the undeleted record as "deleted" if it is confirmed that there is an undeleted record in the hash table.

[0194] Synchronization module 250 is configured to perform a synchronization operation between the data of the i-th partition and the data of the corresponding partition of the source table, based at least on all deletion markers.

[0195] The loop execution judgment module 260 is configured to terminate the current loop upon confirming that all partitions have been processed.

[0196] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0197] Some embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, can implement the data synchronization method as described in any of the above embodiments.

[0198] like Figure 5As shown, some embodiments of this application provide an electronic device 400, which includes a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420. When the processor 420 reads and executes the computer program via a bus 430, it can implement the data synchronization method as described in any of the above embodiments.

[0199] Processor 420 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 420 may be a microprocessor.

[0200] Memory 410 can be used to store instructions executed by processor 420 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 420 of the embodiments of this disclosure can be used to execute the instructions in memory 410 to implement… Figure 2 or Figure 3 The method shown. Memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0201] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0202] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0203] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0205] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0206] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method of data synchronization, characterized by, The method comprises: reading data of an i-th partition from a target table, wherein i is an integer greater than or equal to 1 and less than a total number of partitions, and the i-th partition is obtained by partitioning the target table by a consistent hash function; loading records of the i-th partition read from the target table into a hash table constructed in memory, wherein keys of the hash table are used to store record keys, and the record keys include primary keys or joint primary keys, and values of the hash table are used to store field values; traversing all records of a corresponding partition of the i-th partition of a source table, and comparing each of the traversed records with records stored in the hash table, and dynamically removing records matching each of the traversed records from the hash table, wherein the matching records include: record key matching, or both record key and field value matching; confirming that there are records not deleted in the hash table, and marking the records not deleted as "deleted"; performing a synchronization operation between data of the i-th partition and data of a corresponding partition of the source table according to at least all deletion marks; repeating the above steps until all partitions are processed.

2. The method of claim 1, wherein, The traversing all records of the corresponding partition of the i-th partition of the source table and the comparing each of the traversed records with records stored in the hash table and the dynamically removing records matching each of the traversed records from the hash table comprises: for a j-th traversed record, wherein j is an integer greater than or equal to 1: confirming that a record key of the j-th record is the same as a record key of a k-th record of the hash table, wherein k is an integer greater than or equal to 1; taking out the k-th record from the hash table, comparing field values of the k-th record and the j-th record, and if it is confirmed that the field values are the same, marking the j-th record as "repeated", and if it is confirmed that the field values are different, marking the j-th record as "updated", and deleting the k-th record from the hash table.

3. The method of claim 1, wherein, The traversing all records of the corresponding partition of the i-th partition of the source table and the comparing each of the traversed records with records stored in the hash table and the dynamically removing records matching each of the traversed records from the hash table comprises: traversing all records of the corresponding partition of the i-th partition of the source table, and for a j-th traversed record: confirming that a record key of the j-th record is the same as a record key of a k-th record of the hash table, wherein j and k are both integers greater than or equal to 1; taking out the k-th record from the hash table, comparing field values of the k-th record and the j-th record, and if it is confirmed that the field values are the same, marking the j-th record as "repeated", and if it is confirmed that the field values are different, marking the j-th record as "updated", and deleting the k-th record from the hash table; repeating the above comparison process for each of the traversed records of the corresponding partition; The confirming that there are records not deleted in the hash table, and marking the records not deleted as "deleted" comprises: confirming that all records left in the hash table are marked as "deleted" after traversing all records of the corresponding partition of the i-th partition of the source table.

4. The method of claim 1, wherein, The method further comprises: traversing all records of the source table and the i-th partition corresponding to the i-th partition of the target table, for the m-th record traversed: if the record key of the m-th record does not exist in the hash table, marking the m-th record as "newly added", wherein m is an integer greater than or equal to 1; performing a synchronization operation between the data of the i-th partition and the data of the partition of the source table corresponding to the i-th partition according to at least all the deletion marks, including: performing a synchronization operation between the data of the i-th partition and the data of the partition of the source table corresponding to the i-th partition according to at least all the deletion marks and all the newly added marks.

5. The method of any one of claims 1-4, wherein, The method further comprises: calculating a consistent hash value according to the target table and the source table; partitioning the target table and the source table according to the consistent hash value.

6. The method of claim 5, wherein, The calculation of the consistent hash value according to the target table and the source table comprises: obtaining the maximum value and the minimum value of the primary key ID in the target table to obtain a target table primary key ID range; querying the primary key ID in the source table that is not within the maximum value and the minimum value range according to the target table primary key ID range to obtain a to-be-labeled record, and marking the to-be-labeled record as "newly added"; querying the primary key ID in the source table that is within the maximum value and the minimum value range according to the target table primary key ID range to obtain a to-be-partitioned record; calculating a consistent hash value corresponding to the source table according to the to-be-partitioned record.

7. The method of claim 6, wherein the calculation of the consistent hash value according to the target table and the source table comprises: calculating an MD5 hash value according to the primary key ID corresponding to the to-be-partitioned record in the target table or the source table, or calculating an MD5 hash value according to the joint primary key corresponding to the to-be-partitioned record in the target table or the source table; the partitioning of the target table and the source table according to the consistent hash value comprises: partitioning the to-be-partitioned record in the target table and the source table according to the MD5 hash value.

8. An apparatus for data synchronization, the apparatus comprising: The device comprises: a target table data reading module configured to read the data of the i-th partition from the target table, wherein i is an integer greater than or equal to 1 and less than the total number of partitions, and the i-th partition is obtained by partitioning the target table by a consistent hash function; a hash table construction module configured to load the records of the i-th partition read from the target table into a hash table constructed in the memory, wherein the key of the hash table is used to store a record key, and the value of the hash table is used to store a field value; a hash table record deletion module configured to traverse all records of the source table and the i-th partition corresponding to the i-th partition of the target table, and compare each traversed record with the records stored in the hash table one by one, and dynamically remove the matching records from the hash table, wherein the matching records include: record key matching, or both record key and field value matching; a deletion mark record module configured to mark the record that has not been deleted in the hash table as "deleted" if the record exists in the hash table; The synchronization module is configured to perform a synchronization operation between the data of the ith partition and the data of the corresponding partition of the source table according to at least all the deletion marks. The loop execution determination module is configured to terminate the loop execution of the partition synchronization when all the partitions are confirmed to be processed.

9. A computer program product, characterised in that, Computer program instructions are included, which, when read and executed by a processor, can implement the method of any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the computer program to implement the method of any one of claims 1-7.

Images