Data synchronization method and device based on hash signature and sharding parallelism, equipment and medium

Abstract

The application provides a data synchronization method and device based on hash signature and parallel fragmentation, equipment and medium, comprising: generating a unique batch number corresponding to a data synchronization task, and obtaining source data records to be synchronized; for each source data record, a hash signature is generated according to the content of the business field to be compared except the primary key; the source data record, the hash signature and the batch number are associated and stored in a temporary table; when data comparison is triggered, the records associated with the batch number in the temporary table are read, the read records and the records in the target business table are divided into multiple data fragments according to the primary key, and a set comparison operation is performed on each data fragment in parallel; for the inserted record set, the updated record set and the deleted record set, database insertion, update and deletion operations are performed, so as to solve the problems of low comparison performance under large data volume, excessive invalid database operations under low change rate and difficult guarantee of business table high availability in the synchronization process.

Description

Technical Field

[0001] This invention relates to the field of data processing and data synchronization technology, and in particular to a data synchronization method, apparatus, device and medium based on hash signature and data sharding in parallel. Background Technology

[0002] In the practice of distributed software architecture and heterogeneous information system integration, data synchronization is a fundamental and critical requirement. Typical application scenarios include: data platforms extracting data from multiple business systems, synchronizing customer information between customer relationship management systems and enterprise resource planning systems, and the extraction, transformation, and loading processes of data warehouses. As enterprises become more information-driven, the amount of data synchronized in a single instance has increased from thousands to hundreds of thousands or even millions of records. Simultaneously, business systems have increasingly stringent requirements for high availability, demanding that the data synchronization process not lead to prolonged locking of business tables or service instability. In business scenarios such as daily incremental synchronization, the data change rate is typically less than 5%, meaning that over 95% of records have not actually changed.

[0003] Currently, while mature data synchronization tools exist in the industry, they often require the deployment of independent service clusters, which are prohibitively expensive in terms of hardware and maintenance costs for small and medium-sized systems. Therefore, many development teams still employ a self-developed solution based on "temporary tables and program loop comparisons." This solution first stores the interface data in a temporary table, then uses application code to iterate through each record in the temporary table, querying the business table for each record and comparing each field. Based on the comparison results, it determines whether to perform insert, update, or delete operations. However, this solution has significant performance bottlenecks: when the amount of data synchronized in a single instance reaches hundreds of thousands or more, the comparison time complexity is O(n), requiring the program to perform tens of thousands or even millions of comparison operations, taking several minutes to tens of minutes; in low-change-rate scenarios, a large number of unchanged records are still queried and compared one by one, resulting in serious waste of database and CPU resources; simultaneously, long transactions and a large number of query lock contentions severely affect the normal reading and writing of the business table, failing to guarantee high availability.

[0004] Therefore, there is an urgent need for a data synchronization method based on hash signature and sharding in parallel to solve the technical problems of low comparison performance under large data volume, excessive invalid database operations under low change rate, and difficulty in ensuring high availability of business tables during the synchronization process. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides a data synchronization method, apparatus, device and medium based on hash signature and sharding parallelism, so as to solve the technical problems in related technologies such as low comparison performance under large data volume, excessive invalid database operations under low change rate, and difficulty in ensuring high availability of business tables during the synchronization process.

[0006] This specification provides one or more embodiments of a data synchronization method based on hash signature and sharding parallelism, including the following steps: Generate a unique batch number corresponding to a data synchronization task and obtain the source data record to be synchronized; For each of the aforementioned source data records, a hash signature is generated based on the content of the business fields to be compared, excluding the primary key. The source data record, the hash signature, and the batch number are associated and stored in a temporary table, while retaining existing data associated with other batch numbers in the temporary table during storage; When data comparison is triggered, all records associated with the batch number in the temporary table are read. Based on the primary key, the read records and the records in the target business table are divided into multiple data shards. A set comparison operation is performed on each data shard in parallel. The set comparison operation includes: using a first set operation of the database to determine the record to be inserted that exists in the temporary table but does not exist in the business table; using a second set operation of the database, combined with the comparison of the hash signature, to determine the record to be updated that exists in both the temporary table and the business table but has inconsistent hash signatures; and using a third set operation of the database to determine the set of records to be deleted that exists in the business table but is to be deleted in the temporary table. Perform the corresponding database insert, update, and delete operations on the record set to be inserted, the record set to be updated, and the record set to be deleted.

[0007] Preferably, the step of dividing the read records and the records in the target business table into multiple data shards based on the primary key specifically includes the following steps: When storing the source data record in the temporary table, a shard number is calculated for the source data record based on the consistent hash value of the primary key of the source data record and then stored in the temporary table. The business table also stores the corresponding shard number field; When data comparison is triggered, the data of the corresponding batch in the temporary table and the data of the target business table are divided into the corresponding shards according to the shard number.

[0008] Preferably, before calculating and generating the hash signature, the method further includes the following steps: An existence query is performed on the primary key of the source data record using a distributed Bloom filter based on a remote dictionary service; If the query result indicates that the record definitely does not exist, the corresponding record will be marked as a fast insertion candidate, and the corresponding primary key will be stored in the fast insertion candidate set of the remote dictionary service. When performing set comparison operations in parallel on each of the data shards, the primary key is preferentially obtained from the fast insertion candidate set for processing.

[0009] Preferably, performing the corresponding database insert, update, and delete operations on the record set to be inserted, the record set to be updated, and the record set to be deleted specifically includes the following steps: The primary keys of the record set to be inserted, the record set to be updated, and the record set to be deleted are respectively pushed into the operation queue based on the remote dictionary service; The rate-limited consumer pulls the primary key from the operation queue and obtains the operation token according to the distributed token bucket algorithm based on the remote dictionary service; Only after successfully acquiring the token will the rate-limited consumer perform batch database insert, update, and delete operations on the corresponding primary key.

[0010] Preferably, the method further includes the following steps: Collect the runtime load metrics of the target business table and write the runtime load metrics into the remote dictionary service; The distributed token bucket algorithm dynamically adjusts the token generation rate based on the load metrics read from the remote dictionary service.

[0011] Preferably, before reading all records associated with the batch number in the temporary table, the method further includes the following steps: An attempt is made to acquire the processing lock for the batch number by executing a distributed lock setting command with an expiration time to a remote dictionary service; If the lock is successfully acquired, proceed to the next step; otherwise, give up or wait.

[0012] Preferably, the method further includes the following steps: When performing the insert, update, and delete operations, the operations are subdivided into multiple independent database sub-transactions; After each sub-transaction is successfully committed, the primary key of the corresponding sub-transaction is recorded in the processed set of the remote dictionary service. When the synchronization process is restarted after being interrupted by an exception, the completed primary keys are skipped by querying the processed set, and subsequent operations are performed only on the remaining unprocessed primary keys.

[0013] This specification provides one or more embodiments of a data synchronization device based on hash signature and sharding parallelism, including: The batch number generation and data acquisition module is used to generate a unique batch number corresponding to a data synchronization task and to acquire the source data record to be synchronized. The hash signature calculation module is used to calculate and generate a hash signature for each of the source data records based on the content of the business fields to be compared, excluding the primary key. The temporary table storage module is used to store the source data record, the hash signature, and the batch number in a temporary table, and retain the existing data in the temporary table associated with other batch numbers during storage; The data sharding module is used to read all records associated with the batch number in the temporary table when a data comparison is triggered, divide the read records and the records in the target business table into multiple data shards according to the primary key, and perform a set comparison operation on each data shard in parallel. The set comparison operation includes: using a first set operation of the database to determine the record to be inserted that exists in the temporary table but does not exist in the business table; using a second set operation of the database, combined with the comparison of the hash signature, to determine the set of records to be updated that exists in both the temporary table and the business table but have inconsistent hash signatures; and using a third set operation of the database to determine the set of records to be deleted that exists in the business table but does not exist in the temporary table. The batch operation module is used to perform corresponding database insert, update, and delete operations on the record set to be inserted, the record set to be updated, and the record set to be deleted.

[0014] This specification provides one or more embodiments of a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the data synchronization method based on hash signature and sharding parallelism as described above.

[0015] This specification provides one or more embodiments of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data synchronization method based on hash signature and sharding parallelism described above.

[0016] This disclosure provides a data synchronization method, apparatus, device, and medium based on hash signature and sharding parallel processing. Its advantages lie in generating a unique batch number corresponding to each synchronization task and obtaining the source data records to be synchronized, assigning an independent identifier to each synchronization, accurately defining the range of source data to be compared, avoiding confusion with other tasks, and supporting subsequent batch isolation processing. For each source data record, a hash signature is generated based on the business fields to be compared (excluding the primary key), compressing the content of multiple fields into a fixed-length digest. This allows subsequent change detection to only compare the short signature, significantly reducing the overhead of comparing fields one by one, and accurately reflecting whether the business content has undergone substantial changes. The source data records, hash signatures, and batch numbers are associated and stored in a temporary table, while retaining existing data from other batch numbers, achieving batch identifier isolation. The temporary storage of this data ensures its security without affecting historical batches and supports multiple tasks sharing the temporary storage concurrently without conflict. When data comparison is triggered, all records belonging to this batch in the temporary table are read and split into multiple data shards for parallel processing based on the primary key and the records in the target business table. The first set operation of the database directly extracts records that exist in the temporary table but not in the business table as items to be inserted. The second set operation, combined with the condition of inconsistent hash signatures, filters out records that exist in the primary key but have changed content and are to be updated. By using sharding parallelism and set operations, the massive comparison is decomposed into efficient and scalable small tasks, significantly shortening the synchronization time. For the identified records to be inserted and updated, batch database insert, update, and delete operations are performed respectively to completely and efficiently synchronize the changes from the source to the target business table, ultimately achieving data consistency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a data synchronization method based on hash signature and sharding parallelism provided for one or more embodiments of this specification; Figure 2 System architecture diagrams provided for one or more embodiments of this specification; Figure 3 A schematic diagram illustrating distributed funnel current limiting and adaptive adjustment provided for one or more embodiments of this specification; Figure 4 A first-level synchronization flowchart provided for one or more embodiments of this specification; Figure 5A second-level synchronization flowchart provided for one or more embodiments of this specification; Figure 6 A schematic diagram of consistent hashing sharding provided for one or more embodiments of this specification; Figure 7 A schematic diagram of a data synchronization device based on hash signature and fragmentation parallelism provided for one or more embodiments of this specification; Figure 8 This is a schematic diagram of the structure of a computer device provided for one or more embodiments of this specification. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this invention.

[0020] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings.

[0021] Method Implementation Examples According to embodiments of the present invention, a data synchronization method based on hash signature and sharding parallelism is provided, which is suitable for heterogeneous data synchronization scenarios with hundreds of thousands to millions of records synchronized in a single operation, a change rate of less than 5%, and high availability of business tables. Figure 1 The diagram shown illustrates the flowchart of the data synchronization method based on hash signature and sharding parallel processing provided in this embodiment. According to this embodiment, the data synchronization method based on hash signature and sharding parallel processing is divided into two independently scheduled stages: the first stage is source data acquisition and preprocessing storage, and the second stage is parallel sharding comparison and synchronization. The two stages are triggered by independent scheduled tasks and can be executed at different times. For example, the scheduled task for the first stage is set to execute every 10 minutes, and the scheduled task for the second stage is set to execute every 15 minutes. The two stages do not interfere with each other and include the following steps: S110, First-level synchronization step: The batch number manager generates a unique batch number corresponding to each data synchronization task. The batch number can be a combination of timestamp and machine number, such as "20250608143000_001", ensuring global uniqueness. At the same time, a request is sent to the source data system through the interface call module to obtain the source data records to be synchronized. The data format is a lightweight data exchange format.

[0022] S120. For each source data record obtained, the hash calculation module calculates the hash value based on the content of the business fields to be compared (excluding the primary key), concatenates them into a string according to a predetermined order, and then uses the Message Digest Algorithm Version 5 or the Secure Hash Algorithm 256-bit to calculate the hash value of the concatenated string, generating a fixed-length hexadecimal string as the hash signature. For example, concatenating "Name + Phone Number + Level" of a customer record and calculating "a3f2b109c8e7d5..." using MD5, this becomes the hash signature of the record. The predetermined order and participating fields can be predefined in the configuration file, eliminating the need to write separate comparison code for different business tables.

[0023] S130. The source data records, hash signatures, and batch numbers are associated and stored in a temporary table. Write operations use a batch insert method, for example, committing once every 1000 records. Existing data associated with other batch numbers in the temporary table is retained during storage, allowing the temporary table to simultaneously store synchronized data from multiple batches, achieving batch-level data isolation. After the first stage is completed, the batch status is recorded as "pending" in the remote dictionary service, and an expiration time is set, such as automatic expiration after 24 hours, to prevent historical data from occupying storage space for extended periods. S140. Second-level synchronization step: When data comparison is triggered by an independent scheduled task, all records associated with the batch number in the temporary table are read. Based on the primary key, the read records and records in the target business table are divided into multiple data shards. The number of shards can be pre-configured, typically set to 2 to 4 times the number of CPU cores. Each shard contains a subset of temporary table records and business table records, and all shards cover the full data. A comparison thread performs a set comparison operation on each data shard in parallel, with each thread processing one data shard. Each thread executes database-based set operations within its assigned shard. These set comparison operations include: a first database set operation (using a left join query to identify records to be inserted that exist in the temporary table but not in the business table); a second database set operation (using an inner join combined with hash signature inequality comparison to identify records to be updated that exist in both the temporary and business tables but have inconsistent hash signatures); and a third database set operation (using the business table as the main table and a left join with the current batch number data in the temporary table to identify records to be deleted that exist in the business table but are in the temporary table). Since the hash signatures are pre-calculated during the first-level synchronization process, this comparison only requires string equality comparison and does not require field-by-field expansion.

[0024] S150. For the recordsets to be inserted, updated, and deleted, perform the corresponding database insert, update, and delete operations. Specifically, for the recordsets to be inserted, perform a batch insert operation to the business table, writing the corresponding complete records from the temporary table to the business table; for the recordsets to be updated, perform a batch update operation to the business table, updating the field values ​​of the corresponding records in the business table to the latest content in the temporary table, and simultaneously updating the hash signature in the business table; for the recordsets to be deleted, perform a batch delete operation to the business table, removing records that no longer exist in the source data from the business table. For example... Figure 2 The diagram shown is a system architecture diagram provided in this embodiment. The left side represents the source data system (System B), the right side represents the target data system (System A), and the bottom represents the remote dictionary service (an independent component). The target data system internally includes the following modules: a timed scheduling module (triggers first-level and second-level synchronization respectively), an interface call module, a Bloom filter module, a hash calculation module, a temporary table (relational database), a batch number manager, a consistent hash sharder, a parallel comparison executor, a distributed funnel rate limiter, a load monitoring module, a batch operation module, and a business table (relational database). The remote dictionary service stores data structures such as Bloom filter bitmaps, batch states, distributed locks, fast insertion candidate sets, operation queues, processed sets, token buckets, and load metrics. The connection relationships are as follows: The source data system connects to the interface call module; the interface call module sequentially connects to the Bloom filter module (for bidirectional interaction with the remote dictionary service), the hash calculation module, and the temporary table (for writing); the timed scheduling module connects to the interface call module and the parallel comparison executor; the batch number manager connects to the remote dictionary service; the consistent hash sharder connects to the parallel comparison executor; the parallel comparison executor connects to the temporary table (for reading), the business table (for reading), the remote dictionary service (for reading and writing), and the distributed funnel rate limiter; the distributed funnel rate limiter connects to the batch operation module; the batch operation module connects to the business table (for writing); and the load monitoring module connects to the business table (for reading metrics) and the remote dictionary service (for writing metrics). The arrow direction indicates the data flow or control flow.

[0025] like Figure 3The diagram illustrates the distributed funnel rate limiting and adaptive adjustment provided in this embodiment. The diagram is divided into four areas: the top left represents "each shard thread," the top right represents "remote dictionary service," the bottom left represents "rate limiting consumer," and the bottom right represents "load monitoring module." Each shard thread pushes operation requests (primary keys) to the operation queue (list) of the remote dictionary service via RPUSH. The rate limiting consumer LPOPs the batch primary keys from the queue and then calls the Lua script of the remote dictionary service to obtain tokens. The remote dictionary service has a token bucket (string keys) that generates tokens at a constant rate (arrows indicate "constant rate generation"). If a token is successfully obtained, the rate limiting consumer executes a batch structured query language sub-transaction, writing the data to the business table. The load monitoring module periodically collects load metrics (connection count, lock wait time, CPU utilization, etc.) from the business table and writes them to the load metric keys of the remote dictionary service via the SET command. The token bucket in the remote dictionary service dynamically adjusts the token generation rate based on the load metrics (feedback loops are represented by dashed arrows). Data flow between the various parts in the diagram is represented by solid arrows, and control flow is represented by dashed arrows.

[0026] like Figure 4 The diagram shows the first-level synchronization flowchart provided in this embodiment. After the process begins, a unique batch number is generated first. Then, a Bloom filter for the remote dictionary service is maintained (the primary key bitmap of the business table is periodically rebuilt). Next, the interface is called to obtain the data to be synchronized (N records in total). The counter i is initialized to 0. A loop is entered to determine if i is less than N: if not, the process jumps to the record batch status step; if yes, the primary key of the i-th record is retrieved, and the Bloom filter for the remote dictionary service is queried. Based on the return result: if it returns "definitely does not exist," is_fast_insert=1 is set, and the primary key is added to the fast insert candidate set of the remote dictionary service (SADD fast_insert_candidates:{batch number} primary key); if it returns "may exist," is_fast_insert=0 is set. Then, the hash signature of the record is calculated (the business fields are concatenated and then the Message Digest Algorithm Version 5 or Secure Hash Algorithm 256-bit is calculated). Next, the record (containing all fields, hash signature, batch number, is_fast_insert, and shard number) is written to a temporary table (note: the temporary table is not cleared). Then, `i` is incremented by 1, and the loop returns to check the status. After the loop ends, the batch status is recorded as "pending" in the remote dictionary service, and a time-to-live (TTL) of 24 hours is set. The process ends.

[0027] like Figure 5The diagram shows the second-level synchronization flowchart provided in this embodiment. After the process begins, a batch number with a status of "pending" is first obtained from the remote dictionary service. Then, an attempt is made to acquire a distributed lock: executing `SET batch:{batch number}:lock "locked" NX EX 300`. If acquiring the lock fails, the process waits or gives up and ends; if acquiring the lock succeeds, the batch status is updated to "processing". Next, consistent hash sharding is performed: the number of shards P is determined, and P threads are started for parallel processing. Each thread executes set operations using Structured Query Language (SQL statements all include batch number and shard number conditions) within its own shard, obtaining the sets to be inserted, updated, and deleted, respectively. Afterwards, each thread pushes the primary key to the corresponding list in the remote dictionary service (e.g., `RPUSH batch:{batch number}:insert_queue`). Subsequently, a rate-limited consumer thread is started: the consumer batch-pushes primary keys (LPOP) from the remote dictionary service list and calls the remote dictionary service Lua script to obtain tokens from the token bucket. If acquiring the token fails, wait and retry; if successful, execute batch Structured Query Language operations (insert / update / delete), with each batch operation as a sub-transaction. Upon successful execution, add the processed primary key to the processed set in the remote dictionary service (SADD batch:{batch number}:processed). Check if all shards have been processed (by comparing the size of the processed set with the total number of records in the batch in the temporary table). If not, continue consuming the queue; if completed, update the batch status to "done," clean up the batch data in the temporary table, delete the relevant keys for that batch from the remote dictionary service (fast insert set, processed set, queue, etc.), and release the distributed lock. The process ends.

[0028] The method provided in this embodiment generates a unique batch number corresponding to a synchronization task and obtains the source data records to be synchronized, assigning an independent identifier to this synchronization, accurately defining the range of source data to be compared, avoiding confusion with other tasks, and supporting subsequent batch isolation processing. For each source data record, a hash signature is generated based on the business fields to be compared (excluding the primary key), compressing the content of multiple fields into a fixed-length digest, so that subsequent change detection only needs to compare the short signature, significantly reducing the overhead of comparing fields one by one, and accurately reflecting whether the business content has undergone substantial changes. The source data record, hash signature, and batch number are associated and stored in a temporary table, while retaining existing data of other batch numbers, achieving secure temporary storage of data under batch identifier isolation without affecting historical data. The batch processing mechanism supports multiple tasks sharing temporary storage concurrently without conflict. When data comparison is triggered, all records belonging to that batch in the temporary table are read and split into multiple data shards based on the primary key and the target business table records for parallel processing. The first set operation of the database directly extracts records that exist in the temporary table but not in the business table as items to be inserted. The second set operation, combined with the condition of inconsistent hash signatures, filters out records that exist in the primary key but have changed content and are to be updated. By using sharding parallelism and set operations, the massive comparison is decomposed into efficient and scalable small tasks, significantly shortening the synchronization time. For the identified records to be inserted and updated, batch database insert, update and delete operations are performed respectively to completely and efficiently synchronize the changes from the source to the target business table, ultimately achieving data consistency.

[0029] In one embodiment, the read records and records in the target business table are divided into multiple data shards based on the primary key, specifically including the following steps: When storing source data records in a temporary table, the consistent hash sharder calculates the shard number for each source data record by taking the consistent hash value of its primary key and modulo the total number of shards. For example, calculating the consistent hash value of the primary key "CUST001" yields an integer hash value, which is then modulo the total number of shards (32) to obtain shard number 5. This shard number, along with other fields, is then stored in the temporary table.

[0030] The business table also stores the corresponding shard number field, which is maintained synchronously when data is written.

[0031] When data comparison is triggered, the sharding module no longer dynamically calculates hash values, but directly uses the pre-stored shard number and uses the equality condition "WHERE shard number = current shard number" to divide the data of the corresponding batch in the temporary table and the data of the target business table into the corresponding shards.

[0032] like Figure 6The diagram illustrates the consistent hashing sharding provided in this embodiment. The upper part of the diagram is a ring-shaped hash space, ranging from 0 to 2^32-1. Four nodes are evenly distributed on the ring, labeled "Shard Node 0", "Shard Node 1", "Shard Node 2", and "Shard Node 3". Several primary keys (such as "Primary Key A", "Primary Key B", and "Primary Key C") are mapped to different positions on the ring through a hash function. The first node found by each primary key in a clockwise direction is its corresponding shard. This mapping relationship is indicated by arrows in the diagram. The lower part of the diagram shows the situation when the number of shards increases: a new "Shard Node 4" is inserted into the ring. The dashed arrows in the diagram indicate that the clockwise direction of a few primary keys changes (requiring reallocation to new nodes), while the direction of most primary keys remains unchanged.

[0033] The method provided in this embodiment adopts a strategy of pre-compiling shard numbers based on primary key consistent hashing. The sharding is completed when the source data is entered into the database, and the business tables persist the same sharding tag. This allows the comparison stage to quickly and evenly divide the temporary table and business table data into multiple parallel processing units according to the shard number, avoiding dynamic computing overhead and data skew, and ensuring high-concurrency execution and horizontal scaling capabilities of massive data comparison tasks.

[0034] In one embodiment, a pre-screening step using a Bloom filter based on a remote dictionary service is included before calculating and generating the hash signature. In practice, the system periodically reads all primary keys from the business table and uses the SETBIT command from the remote dictionary service to construct a Bloom filter bitmap. The bitmap length is 10 times the estimated number of records in the business table, and it uses three independent hash functions. The Bloom filter is stored in the remote dictionary service and shared by all synchronization nodes, eliminating the need for independent reconstruction on each node. Reconstruction is typically performed before each first-level synchronization or at fixed intervals. After reconstruction, the new bitmap is set to point to the active key "bloom:active," and the old key is delayed in deletion to ensure uninterrupted query service during the reconstruction process.

[0035] Using a distributed Bloom filter based on a remote dictionary service, the existence of the primary key of the source data record is checked in the remote dictionary service. Three hash values ​​are calculated for the primary key, and the GETBIT command is used to check if each bit is 1. If all bits are 1, "may exist" is returned; otherwise, "definitely does not exist" is returned.

[0036] If the query result indicates that the record definitely does not exist, the corresponding record will be marked as a fast insertion candidate, and the corresponding primary key will be stored in the fast insertion candidate set of the remote dictionary service using the SADD command. When performing set comparison operations on each data shard in parallel, each comparison thread can preferentially obtain the primary key from the fast insertion candidate set for processing.

[0037] The method provided in this embodiment introduces a distributed Bloom filter based on a remote dictionary service to predict the existence of primary keys before hash signature calculation. Records that do not exist in the target table are directly marked as fast insertion candidates and stored in a remote cache set. This allows these records to be extracted first during the parallel comparison stage, completely eliminating hash calculation and complex set operations, thereby significantly reducing invalid calculations and improving the efficiency of identifying and writing new data in massive data synchronization scenarios.

[0038] In one embodiment, performing corresponding database insert, update, and delete operations on the record set to be inserted, the record set to be updated, and the record set to be deleted includes the following steps: In practice, after determining the record sets to be inserted, updated, and deleted, each comparison shard does not directly perform write operations on the business tables. Instead, it pushes the primary keys of these record sets into operation queues based on remote dictionary services, according to their types. For example, the primary keys to be inserted are pushed into the "batch:batch number:insert_queue" list using the RPUSH command, the primary keys to be updated are pushed into the "batch:batch number:update_queue" list, and the primary keys to be deleted are pushed into the "batch:batch number:delete_queue" list.

[0039] A dedicated rate-limiting consumer thread retrieves primary keys in batches from the operation queue, for example, 100 records at a time using the LPOP command. Before writing to the business table, the rate-limiting consumer atomically checks and retrieves operation tokens from the token bucket using a Lua script based on a remote dictionary service. The token bucket maintains the current number of available tokens and is replenished at a constant rate. A distributed token bucket algorithm is used to obtain operation tokens.

[0040] Only after successfully acquiring a token will the rate-limited consumer perform batch database insert, update, and delete operations on the corresponding primary key. If acquiring a token fails, it will wait for a preset time before retrying.

[0041] The method provided in this embodiment pushes the primary keys of the records to be inserted and updated into the operation queue based on the remote dictionary service, and uses a distributed token bucket algorithm to control the rate-limiting consumer to obtain a token before executing batch database write operations. This achieves asynchronous peak shaving and valley filling and precise rate control, effectively avoiding the impact of sudden writes on the database, and ensuring the stable operation of the system and eventual data consistency in high-concurrency synchronous scenarios.

[0042] In one embodiment, the following steps are also included: The load monitoring module periodically collects operational load metrics for the target business tables, including the percentage of active database connections relative to the maximum number of connections, average lock wait time and 99th percentile lock wait time, master-slave replication latency, and CPU utilization of the database server. The collection cycle is configurable, for example, every 5 seconds. After collection, the operational load metrics are written to a remote dictionary service using the SET command, with a short expiration time set, such as 10 seconds.

[0043] When generating tokens, the background task of the distributed token bucket dynamically adjusts the token generation rate in real time based on the load metrics read from the remote dictionary service according to the preset strategy: when the percentage of active connections exceeds 80% or the lock wait time exceeds 100 milliseconds, the token generation rate is reduced by 20% from the current level; when all metrics return to normal and remain stable for 30 seconds, the token generation rate is gradually restored, for example, by increasing it by 10% each time, until it is restored to the maximum configured rate.

[0044] The method provided in this embodiment collects the operating load indicators of the target business table and writes them to a remote dictionary service. This enables the distributed token bucket algorithm to dynamically adjust the token generation rate based on the real-time load, achieving adaptive perception and control of the synchronous write rate in response to the database's busy level. When the load is high, the write speed is automatically slowed down to protect the business table, and when the load is low, the speed is appropriately increased to accelerate synchronization, thus achieving a dynamic optimal balance between throughput efficiency and system stability.

[0045] In one embodiment, a distributed concurrency control step based on a remote dictionary service is also performed before reading all records associated with the batch number in the temporary table: By executing a distributed lock setting command with an expiration time to the remote dictionary service, an attempt is made to acquire a processing lock for the batch number. If the lock is successfully acquired, subsequent steps continue; otherwise, the process is abandoned or waits. For example, executing "SETbatch:batch number:lock 'locked' NX EX 300" will only succeed if the key does not exist, and will automatically set an expiration time of 300 seconds. If the command executes successfully, it means that the current node has acquired the processing rights for the batch number, and the batch status is updated from "pending" to "processing," continuing with subsequent data reading, sharding, and comparison steps. If the command fails, it means that other nodes are processing the batch, and the current node abandons processing or waits for a preset time before retrying. After synchronization is complete, the corresponding lock key is deleted using the DEL command, the distributed lock is released, and the batch status is updated to "completed." If the processing node crashes abnormally, the lock key will automatically expire after the expiration time, and the batch status will remain "processing."

[0046] The method provided in this embodiment requests a batch number distributed lock with an expiration time from a remote dictionary service before reading temporary table data. Only instances that successfully acquire the lock are allowed to perform subsequent comparison and synchronization operations. This ensures that the same batch of tasks is not processed repeatedly in a multi-node deployment environment, avoiding concurrent conflicts and data corruption. At the same time, the expiration mechanism is used to prevent deadlocks, ensuring high reliability and single effective execution of synchronization tasks.

[0047] In one embodiment, the step of a breakpoint resumption mechanism is also included: When performing insert, update, and delete operations, the operation is divided into multiple independent database sub-transactions. Each sub-transaction contains operations corresponding to a preset number of primary keys, for example, every 100 primary keys constitute one sub-transaction. Each sub-transaction is committed independently and does not affect the others.

[0048] After each subtransaction is successfully committed, the primary key of the corresponding subtransaction is immediately recorded in the processed set of the remote dictionary service using the SADD command, for example, "batch:batch number:processed". The processed set records all primary keys of the current batch that have completed operations.

[0049] If a sub-transaction fails, only that sub-transaction is rolled back; successfully committed sub-transactions are unaffected. The failed primary key is recorded and pushed back into the corresponding operation queue, awaiting subsequent retry processing.

[0050] When the synchronization process restarts after being interrupted due to node crashes, network outages, or other abnormal reasons, it skips completed primary keys by querying the processed set and only performs subsequent operations on the remaining unprocessed primary keys. Specifically, it first checks the status of the corresponding batch in the remote dictionary service. If the status is "processing" and the distributed lock has expired and no longer exists, it means that the original processing node has failed, and the status can be rolled back to "pending processing," allowing the current node or another available node to regain processing rights. After gaining processing rights, it queries the processed set of the corresponding batch in the remote dictionary service to obtain the list of completed primary keys. In subsequent processing, the "NOT IN" condition is used to exclude these primary keys, and only set comparison operations and write operations are performed on the remaining unprocessed primary keys.

[0051] The method provided in this embodiment subdivides database write operations into multiple independent sub-transactions, and records the primary key processed by each sub-transaction to the processed set of a remote dictionary service after each sub-transaction is successfully committed. When synchronization is restarted due to an abnormal interruption, the set can be queried directly and the completed primary keys can be skipped, and only the remaining parts can be processed. This enables breakpoint resume, avoids performance waste and data inconsistency risks caused by repeated processing, and greatly improves the fault tolerance and recovery efficiency of the synchronization task.

[0052] Device Examples According to embodiments of the present invention, a data synchronization device based on hash signature and sharding parallelism is provided, such as... Figure 7 The diagram shown is a structural schematic of the data synchronization device based on hash signature and sharding parallel processing provided in this embodiment. The data synchronization device based on hash signature and sharding parallel processing according to this embodiment of the invention includes: The batch number generation and data acquisition module 71 is used to generate a unique batch number corresponding to a data synchronization task and acquire the source data record to be synchronized. The hash signature calculation module 72 is used to calculate and generate a hash signature for each source data record based on the content of the business fields to be compared, excluding the primary key.

[0053] The temporary table storage module 73 is used to store source data records, hash signatures and batch numbers in a temporary table, while retaining existing data in the temporary table associated with other batch numbers during storage.

[0054] The sharding module 74 is used to read all records associated with the batch number in the temporary table when a data comparison is triggered, divide the read records and the records in the target business table into multiple data shards according to the primary key, and perform a set comparison operation on each data shard in parallel. The set comparison operation includes: using a first set operation in the database to determine the record to be inserted that exists in the temporary table but does not exist in the business table; using a second set operation in the database, combined with the comparison of hash signatures, to determine the set of records to be updated that exists in both the temporary table and the business table but have inconsistent hash signatures; and using a third set operation in the database to determine the set of records to be deleted that exists in the business table but is to be deleted in the temporary table.

[0055] The batch operation module 75 is used to perform corresponding database insert, update and delete operations on the record sets to be inserted, updated and deleted.

[0056] The apparatus provided in this embodiment generates a unique batch number corresponding to a synchronization task and acquires the source data records to be synchronized through the batch number generation and data acquisition module 71. This assigns an independent identifier to this synchronization, accurately delineates the range of source data to be compared, avoids confusion with other tasks, and supports subsequent batch isolation processing. The hash signature calculation module 72 calculates and generates a hash signature for each source data record based on the business fields to be compared, excluding the primary key. This compresses the content of multiple fields into a fixed-length digest, so that subsequent change detection only needs to compare the short signature, significantly reducing the overhead of comparing fields one by one, and accurately reflecting whether the business content has undergone substantial changes. The temporary table storage module 73 associates and stores the source data records, hash signatures, and batch numbers in a temporary table, while retaining existing data of other batch numbers, achieving data security under batch identifier isolation. Temporary storage does not affect historical batches and supports multiple tasks to share temporary storage concurrently without conflict. When the sharding module 74 triggers data comparison, it reads all records belonging to the batch in the temporary table and splits them into multiple data shards for parallel processing based on the primary key and the records in the target business table. By using the first set operation of the database, it directly extracts records that exist in the temporary table but not in the business table as items to be inserted. By using the second set operation combined with the condition of inconsistent hash signatures, it filters out records that exist in the primary key but have changed content and are to be updated. By using sharding parallelism and set operations, the massive comparison is decomposed into efficient and scalable small tasks, which significantly shortens the synchronization time. The batch operation module 75 performs batch database insert, update and delete operations on the identified records to be inserted and updated, respectively, to completely and efficiently synchronize the changes from the source end to the target business table, and finally achieve data consistency.

[0057] The embodiments of the present invention are device embodiments corresponding to the above method embodiments. The specific operations of each module processing step can be understood with reference to the description of the method embodiments, and will not be repeated here.

[0058] like Figure 8 As shown, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the data synchronization method based on hash signature and sharding parallel in the above embodiments, or when the computer program is executed by a processor, it implements the data synchronization method based on hash signature and sharding parallel in the above embodiments.

[0059] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0060] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and the contents not described in detail in the specification of the present invention are known to those skilled in the art.

Claims

1. A data synchronization method based on hash signature and sharding in parallel, characterized in that, Includes the following steps: Generate a unique batch number corresponding to a data synchronization task and obtain the source data record to be synchronized; For each of the aforementioned source data records, a hash signature is generated based on the content of the business fields to be compared, excluding the primary key. The source data record, the hash signature, and the batch number are associated and stored in a temporary table, while retaining existing data associated with other batch numbers in the temporary table during storage; When data comparison is triggered, all records associated with the batch number in the temporary table are read. Based on the primary key, the read records and the records in the target business table are divided into multiple data shards. A set comparison operation is performed on each data shard in parallel. The set comparison operation includes: using a first set operation of the database to determine the set of records to be inserted that exists in the temporary table but not in the business table; using a second set operation of the database, combined with the comparison of the hash signatures, to determine the set of records to be updated that exists in both the temporary table and the business table but have inconsistent hash signatures; and using a third set operation of the database to determine the set of records to be deleted that exists in the business table but not in the temporary table. Perform the corresponding database insert, update, and delete operations on the record set to be inserted, the record set to be updated, and the record set to be deleted.

2. The data synchronization method based on hash signature and sharding parallel processing as described in claim 1, characterized in that, The step of dividing the read records and the records in the target business table into multiple data shards based on the primary key specifically includes the following steps: When storing the source data record in the temporary table, a shard number is calculated for the source data record based on the consistent hash value of the primary key of the source data record and then stored in the temporary table. The business table also stores the corresponding shard number field; When data comparison is triggered, the data of the corresponding batch in the temporary table and the data of the target business table are divided into the corresponding shards according to the shard number.

3. The data synchronization method based on hash signature and sharding parallel processing as described in claim 1, characterized in that, Before calculating and generating the hash signature, the process also includes the following steps: An existence query is performed on the primary key of the source data record using a distributed Bloom filter based on a remote dictionary service; If the query result indicates that the record definitely does not exist, the corresponding record will be marked as a fast insertion candidate, and the corresponding primary key will be stored in the fast insertion candidate set of the remote dictionary service. When performing set comparison operations in parallel on each of the data shards, the primary key is preferentially obtained from the fast insertion candidate set for processing.

4. The data synchronization method based on hash signature and sharding parallel processing as described in claim 1, characterized in that, The process of performing corresponding database insert, update, and delete operations on the record set to be inserted, the record set to be updated, and the record set to be deleted specifically includes the following steps: The primary keys of the record set to be inserted, the record set to be updated, and the record set to be deleted are respectively pushed into the operation queue based on the remote dictionary service; The rate-limited consumer pulls the primary key from the operation queue and obtains the operation token according to the distributed token bucket algorithm based on the remote dictionary service; Only after successfully acquiring the token will the rate-limited consumer perform batch database insert, update, and delete operations on the corresponding primary key.

5. The data synchronization method based on hash signature and sharding parallel processing as described in claim 4, characterized in that, It also includes the following steps: Collect the runtime load metrics of the target business table and write the runtime load metrics into the remote dictionary service; The distributed token bucket algorithm dynamically adjusts the token generation rate based on the load metrics read from the remote dictionary service.

6. The data synchronization method based on hash signature and sharding parallel processing as described in claim 1, characterized in that, Before reading all records associated with the batch number in the temporary table, the method further includes the following steps: An attempt is made to acquire the processing lock for the batch number by executing a distributed lock setting command with an expiration time to a remote dictionary service; If the lock is successfully acquired, proceed to the next step; otherwise, give up or wait.

7. The data synchronization method based on hash signature and fragmentation parallel processing as described in any one of claims 1 to 6, characterized in that, It also includes the following steps: When performing the insert, update, and delete operations, the operations are subdivided into multiple independent database sub-transactions; After each sub-transaction is successfully committed, the primary key of the corresponding sub-transaction is recorded in the processed set of the remote dictionary service. When the synchronization process is restarted after being interrupted by an exception, the completed primary keys are skipped by querying the processed set, and subsequent operations are performed only on the remaining unprocessed primary keys.

8. A data synchronization device based on hash signature and data sharding in parallel, characterized in that, include: The batch number generation and data acquisition module is used to generate a unique batch number corresponding to a data synchronization task and to acquire the source data record to be synchronized. The hash signature calculation module is used to calculate and generate a hash signature for each of the source data records based on the content of the business fields to be compared, excluding the primary key. The temporary table storage module is used to store the source data record, the hash signature, and the batch number in a temporary table, and retain the existing data in the temporary table associated with other batch numbers during storage; The sharding module is used to read all records associated with the batch number in the temporary table when a data comparison is triggered, divide the read records and the records in the target business table into multiple data shards according to the primary key, and perform a set comparison operation on each data shard in parallel. The set comparison operation includes: using a first set operation of the database to determine the record to be inserted that exists in the temporary table but does not exist in the business table; using a second set operation of the database, combined with the comparison of the hash signature, to determine the set of records to be updated that exist in both the temporary table and the business table but have inconsistent hash signatures; and using a third set operation of the database to determine the set of records to be deleted that exist in the business table but are to be deleted in the temporary table. The batch operation module is used to perform corresponding database insert, update, and delete operations on the record set to be inserted, the record set to be updated, and the record set to be deleted.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data synchronization method based on hash signature and sharding parallel as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data synchronization method based on hash signature and fragmentation as described in any one of claims 1 to 7.

Images