Database data synchronization method and device, equipment and medium

Abstract

The invention discloses a data synchronization method, device and equipment of a database and a medium, and belongs to the field of data synchronization, the method is suitable for a producer cluster of a distributed database, the producer cluster is in communication connection with a plurality of consumer clusters, and the method comprises the following steps: constructing Binlog information; sending the Binlog information to a plurality of consumer clusters in parallel, so that each consumer cluster identifies a plurality of missing serial numbers in the corresponding partition according to the serial number of the Binlog information; and sending corresponding completion data to each consumer cluster according to the completion request uploaded by each consumer cluster, so that the consumer clusters perform data synchronization by using the completion data. According to the method, the Binlog information is sent in parallel, so that the multiple consumer clusters synchronously recognize the missing data, the partition of each consumer cluster does not need to be recognized and tracked one by one, the processing time consumption is greatly shortened, and the data synchronization efficiency is remarkably improved.

Description

Technical Field

[0001] This invention relates to the technical field of copy data processing, and in particular to a database data synchronization method, apparatus, device, and medium. Background Technology

[0002] In a distributed database, nodes are deployed on different machines, and different partitions are handled by different sites within each node. Each site has independent CPU and memory for transaction processing, improving efficiency. However, in a distributed environment, this multi-replica mechanism can easily lead to data inconsistency. Data synchronization ensures data consistency across all nodes, preventing problems caused by outdated or erroneous data on some nodes. For example, consistency protocols like the Raft algorithm coordinate the order of operations between replicas through synchronization mechanisms, preventing data conflicts.

[0003] To achieve data synchronization across different nodes in a database, the commonly used data synchronization method is to periodically update the replication site corresponding to each partition, identify missing sequences starting from the replication site, and then transmit data to fill in the missing parts, thereby achieving data synchronization between different partitions.

[0004] However, the above method has obvious technical drawbacks: distributed databases usually contain thousands of data partitions, and each partition has a different basic location. Each partition needs to be identified and tracked one by one, which results in long processing time and low efficiency. Summary of the Invention

[0005] This invention provides a database data synchronization method, apparatus, device, and medium, which can solve the technical problem of low processing efficiency in existing data synchronization technologies.

[0006] A first aspect of this invention provides a database data synchronization method, applicable to a producer cluster of a distributed database, wherein the producer cluster is communicatively connected to several consumer clusters, the method comprising: Construct Binlog information, wherein the Binlog information is generated by binding the monotonically increasing DR sequence number of each partition with Binlog events; The Binlog information is sent in parallel to several consumer clusters, so that each consumer cluster can identify several missing sequence numbers in the corresponding partition based on the sequence number of the Binlog information. Based on the completion request uploaded by each consumer cluster, corresponding completion data is sent to each consumer cluster for the consumer cluster to use for data synchronization. The completion request is a request constructed by the consumer cluster based on several missing sequence numbers of the identified partition.

[0007] A second aspect of this invention provides a data synchronization method for a database, the method being applicable to a consumer cluster of a distributed database, the consumer cluster being communicatively connected to a producer cluster, the method comprising: Obtain the Binlog information sent by the producer cluster, wherein the Binlog information is generated by the producer cluster binding the monotonically increasing DR sequence number of each partition with the Binlog event; Based on the sequence number of the Binlog information, identify several missing sequence numbers within the corresponding partition; Based on the missing sequence numbers, complete data is obtained from the producer cluster, and the complete data is used to update the data so that the partition storage data is synchronized with the data stored in the producer cluster.

[0008] A third aspect of this invention provides a data synchronization device for a database. The device is applicable to a producer cluster of a distributed database, the producer cluster being communicatively connected to several consumer clusters. The device includes: The construction module is used to construct Binlog information, wherein the Binlog information is generated by binding the monotonically increasing DR sequence number of each partition with Binlog events; A parallel sending module is used to send the Binlog information to several consumer clusters in parallel, so that each consumer cluster can identify several missing sequence numbers in the corresponding partition according to the sequence number of the Binlog information. Sending data is used to send corresponding completion data to each of the consumer clusters according to the completion request uploaded by each of the consumer clusters, so that the consumer clusters can use the completion data to perform data synchronization. The completion request is a request constructed by the consumer cluster based on several missing sequence numbers of the identified partition.

[0009] A fourth aspect of this invention provides a data synchronization apparatus for a database, the apparatus being applicable to a consumer cluster of a distributed database, the consumer cluster being communicatively connected to a producer cluster, the apparatus comprising: The acquisition module is used to acquire Binlog information sent by the producer cluster, wherein the Binlog information is generated by the producer cluster binding the monotonically increasing DR sequence number of each partition with the Binlog event; The identification module is used to identify several missing sequence numbers in the corresponding partition based on the sequence number of the Binlog information; The synchronization module is used to obtain supplementary data from the producer cluster based on the missing sequence numbers, and use the supplementary data to update the data so that the partition storage data is synchronized with the data stored in the producer cluster.

[0010] Compared to existing technologies, the data synchronization method, apparatus, device, and medium for a database provided in this invention offer the following advantages: This invention can construct Binlog information and send it in parallel to several consumer clusters, enabling each consumer cluster to identify several missing sequence numbers within the corresponding partition based on the sequence number in the Binlog information. It also sends corresponding completion data to each consumer cluster based on the completion request uploaded by each cluster, allowing the consumer clusters to use the completion data for data synchronization. By sending Binlog information to multiple consumer clusters in parallel, this invention allows multiple consumer clusters to simultaneously identify their missing data, enabling data synchronization based on the identification results of each consumer cluster. This eliminates the need to identify and track each consumer cluster's partition individually, significantly reducing processing time and improving processing efficiency. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a data synchronization method for a database (producer cluster end) provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a database data synchronization method (consumer cluster end) provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a database data synchronization device (producer cluster end) provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a database data synchronization device (consumer cluster end) provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] In a distributed database, nodes are deployed on different machines, and different partitions are handled by different sites within each node. Each site has independent CPU and memory for transaction processing, improving efficiency. However, in a distributed environment, this multi-replica mechanism can easily lead to data inconsistency. Data synchronization ensures data consistency across all nodes, preventing problems caused by outdated or erroneous data on some nodes. For example, consistency protocols like the Raft algorithm coordinate the order of operations between replicas through synchronization mechanisms, preventing data conflicts.

[0014] To achieve data synchronization across all nodes of a database, one common method is to periodically update the corresponding replication site for each partition. Starting from the replication site of each partition, the system identifies the missing sequence between the replication site and the current real-time site, and then transmits data to fill in the missing sequence, thereby achieving data synchronization between different partitions.

[0015] However, the above method has the following technical problems: Since distributed databases generally contain thousands of data partitions, and each partition has a different basic location, each partition needs to be identified and tracked one by one, which is time-consuming and inefficient.

[0016] To address the aforementioned issues, the following detailed description and explanation will be provided through specific embodiments of a database data synchronization method, apparatus, device, and medium provided in this application.

[0017] To address the low processing efficiency of existing data synchronization technologies, referring to Figure 1 The diagram illustrates a flowchart of a database data synchronization method (producer cluster end) provided by an embodiment of the present invention.

[0018] In one embodiment, the data synchronization method for the database is applicable to a producer cluster of a distributed database, which is communicatively connected to several consumer clusters.

[0019] As an example, the data synchronization method for the database may include: S11. Construct Binlog information, wherein the Binlog information is generated by binding the monotonically increasing DR sequence number of each partition with the Binlog event.

[0020] In one embodiment, Binlog information can be generated in the producer cluster. This Binlog information is generated by binding the monotonically increasing DR sequence number of each partition with Binlog events. Specifically, a monotonically increasing, partition-unique sequence number (DR sequence number) can be assigned to each generated Binlog. This step considers scenarios with large-scale partitions, and the sequence number allocation uses an efficient counter mechanism to ensure order and uniqueness under high-concurrency writes. For example, in a cluster containing multiple partitions, each partition manages its own sequence number, avoiding global sequence number conflicts.

[0021] It should be noted that a distributed database has multiple partitions, each storing different types of data. For example, partition 1 stores user A's business data, and partition 2 stores user B's business data.

[0022] As an example, the construction of Binlog information may include the following sub-steps: S111. Determine the Binlog event of the target partition, wherein the target partition is an extended partition or a partition on which a write transaction is executed.

[0023] S112. Obtain the DR sequence number corresponding to the target partition according to the Binlog event, wherein the real-time DR sequence number is a sequence number range constructed by monotonically increasing the counter established by the target partition when the partition is created or initialized based on the executed events.

[0024] S113. Bind the DR sequence number with the Binlog event to obtain the Binlog information.

[0025] In one embodiment, the Binlog event of the target partition is determined, which is either an extended partition or a partition where a write transaction is performed.

[0026] When a transaction is executed or a new partition is added, a Binlog event is generated, which can be a Binlog file.

[0027] Next, the real-time sequence number corresponding to the target partition can be obtained, as well as the number of Binlog entries in the Binlog event can be determined. Combining the number of Binlog entries and the real-time sequence number, the DR sequence number corresponding to the target partition can be obtained.

[0028] Finally, the DR sequence number can be bound to the Binlog to obtain the Binlog information. The producer cluster can generate Binlog events sequentially, with each Binlog entry accompanied by a uniquely assigned DR sequence number.

[0029] This sequence number directly reflects the global monotonic order of the Binlog entry within the partition, ensuring order and uniqueness in subsequent replication and synchronization processes.

[0030] It should be noted that when each partition is created or initialized, a counter can be set for that partition, with the initial value set to 0 (or the last synchronization point + 1).

[0031] The counter state is persistently stored in a local or distributed configuration storage service (such as ZooKeeper, Etcd, etc.) for recovery.

[0032] Each partition has a counter, initially set to 0 (or the last synchronization point + 1). The initial sequence numbers for each partition are not identical. Although the initial value may start from 0 or the synchronization point + 1, the sequence number is partition-level, uniquely identified and distinguished by the partition ID. Specifically, different partitions have independent counter and sequence number spaces. Even if the initial values ​​are the same (e.g., all starting from 0), the sequence numbers will not conflict because they are always used in conjunction with the partition ID to form a unique identifier such as "partition ID - sequence number" (e.g., sequence number 100 for partition P0 corresponds to the identifier "P0-100", and sequence number 100 for partition P1 corresponds to the identifier "P1-100"). This ensures the uniqueness and distinguishability of sequence numbers in a distributed environment, eliminating the need for a global sequence number mechanism.

[0033] This invention employs a "partition-independent counter + combined identifier" mechanism. Each partition maintains an independent counter, and the sequence number allocation follows the [CurrentSeq+1, CurrentSeq+N] rule to ensure monotonically increasing within a partition. The sequence numbers of different partitions are distinguished by the combination of "partition ID - sequence number" (e.g., P0-100), avoiding cross-partition conflicts. At the same time, the persistent storage of the counter ensures orderliness after fault recovery, thereby ensuring the uniqueness and orderliness of the sequence numbers in a distributed environment.

[0034] In addition, the status of the sequence number counter is synchronously written to persistent storage to prevent the count from being lost due to node crashes or restarts.

[0035] During distributed database recovery, the persistent counter value is read to ensure that the newly generated sequence number starts from the correct position.

[0036] Alternatively, lock-free or fine-grained locking mechanisms (such as CAS atomic operations or position locks) can be used to quickly respond to multiple concurrent sequence number requests and avoid race conditions.

[0037] Supports a serial number pre-allocation mechanism: Under high load, a batch of serial numbers are pre-allocated to reduce lock contention and the number of calls, thereby improving pipeline efficiency.

[0038] For extremely high-concurrency scenarios, the serial number management module can use local caching combined with conflict detection strategies to ensure global uniqueness and increment while maximizing throughput.

[0039] During processing, the serial number generation status can be monitored in real time, and serial number allocation logs can be recorded, which facilitates later auditing and problem tracking.

[0040] When an anomaly occurs (such as an abnormal rollback of the counter or a sequence number jump), an alarm can be automatically triggered and the sequence number correction process can be started to prevent it from affecting downstream data synchronization.

[0041] After the producer cluster calculates the allocation sequence number range corresponding to each Binlog in parallel, the producer cluster can then send it to the consumer cluster in batches.

[0042] In dynamically expanding scenarios, when a new partition is added, the consumer cluster can quickly initialize an empty range set and synchronize starting from sequence number 0, ensuring seamless integration. This process smoothly transitions from log generation to batch application and range optimization, forming an efficient synchronization closed loop.

[0043] In this way, the data processing remains efficient and scalable in large-scale environments, avoiding the bottlenecks of traditional methods.

[0044] In an optional embodiment, obtaining the DR sequence number corresponding to the target partition based on the Binlog event may include the following sub-steps: S1121. Determine the corresponding Binlog entry value based on the transaction type of the Binlog event, and obtain the real-time count value of the counter of the target partition.

[0045] S1122. The initial sequence number range is determined using the Binlog line values ​​and the real-time count values.

[0046] S1123. After dynamically adjusting and updating the initial serial number range value, a DR serial number is obtained. The dynamic adjustment is to adjust the maximum value of the initial serial number range according to a preset reserved buffer or a preset allocation step size. The update verification is to update the counter of the target partition with the initial serial number range and then detect the validity of the serial number range.

[0047] When obtaining the DR sequence number at the start of a transaction, this can be done by the sequence number management module in the producer cluster.

[0048] Specifically, when a partition in the producer cluster receives a write transaction request, the transaction coordinator sends a request to the sequence number management module of that partition to apply for the sequence number range required for this write operation (usually a continuous interval of sequence numbers corresponding to one or more Binlog entries).

[0049] The serial number management module performs the following operations: Read the real-time counter value CurrentSeq of the target partition.

[0050] The corresponding Binlog entry value N is determined based on the transaction type of the Binlog event. The initial sequence number range is determined based on the Binlog entry value and the real-time count value. The initial sequence number range is [CurrentSeq+1, CurrentSeq+N].

[0051] Atomicity updates the counter to CurrentSeq+N, ensuring that the sequence number is correctly linked for the next request.

[0052] To determine the number of Binlog entries corresponding to the transaction type of a Binlog event, the transaction type can be identified, and the corresponding scope of influence can be determined based on the identified transaction type to obtain the number of Binlog entries.

[0053] In an optional embodiment, different transaction types can correspond to different Binlog entry values, as shown below: (1) Transaction type of single table operation: For a single INSERT, UPDATE, or DELETE operation, the estimated number of Binlog entries N = 1; (2) Transaction type of batch operations: For batch INSERT operations, estimate N = M based on the batch size M; for batch UPDATE / DELETE operations, estimate the number of rows affected R based on the WHERE condition, and set N = R. (3) Transaction type of multi-table transactions: For cross-table transactions, the estimated number of rows of operations in each table is accumulated, i.e., N = Σ(N_i), where N_i is the estimated number of rows of operations in the i-th table; (4) Transaction type of DDL operation: For table structure change operations (such as ALTER TABLE), N is fixed to 1.

[0054] The calculation of the initial sequence number range can be performed as follows: Based on the estimated Binlog number N and the current counter value CurrentSeq, the sequence number allocation range is determined using the following formula: Basic calculation formula: Assign a starting sequence number: StartSeq = CurrentSeq + 1; End sequence number allocation: EndSeq = CurrentSeq + N; Serial number allocation range: [StartSeq, EndSeq] = [CurrentSeq + 1, CurrentSeq + N]; Explanation of formula parameters: CurrentSeq: The value of the serial number counter for the current partition, representing the largest assigned serial number; N: The estimated number of Binlogs generated in this transaction; StartSeq: The starting serial number allocated for this transaction; EndSeq: The ending serial number allocated for this transaction.

[0055] After determining the initial serial number range value, the initial serial number range value can be dynamically adjusted. The dynamic range adjustment mechanism can improve the utilization efficiency of serial numbers. Among them, the dynamic adjustment is to adjust the maximum value of the initial serial number range according to a preset reserved buffer or a preset allocation step size.

[0056] Specifically, the adjustment of the preset reserved buffer can be as follows: In a high-concurrency environment, to avoid insufficient serial numbers, the actual allocation range is expanded to: Adjusted N value: N' = N × (1 + BufferRatio); BufferRatio is the buffer ratio, usually set to 0.1 - 0.2; Final allocation range: [CurrentSeq + 1, CurrentSeq + N'].

[0057] The adjustment of the minimum allocation unit control can be as follows: Set the minimum allocation step size MinStep. When N < MinStep, force N = MinStep to ensure the continuity and management efficiency of serial number allocation.

[0058] After completing the dynamic adjustment, update verification can be performed to obtain the DR serial number. The update verification is to detect the validity of the serial number range after updating the counter of the target partition with the initial serial number range.

[0059] Specifically, the update can be that the serial number management module can update the atomic counter. The specific update process can be as follows: (1) Atomic read - calculate - write: Use the Compare-And-Swap (CAS) operation to read CurrentSeq; Calculate the new counter value in memory: NewSeq = CurrentSeq + N; Atomically update the counter to NewSeq; (2) Update the verification mechanism: Verification before update: NewSeq > CurrentSeq and NewSeq - CurrentSeq = N; Post-update verification: Confirmed that the counter value equals the expected NewSeq; If verification fails, re-execute the read-calculate-write process; (3) Concurrency conflict handling: When multiple transactions concurrently request a sequence number, a retry mechanism is used; Retry limit: Maximum number of retries: RetryLimit (usually set to 3-5). Retry interval: An exponential backoff strategy is adopted, and the retry interval for the kth retry is BaseDelay×2^k milliseconds.

[0060] After the update is complete, the validity of the initial serial number range can be checked. Specifically, the following steps can be performed to ensure the range is valid: (1) Continuity detection: Verify that there are no gaps within the assigned sequence number range [StartSeq, EndSeq]; (2) Uniqueness check: Confirm that the range of assigned serial numbers does not overlap with the range of previously assigned serial numbers; (3) Boundary detection: Verify that EndSeq does not exceed the maximum allowed value MaxSeqLimit of the partition sequence number; (4) Persistent state synchronization: The updated counter value NewSeq is synchronously written to persistent storage to ensure the correctness of sequence number allocation during fault recovery.

[0061] S12. Send the Binlog information to several consumer clusters in parallel, so that each consumer cluster can identify several missing sequence numbers in the corresponding partition based on the sequence number of the Binlog information.

[0062] The producer cluster sends batches of Binlog data, each with a sequence number, to the consumer cluster. This transmission process supports parallelization, such as through multi-threading or partition-level flow control, to accommodate large-scale data synchronization needs and reduce network bottlenecks.

[0063] Each consumer cluster can identify several missing sequence numbers in its corresponding partition based on the sequence number of the Binlog information to determine whether there are data gaps between the producer cluster and the consumer cluster.

[0064] To meet the high-efficiency synchronization requirements of massive Binlog data in large-scale distributed database clusters, the producer cluster employs a multi-threaded parallel batch transmission mechanism to send Binlog data with sequence numbers to the consumer clusters. In one embodiment, the parallel transmission of the Binlog information to several consumer clusters may include the following sub-steps: S121. The Binlog information is allocated to the corresponding partition tasks according to the number of partitions in the database, and the thread corresponding to the partition task is matched in the preset thread pool so that each thread performs batch reading and sequence number packaging of the Binlog information of the partition task to obtain a Binlog data packet.

[0065] S122. Call the preset network transmission interface to send the Binlog data packet to several consumer clusters, and clear the cache after receiving the confirmation signal from the consumer cluster. The confirmation signal is a signal returned by the consumer cluster to the corresponding producer cluster after receiving the Binlog data packet and verifying the continuity of the sequence number of the Binlog data packet.

[0066] Specifically, based on the number of database partitions in the producer cluster, Binlog data can be partitioned into corresponding partition tasks.

[0067] For example, a cluster contains 1000 partitions, and the data is divided into 1000 independent transmission units based on the partition ID. To avoid single-threaded bottlenecks, these 1000 partition tasks can be dynamically allocated to N parallel threads according to business load and network bandwidth (N is adjusted based on hardware resources and network environment, typically set to tens to hundreds). Optionally, this invention can support parallel synchronization of more than 1000 partitions, maintaining stable efficiency when the number of partitions doubles.

[0068] The producer starts a fixed-size thread pool, with each thread responsible for the Binlog transmission tasks of several partitions.

[0069] For example, initialize a thread pool of size 50, with each thread responsible for transmitting Binlog data from an average of 20 partitions.

[0070] Each thread periodically reads newly added Binlog from the partition it is responsible for, packages them in batches according to their sequence numbers from smallest to largest, forming continuous and ordered log data blocks, thus obtaining a Binlog data packet.

[0071] Data is packaged according to a certain batch size (e.g., 1000 logs per batch) or time window (e.g., every 100ms) to balance transmission latency and throughput.

[0072] Since each Binlog entry is accompanied by a uniquely assigned DR sequence number, after the Binlog is split into multiple packets, each Binlog entry is accompanied by a uniquely assigned DR sequence number.

[0073] Next, each thread calls a unified network transmission interface (such as an RPC call or a message queue push interface) to send the packaged Binlog data stream to the corresponding receiving node in the consumer cluster.

[0074] This network interface supports asynchronous transmission and flow control mechanisms to ensure that no blocking or packet loss occurs during parallel transmission due to a sudden increase in traffic.

[0075] During transmission, each thread independently manages its own transmission status and retry mechanism to ensure reliable data delivery.

[0076] During transmission, the producer cluster has a scheduling module that dynamically monitors the running status of each thread and the network load, and adjusts the correspondence between partitions and threads in real time.

[0077] When some threads handle too many partitions or network transmission bottlenecks occur, the scheduling module can migrate some partition tasks to idle threads to achieve load balancing and improve overall transmission efficiency.

[0078] After receiving and verifying the continuity of the sequence numbers, the consumer cluster returns an acknowledgment signal to the corresponding producer thread.

[0079] The producer thread cleans up the sending cache, updates the synchronization progress, and prepares for the next batch of transmission based on the confirmation results, ensuring that data is transmitted batch by batch without omission.

[0080] S13. Send corresponding completion data to each consumer cluster according to the completion request uploaded by each consumer cluster, so that the consumer cluster can use the completion data to perform data synchronization. The completion request is a request constructed by the consumer cluster based on several missing sequence numbers of the identified partition.

[0081] Specifically, after identifying several missing sequence numbers, the consumer cluster can construct a completion request using these missing sequence numbers and send it back to the producer cluster. The producer cluster can respond to the completion request and find the corresponding data, specifically the data corresponding to each missing sequence number, and then package and integrate it into a single completion data set. Finally, the corresponding completion data can be sent to each consumer cluster, allowing the consumer clusters to update their data and ensure that their stored data is identical to that stored in the producer clusters, thereby achieving data synchronization between the consumer and producer clusters.

[0082] This invention proposes a data synchronization method based on sequence number range tracking, optimizing the large-scale data synchronization process. The core of this method is to assign a monotonically increasing sequence number to the Binlog of each data partition and maintain a scalable set of sequence number ranges on the consumer side to achieve efficient gap detection and batch requests. The data processing is natural and smooth, divided into three stages: sequence number generation, log transmission and application, and range update and synchronization optimization. The logic is clear and easy to deploy on a large scale.

[0083] This invention achieves three-layer optimization: 1. Multi-threaded parallel transmission on the producer side, dynamically allocating thread pools by partition (e.g., 50 threads for 1000 partitions), and batch packaging of logs (1000 records / batch); 2. Batch gap detection and resending on the consumer side, aggregating the missing range of multiple partitions in a single request to reduce network interaction; 3. Pre-allocation of sequence numbers and lock-free mechanism, reducing lock contention under high concurrency, improving throughput, and ultimately reducing synchronization latency by more than 30%, ensuring transmission efficiency in high-concurrency scenarios.

[0084] In this embodiment, the present invention provides a database data synchronization method, the advantages of which are: the present invention can construct Binlog information; and send Binlog information to several consumer clusters in parallel, so that each consumer cluster can identify several missing sequence numbers in the corresponding partition according to the sequence number of the Binlog information; and send corresponding completion data to each consumer cluster according to the completion request uploaded by each consumer cluster, so that the consumer clusters can use the completion data for data synchronization. By constructing Binlog information and sending it in parallel to multiple consumer clusters, the present invention allows each consumer cluster to synchronously identify its own missing data, and then send corresponding completion data according to the completion request, eliminating the need to identify and track the partitions of each consumer cluster one by one, greatly shortening the processing time and significantly improving the data synchronization efficiency.

[0085] Reference Figure 2 The diagram illustrates a flowchart of a database data synchronization method (consumer cluster end) provided by an embodiment of the present invention.

[0086] In one embodiment, the data synchronization method for the database is applicable to a consumer cluster of a distributed database, the consumer cluster being communicatively connected to a producer cluster.

[0087] As an example, the data synchronization method for the database may include: S21. Obtain the Binlog information sent by the producer cluster, wherein the Binlog information is generated by the producer cluster binding the monotonically increasing DR sequence number of each partition with the Binlog event.

[0088] In one embodiment, the consumer cluster can obtain Binlog information sent by the producer cluster in a parallel manner, wherein the Binlog information is generated by the producer cluster binding the monotonically increasing DR sequence number of each partition with Binlog events.

[0089] Specifically, the method for constructing the Binlog information can be referred to the analysis and description of the above embodiments. To avoid repetition, it will not be repeated here.

[0090] S22. Identify several missing sequence numbers in the corresponding partition based on the sequence number of the Binlog information.

[0091] After receiving the Binlog, the consumer cluster can apply the log to its local partition and update the sequence number tracker. This tracker is a mapping table where the key is the producer partition ID and the value is a set of sequence number ranges. The range set dynamically maintains continuous intervals of applied sequence numbers, such as {[1, 1000], [1200, 1500]}, which is particularly effective in large-scale scenarios. The tracker identifies several missing sequence numbers within the corresponding partition.

[0092] In an optional embodiment, identifying several missing sequence numbers within the corresponding partition based on the sequence numbers of the Binlog information may include the following sub-steps: S221. Extract the serial number to be updated and the partition ID from the Binlog information, wherein the serial number to be updated is a serial number that has completed the audit verification, and the audit verification verifies the value of the serial number, the maximum serial number of the serial number, and the range of the serial number.

[0093] S222. Obtain the real-time sequence range from the built-in tracker of the partition ID, wherein the built-in tracker is a mapping table of partition IDs and sequence number ranges of the producer cluster.

[0094] S223. Traverse all intervals of the real-time sequence range and determine the attribution relationship between the sequence number to be updated and all intervals of the real-time sequence range.

[0095] S224. Based on the attribution relationship, determine several sequence extension ranges and obtain the sequence number of each sequence extension range to obtain several missing sequence numbers.

[0096] When the consumer cluster receives Binlog events from the producer cluster, it can first extract two key fields from the Binlog header information: the producer partition ID (Partition_ID) and the sequence number (Sequence_Number) corresponding to the Binlog. During this process, the consumer cluster needs to verify the legality of the sequence number, including confirming that the sequence number is a positive integer, the sequence number is not less than the maximum sequence number applied last time for this partition minus 1 (allowing a certain range of duplicates or out-of-order reception), and whether the sequence number is within the expected range. This verification step prevents illegal or abnormal sequence numbers from entering the tracker and ensures data consistency.

[0097] Before the update, the current range set (Range_Set) corresponding to this partition can be queried in the sequence number tracker according to the extracted Partition_ID. The range set is stored in the form of an ordered list, and each element is a closed interval [Start, End], where Start ≤ End. The consumer cluster can sort these intervals in ascending order of the Start value of the range for subsequent range matching and merging operations. If there is no record of this partition in the tracker, an empty range set is initialized.

[0098] Next, all intervals in the current range set can be traversed, and the relationship between the new sequence number (S) and each interval can be compared to determine its belonging. The specific judgment logic is as follows: If S is within an existing interval [A, B] (i.e., A ≤ S ≤ B), it means that the Binlog corresponding to this sequence number has been applied. The consumer cluster does not need to update the range set, directly marks it as a duplicate application, and records the log.

[0099] If S is equal to the right endpoint B of an interval [A, B] plus 1 (i.e., S = B + 1), it means that the new sequence number is continuously connected to this interval. A range extension operation needs to be performed, and the sequence between S and B can be determined to obtain the missing sequence number.

[0100] If S falls in the gap between two adjacent intervals [A, B] and [C, D] (i.e., B < S < C), it is necessary to determine whether a new single-point interval should be created or special handling should be performed to determine the missing sequence number.

[0101] If S is less than the minimum value of all existing intervals or S is greater than the maximum value of all existing intervals, the consumer cluster needs to add a new interval or mark it as an abnormal out-of-order according to the specific situation to determine the missing sequence number.

[0102] Among them, determining whether a new single-point interval should be created or special handling should be performed can include the following steps: The first step is to calculate the gap span and classify: First, calculate the span value of the gap: Gap = C - B - 1, which is used to judge the nature of the gap and the processing strategy: If Gap = 1, it means that the gap only contains a unique serial number position (i.e., S = B + 1 = C - 1), which is the minimum gap case.

[0103] If Gap>1 and S = B + 1, it means that the new serial number is adjacent to the right boundary of the previous interval, then incrementally search for the missing serial numbers starting from S.

[0104] If Gap>1 and S = C - 1, it means that the new serial number is adjacent to the left boundary of the next interval, then decrementally search for the missing serial numbers starting from S.

[0105] If Gap>1 and B + 1 < S < C - 1, it means that the new serial number falls in the middle of the gap, then take the middle segment of B and C as the missing serial numbers.

[0106] This classification provides a basis for subsequent processing decisions, ensuring that the system can select the optimal strategy according to the gap characteristics.

[0107] The second step is to check the serial number continuity expectation: Query the synchronization configuration information of this partition to obtain the continuity expectation threshold parameter Threshold_Gap (the default value can be set to 10). This parameter represents the maximum acceptable gap span: If Gap ≤ Threshold_Gap, it means that the gap is small and the number of expected missing serial numbers is limited. The system tends to keep the gap state and wait for subsequent reissuance to fill it.

[0108] If Gap>Threshold_Gap, it means that the gap is large and there may be log segments that have not been synchronized for a long time, and a reissuance request needs to be actively triggered.

[0109] This step prevents the premature creation of too many single-point intervals, resulting in fragmentation of the range set and affecting storage efficiency.

[0110] The third step is to judge the serial number source and timeliness: The reception timestamp (Timestamp_Received) of each serial number and the generation timestamp (Timestamp_Generated) at the producer side can be recorded, and the data timeliness can be judged by comparing the time difference: Calculate the time delay: Delay = Timestamp_Received - Timestamp_Generated.

[0111] If Delay < Timeout_Threshold (e.g., 5 seconds), it indicates that the sequence number is real-time data generated recently. The missing sequence numbers in the gap may be in transit. The system selects a waiting strategy and does not create a single-point interval temporarily.

[0112] If Delay ≥ Timeout_Threshold, it indicates that there is a large delay or serious out-of-order in data transmission. The system needs to immediately create a single-point interval [S, S] and record the gap status for subsequent triggering of active retransmission.

[0113] This judgment avoids frequent creation of single-point intervals due to network jitter or short-term out-of-order, and improves the stability of the range set.

[0114] Step 4, execute the conditional judgment decision tree: Based on the calculation results of the previous three steps, the system processes according to the following decision tree: Case A: Minimum gap and real-time data; Condition: Gap = 1 and Delay < Timeout_Threshold; Processing: Create a single-point interval [S, S], and immediately try to merge adjacent intervals. Since the gap is completely filled (S = B + 1 = C - 1), the system merges it into a continuous interval [A, D] to eliminate the gap.

[0115] Example: Given intervals [100, 200] and [202, 300], when the new sequence number 201 arrives, it is merged into [100, 300].

[0116] Case B: Small gap and real-time data; Condition: Gap ≤ Threshold_Gap and Delay < Timeout_Threshold; Processing: Create a single-point interval [S, S], insert it into the range set, and keep the original two intervals [A, B] and [C, D] unchanged. The range set becomes {[A, B], [S, S], [C, D]}. The system does not immediately trigger retransmission, but marks the gap status of this partition as "under monitoring" (Status = Monitoring), waiting for subsequent sequence numbers to fill naturally.

[0117] Example: Given intervals [100, 200] and [210, 300], when the new sequence number 205 arrives (Gap = 9 < 10), create [205, 205], and wait for 206 - 209.

[0118] Case C: Large gap or delayed data; Condition: Gap > Threshold_Gap or Delay ≥ Timeout_Threshold; Processing: Create a single-point interval [S, S] and immediately trigger a gap retransmission request. The system records the missing range as two segments [B+1, S-1] and [S+1, C-1] (if they exist), adds the retransmission request to the pending queue, and sets its priority to high. At the same time, update the synchronization status of this partition to "Gap Repairing" (Status = Gap_Repairing).

[0119] Example: Given the intervals [100, 200] and [250, 300], a new sequence number 220 arrives (Gap = 49>10), create [220, 220], and request resend [201, 219] and [221, 249].

[0120] Case D: The sequence number immediately adjacent to the boundary; Condition: S = B + 1 or S = C – 1; Processing: Prioritize merging with adjacent intervals. If S = B + 1, expand the previous interval to [A, S]; if S = C - 1, expand the next interval to [S, D]. After expansion, check if adjacent intervals can be further merged (for example, if the expanded [A, S] is [A, D] and S = C - 1, then it can be merged into [A, D]).

[0121] Example: Given the intervals [100, 200] and [202, 300], a new sequence number 201 arrives, expanding [100, 200] to [100, 201], and then merging them into [100, 300].

[0122] Step 5: Update gap metadata and statistics: The system maintains a metadata table for each gap, recording the following information: Gap identifier: Gap_ID (consisting of partition ID and interval boundary); Gap range: [B+1, C-1]; Current status: Monitoring, Gap_Repairing, Filled; Creation time: Gap_Created_Time; Last update time: Gap_Updated_Time; Received sequence number set: Received_Seq_Set (the single point intervals that have been reached in the record gap, such as {S1,S2, ...}); When a new serial number is inserted into the gap, the system updates the metadata for the corresponding gap: If the gap is completely filled (Case A), mark the status as Filled and remove it from the list of active gaps.

[0123] If the gap is filled (cases B and D), update Received_Seq_Set and Gap_Updated_Time.

[0124] If a resend is triggered (Case C), record the request time and scope of the resend request for subsequent monitoring of the resend completion status.

[0125] Step 6: Perform periodic gap assessment and optimization. The system is configured with a background evaluation task that scans all intervals in the "Monitoring" state at a fixed period (e.g., every 30 seconds): Assessment condition 1: Failure to complete the task within the time limit.

[0126] If the current time - Gap_Created_Time > Max_Gap_Wait_Time (e.g., 60 seconds), the system determines that the gap has not been filled for a long time, automatically triggers an active resend request, and updates the status to Gap_Repairing.

[0127] Evaluation condition 2: Severe fragmentation of the gaps.

[0128] If the number of received single-point intervals in a gap [B+1, C-1] exceeds the fragmentation threshold (e.g., 50% of the gap span), it means that most of the sequence numbers in that gap have arrived, but a small number are still missing. The system triggers precise retransmission, requesting only the remaining missing small segments to avoid retransmitting already received data.

[0129] Evaluation condition 3: The gap is filled naturally.

[0130] If all sequence numbers in the gap have been received (by traversing Received_Seq_Set), the system performs a batch merging operation, merging all single-point intervals within the gap with the preceding and following intervals into a continuous interval, and marking the status as Filled.

[0131] This periodic assessment ensures that gaps are addressed promptly, preventing long-term, uncorrected data gaps.

[0132] For exception handling in special scenarios, the following methods can be used: For example, scenario 1: duplicate serial number detection.

[0133] If the new sequence number S already exists in the Received_Seq_Set of the gap or duplicates the existing single-point interval [S, S], the system determines it as a duplicate transmission, directly discards this sequence number, does not perform any interval operations, and records duplicate logs for auditing.

[0134] For example, Scenario 2: Sequence number rollback detection.

[0135] If the new sequence number S < B (the right boundary of the previous interval), it indicates that a sequence number rollback has occurred, which may be caused by the producer restarting or configuration errors. The system triggers an alarm, pauses the synchronization of this partition, and notifies the operation and maintenance personnel to troubleshoot the problem.

[0136] For example, Scenario 3: Abnormal gap span.

[0137] If the Gap exceeds the abnormal threshold (such as 10000), it indicates that there may be large-scale data loss or long-term downtime. The system triggers a high-priority alarm, records detailed logs (including partition ID, gap range, received sequence number S), and recommends performing a full data resynchronization instead of only relying on the reissuance mechanism.

[0138] In an optional embodiment, to optimize large-scale synchronization, a batch gap detection mechanism is also introduced: The consumer cluster periodically or when performing synchronization checks analyzes the range set, identifies all missing ranges (such as [1001, 1199]). Specifically, the operations of gap detection can include the following steps: The first step is to scan the sequence number range set periodically or triggered.

[0139] Nodes in the consumer cluster set interval triggers and start scanning the range set at regular intervals (such as every minute or every hour); it can also be triggered based on the synchronization delay threshold or synchronization failure alarm.

[0140] For the sequence number range set of each producer partition, arrange all currently maintained continuous sequence number intervals in order to form an ordered closed interval list, such as: [ [1, 1000], [1002, 1500], [1510, 2000] ].

[0141] The second step is to traverse the range and detect gaps in the continuous intervals.

[0142] Traverse all adjacent interval pairs [A, B] and [C, D] in the list in ascending order: If C > B + 1, then the interval [B + 1, C - 1] is the missing interval.

[0143] Record all such missing intervals in sequence to form a missing range set.

[0144] For a single isolated sequence number (e.g., only sequence numbers 1000 and 1002 are received, but 1001 is not received), it is also determined to be a missing interval with an interval length of 1.

[0145] The third step is to maintain the metadata for the missing range.

[0146] For each missing interval, maintain information such as an identifier, its corresponding partition ID, the start and end sequence numbers of the missing interval, and the detection timestamp. Then, support periodic retries and status tracking of the missing range through metadata.

[0147] The above method identifies several missing sequence numbers and can be proactively triggered to send a request to the producer cluster for data synchronization. Once the missing Binlog data is obtained through the metadata-driven completion process, the consumer will treat it as normal, newly received Binlog and apply it.

[0148] By identifying several missing serial numbers using the above method, the range of missing serial numbers is stored compactly, reducing memory usage and query overhead, and lowering operation and maintenance costs.

[0149] S23. Obtain supplementary data from the producer cluster based on the missing sequence numbers, and use the supplementary data to update the data so that the partition storage data is synchronized with the data stored in the producer cluster.

[0150] After identifying several missing sequence numbers, a batch request can be sent to the producer cluster to request the simultaneous resending of Binlog data for multiple sequence number ranges, thus obtaining the complete data. Specifically, the operation of sending a batch request to obtain the complete data can include the following steps: The first step is to construct batch requests based on the missing sequence number ranges: relying on the set of missing ranges, all missing ranges are merged according to the partition ID to form a batch request packet. For example, a request for [1001, 1010] is made for partition P0, and a request for [5001, 5020] is made for partition P5, forming a single request message.

[0151] It's important to note that the core technical advantage of using single batch requests lies in network transmission efficiency and system resource optimization. In a large-scale distributed environment, each independent request incurs complete network protocol stack overhead, including TCP connection establishment, request header encapsulation, and response processing. When the number of missing intervals reaches hundreds, the independent request method leads to: 1) a surge in network connections, consuming a large amount of system handle resources; 2) accumulated request queuing delays, with the total synchronization time increasing linearly; and 3) the producer needs to maintain multiple concurrent response threads, increasing context switching overhead.

[0152] In contrast, batch requests, through request aggregation technology, encapsulate multiple missing intervals into a single data packet, achieving an exponential improvement in network transmission efficiency.

[0153] The second step is to design the request message structure: The request includes: a list of partition IDs, a set of corresponding missing interval sequence numbers, and a request timestamp. The consumer sends this request to the corresponding producer node or intermediate synchronization broker.

[0154] The third step involves the producer cluster or agent querying the range log data for the received batch requests. The producer quickly locates the requested sequence number range using an index (such as a sequence number-based log storage index). It leverages the storage system's range lookup capabilities (such as skip lists, B-tree indexes, or time range scans in time-series databases) to avoid traversing each entry individually. The corresponding Binlog entries are extracted in batches, packaged sequentially, and returned to complete the data.

[0155] After obtaining the supplementary data, it can be used to update the data so that the data in the partition storage is synchronized with the data in the producer cluster storage.

[0156] Specifically, the sequence number S for completing the data can be obtained, and the data can be completed based on whether the sequence number S is connected to the existing sequence number range.

[0157] In one implementation, when the serial number S of a new application is continuous with the existing range [A, B], the extended range can be determined to be [A, S], and multiple consecutive serial numbers can be applied in batches to reduce the update frequency. Here, it refers to the connection between the upper bound B of the existing range and the serial number S.

[0158] When multiple ranges are joined due to new application logs (e.g., filled with sequence numbers between B+1 and C-1), the system merges the ranges into [A, D] and further compresses the storage. Specifically, the operation when multiple ranges are joined due to new application logs includes the following steps: The first step is to receive and sort the new sequence number range set.

[0159] The system receives several consecutive or non-consecutive new sequence number ranges, for example, new log sequence number ranges are listed as [S1_start, S1_end], [S2_start, S2_end],...; Sort these intervals in ascending order of their serial numbers.

[0160] The second step is to attempt to merge the existing range sets for each new interval.

[0161] Iterate through the existing range set (sorted by the starting index of the interval) and find all that satisfy: There is no gap between the end index B of the existing range and the start index S_start of the new interval (i.e., B + 1 ≥ S_start). Or there is no gap between the end index S_end of the new interval and the start index A of the next range (i.e., S_end + 1 ≥ A). The third step is to construct the merged continuous range.

[0162] Based on the search results from the second step, adjacent or overlapping intervals are merged into a new continuous range [Min(A_1,S_start), Max(B_n, S_end)], where: A_1 is the starting index of the first existing range. B_n is the end number of the last merged range.

[0163] The fourth step is to delete the old range that was merged and insert the new merged range.

[0164] Remove the multiple original ranges and new sequence number intervals merged in the third step from the range set; Insert new merged ranges to replace the deleted ranges, ensuring that there are no intersections or overlaps in the range set.

[0165] The fifth step is to verify and optimize the merged range set.

[0166] To ensure the accuracy and compactness of the range set, the set is scanned periodically, and continuous intervals that can be further merged are merged. Sort the range set according to the starting index, check whether adjacent intervals are consecutive (the end index of the previous interval + 1 ≥ the starting index of the next interval), and merge them if so.

[0167] If the new sequence numbers are not consecutive, a new range [S, S] is added. This logic improves efficiency in high-concurrency environments by minimizing update operations.

[0168] In another implementation, the data completion operation performed by the consumer cluster can also be executed in the following way: The first step is for the consumer cluster to receive batch resend data: the nodes of the consumer cluster receive missing interval data packets (Binlog sequences) from multiple partitions in parallel, ensuring that the receiving end is multi-threaded or event-driven.

[0169] The second step is to complete the logs and update the range set: For each Binlog that is resent back, the consumer calls the sequence number range set update logic defined in step (4): Determine if the current log sequence number S is consecutive to the existing range: The continuity judgment is "S == B + 1", where B is the last index of the current range.

[0170] If continuous, the extended range is [A, S].

[0171] For concatenating two (or more) existing ranges of padded logs, the ranges are merged as follows: If new serial numbers are added to fill the gaps in the old range, they will be merged into a larger continuous interval.

[0172] If they are not continuous, a new independent interval [S, S] is added.

[0173] The third step is to optimize batch updates and merging: newly added data is often a continuous range, so batch merging is used to update it, which reduces the system overhead of updating a single data entry.

[0174] By using transactions or batch operations, ensure that range sets are updated atomically, thus avoiding concurrency conflicts.

[0175] After the update, the missing range identification is re-executed, and the filled ranges are removed.

[0176] Step 4: Synchronize and monitor the status of missing ranges: Update the metadata of missing ranges and mark the missing ranges as cleared or partially cleared.

[0177] Log operations and status completion are written to persistent storage for subsequent fault tolerance and maintenance queries.

[0178] For retry failures or timeout completion ranges, alarms can be triggered or synchronization strategies can be adjusted.

[0179] In addition, the consumer cluster supports partition aggregation, which allows the consumer cluster to perform parallel range updates and gap repairs on multiple partitions, thereby improving the overall synchronization throughput.

[0180] To further illustrate, a comprehensive explanation is provided, combining the operations of producer and consumer clusters. The data synchronization process of this invention revolves around four core stages: "sequence number generation - log transmission - range update - gap repair," and is fully adaptable to high-concurrency, multi-partition scenarios in large-scale distributed database clusters. The specific steps are as follows (broken down in data flow order): I. Preliminary preparations: Partitioning and sequence number initialization (producer cluster end).

[0181] Database partitioning: A database contains multiple independent partitions, each storing different data (e.g., split by user or business module), and each partition has a unique partition ID (e.g., P0, P1, P2).

[0182] Partition Serial Number Counter Initialization: Create an independent counter for each partition, with the initial value set to 0 or the last synchronization breakpoint + 1. The counter status is persistently stored in a distributed storage such as ZooKeeper / Etcd to prevent loss due to node crashes.

[0183] Synchronization Configuration Preset: Set core parameters, including the serial number buffer ratio (0.1 - 0.2), minimum allocation step size (MinStep), gap threshold (Threshold_Gap = 10), data timeout threshold (Timeout_Threshold = 5 seconds), etc., to adapt to high-concurrency and network fluctuation scenarios.

[0184] II. Phase 1: Serial Number Generation and Binlog Binding (Producer Cluster Side).

[0185] When a partition receives a write transaction request, complete the serial number allocation and log binding to ensure that each Binlog is uniquely traceable: Transaction Type Identification and Binlog Count Estimation: Single INSERT / UPDATE / DELETE: N = 1; Batch Operations: Set N = M / R according to the batch size M or the estimated number of affected rows R; Cross-table Transactions: N = the cumulative sum of the number of operations on each table (ΣN_i); DDL Operations: Fixed N = 1.

[0186] Serial Number Range Calculation and Allocation: Read the current counter value CurrentSeq; Calculate the allocation range according to the formula: [StartSeq = CurrentSeq + 1, EndSeq = CurrentSeq + N]; High-concurrency Optimization: Update the counter to CurrentSeq + N through a CAS atomic operation, support retry (3 - 5 times) and exponential backoff strategy to avoid conflicts.

[0187] Dynamic Range Adjustment (Optional): High-concurrency Scenarios: Expand N to N' = N × (1 + buffer ratio) to reserve serial numbers; Small Transaction Scenarios: If N < MinStep, force N = MinStep to ensure allocation continuity.

[0188] Serial Number and Binlog Binding: Each Binlog entry is attached with a unique identifier of "partition ID - serial number" (such as P0 - 100), and the producer-generated timestamp is recorded.

[0189] Counter state persistence: The updated counter value is synchronized to the distributed storage in real time, supporting fault recovery.

[0190] Phase 2: Binlog Batch Transmission (Producer → Consumer).

[0191] Employing a multi-threaded parallel mechanism reduces network bottlenecks and ensures efficient delivery of batch logs. Transmission task division: More than 1,000 partitions are divided into independent transmission units according to partition ID, and dynamically allocated to N parallel threads (the thread pool size is adjusted according to hardware resources, usually from tens to hundreds).

[0192] Batch packaging of logs: Each thread reads the new Binlog from the partition according to the rule of "1000 records per batch" or "every 100ms", and packages it into data blocks in ascending order of sequence number.

[0193] Parallel transmission and load balancing: Asynchronously send data blocks to the corresponding consumer node by calling the RPC / message queue interface, supporting flow control mechanism; The scheduling module monitors thread load in real time and migrates the partitioned tasks of high-load threads to idle threads to avoid blocking.

[0194] Transmission confirmation: After verifying the continuity of the sequence number, the consumer returns a confirmation signal to the producer thread. The producer then clears the cache and prepares for the next batch of transmission.

[0195] Phase 3: Binlog application and serial number range update (consumer end).

[0196] After receiving the logs, the consumer applies them to the local partition and maintains a range set to dynamically track the synchronization status. Binlog reception and sequence number verification: Extract the partition ID and sequence number (Sequence_Number); Validation of validity: The serial number is a positive integer, not less than the previous maximum serial number - 1, and within the expected range.

[0197] Query the current range set: retrieve the list of applied continuous intervals (e.g., {[1,1000], [1200,1500]}) from the sequence number tracker (mapping table) by partition ID; if no record is found, initialize an empty set.

[0198] Serial number matching with existing range: If the sequence number is within the existing range: mark it as a duplicate and discard the log; If the sequence number = the right boundary of the interval + 1 (e.g., 1001 = 1000 + 1): the extended interval is [1, 1001]; If the serial number falls within the gap between two intervals (e.g., 1005 falls between [1000, 1004] and [1006, 1500]): Calculate the gap span Gap = CB - 1, and judge it in combination with the timeliness (Delay = reception time - generation time); For small gaps (Gap≤10) and real-time data (Delay<5 seconds): Create a single-point interval [1005,1005] and mark it as "Monitoring"; Large gaps (Gap>10) or delayed data (Delay≥5 seconds): Create a single-point interval and trigger a retransmission request.

[0199] Scope merging and optimization: When a new interval is consecutive with an adjacent interval (such as [1000,1001] and [1002,1500]), it is merged into [1000,1500]. The background system scans the "monitoring" intervals periodically (every 30 seconds). If the missing information is not completed within the time limit, a resend will be triggered. If the fragmentation is severe, the missing segments will be accurately resent.

[0200] Phase 4: Batch gap detection and repair (consumer-driven).

[0201] Proactively identify missing logs and resend them in batches to ensure seamless data synchronization: Missing range detection: A set of scan ranges that can be performed at set intervals (1 minute / hour) or triggered by events (synchronization delay exceeding limits); Iterate through adjacent interval pairs [A,B] and [C,D]. If C>B+1, then [B+1,C-1] is a missing interval (e.g., [1001,1001], [5001,5009]). Record the missing range metadata (partition ID, start and end sequence numbers, detection time).

[0202] Batch resend request construction: Aggregate all missing intervals by partition ID and encapsulate them into a single batch request (e.g., P0-[1001,1010], P5-[5001,5020]) to reduce network overhead; The request includes a list of partition IDs, a set of missing ranges, and a request timestamp.

[0203] Missing log query and return (producer cluster side): Producers can quickly locate the Binlog corresponding to the missing interval using a sequence number index (skip list / B-tree); Extract logs in batches, encapsulate them in order, and return them to the consumer.

[0204] Log reissue application and scope update: Consumers receive resent data via multiple threads and update the logic application logs according to the scope of phase 3. Merge consecutive intervals, update missing range metadata to "filled", and rescan the range set to clear the repaired gaps.

[0205] VI. Phase 5: Dynamic Partition Synchronization and Adaptation (Extended Scenarios).

[0206] When a producer adds a new partition (such as P100), the consumer integrates seamlessly: Initialize an empty range set (P100→{}); Log requests start from sequence number 0, and a continuous range (e.g., [0, 100]) is gradually built up after application. If gaps occur, follow the batch testing and repair process described above to ensure rapid synchronization of the new partition.

[0207] In practical applications, it is assumed that the producer cluster has multiple partitions (such as P0 to P9), and the consumer cluster C is responsible for large-scale data synchronization. The system is optimized for batch processing and scalability.

[0208] Initial state: Consumer C's range set for all partitions is empty.

[0209] Copying process: (1) The producer generates a large amount of Binlog, such as the sequence numbers 0-1000 of partition P0 and 0-500 of partition P1. C receives and applies the logs, and updates the range set to P0 -> {[0, 1000]} and P1 -> {[0, 500]}.

[0210] (2) In high-concurrency writing, C may receive sequence numbers 1002-1050 from P0 first, but 1001 is missing. After application, the range set becomes P0 -> {[0, 1000], [1002, 1050]}, and the system automatically identifies the gap.

[0211] (3) C requests the missing range [1001, 1001] through the batch mechanism, and after application, the merged range is P0 -> {[0,1050]}.

[0212] Example 1: Large-scale partition synchronization scenario.

[0213] In a cluster containing 1000 partitions, consumer C's range set exhibits multiple gaps (e.g., partition P50 has {[0, 5000], [5010, 6000]}, missing 5001-5009). C uses batch gap detection to identify all missing ranges and sends a single request to the producer to reissue all missing Binlogs (e.g., [5001, 5009] in P50). The synchronization process handles multiple partitions in parallel, and the recovered range set is optimized into continuous intervals, improving overall throughput and reducing synchronization time from hours in traditional methods to minutes.

[0214] Example 2: Adding scenes through dynamic partitioning.

[0215] When a producer adds a new partition P100, C initializes an empty range set P100 -> {}. C requests logs starting from sequence number 0, gradually building the range after application (e.g., after applying 0-100, it becomes [0, 100]). If gaps occur, C uses range analysis to batch request re-sending, ensuring rapid synchronization of the new partition. The entire process demonstrates the scalability of the method, maintaining stable synchronization efficiency even when the number of partitions doubles.

[0216] Summary of the core circulation loop: Producer: Transaction → Sequence Number Allocation → Binlog Binding → Batch Transmission → Confirmation Feedback; Consumer: Receive verification → Range update → Gap detection → Batch resend → Log application; The entire process utilizes "independent partition serial numbers + range set tracking + batch processing" to reduce synchronization latency by more than 30%, supports parallel synchronization of thousands of partitions, and is suitable for high-concurrency scenarios such as big data and cloud storage.

[0217] In this embodiment, the present invention provides a database data synchronization method, the advantages of which are as follows: The present invention significantly improves the performance and scalability of large-scale data synchronization through a sequence number range tracking mechanism. First, the state representation is compact, and the range set reduces storage and query overhead, enabling efficient management even in scenarios with tens of thousands of logs. Second, the synchronization efficiency is high; the batch gap detection and request mechanism reduces network interaction, reducing synchronization latency by more than 30%. Third, it has strong scalability, supporting dynamic partition addition and parallel processing, making it suitable for big data platforms. Fourth, gap detection is fast and natural through range gap representation, reducing manual intervention and lowering operation and maintenance costs, making it particularly suitable for high-concurrency applications such as cloud storage and the Internet of Things.

[0218] This invention also provides a database data synchronization device, see [link to documentation]. Figure 3 The diagram shows a schematic representation of a data synchronization device (producer cluster end) for a database according to an embodiment of the present invention.

[0219] The device is suitable for a producer cluster of a distributed database, which communicates with several consumer clusters. As an example, the database data synchronization device may include: The construction module 201 is used to construct Binlog information, wherein the Binlog information is generated by binding the monotonically increasing DR sequence number of each partition with Binlog events; Parallel transmission module 202 is used to send the Binlog information in parallel to several consumer clusters, so that each consumer cluster can identify several missing sequence numbers in the corresponding partition according to the sequence number of the Binlog information. Sending data 203 is used to send corresponding completion data to each of the consumer clusters according to the completion request uploaded by each of the consumer clusters, so that the consumer clusters can use the completion data to perform data synchronization. The completion request is a request constructed by the consumer cluster based on several missing sequence numbers of the identified partition.

[0220] Optionally, the construction of Binlog information includes: Determine the Binlog event of the target partition, which is either an extended partition or a partition where a write transaction was executed; The DR sequence number corresponding to the target partition is obtained based on the Binlog event. The DR sequence number is a range of sequence numbers constructed by monotonically increasing the counter established when the target partition is created or initialized based on the executed events. Bind the DR sequence number with the Binlog event to obtain the Binlog information.

[0221] Optionally, obtaining the DR sequence number corresponding to the target partition based on the Binlog event includes: The corresponding Binlog entry value is determined based on the transaction type of the Binlog event, and the real-time count value of the counter for the target partition is obtained. The initial sequence number range is determined using the Binlog line values ​​and the real-time count values; After dynamically adjusting and updating the initial sequence number range, a DR sequence number is obtained. The dynamic adjustment is to adjust the maximum value of the initial sequence number range according to a preset reserved buffer or a preset allocation step size. The update verification is to check the validity of the sequence number range after updating the counter of the target partition with the initial sequence number range.

[0222] Optionally, sending the Binlog information to several consumer clusters in parallel includes: The Binlog information is allocated to the corresponding partition tasks according to the number of database partitions, and the threads corresponding to the partition tasks are matched in a preset thread pool so that each thread performs batch reading and sequence number packaging of the Binlog information of the partition task to obtain Binlog data packets. The Binlog data packet is sent to several consumer clusters by calling a preset network transmission interface. After receiving the confirmation signal from the consumer cluster, the cache is cleared. The confirmation signal is a signal returned by the consumer cluster to the corresponding producer cluster after receiving the Binlog data packet and verifying the continuity of the sequence number of the Binlog data packet.

[0223] This invention also provides a database data synchronization device, see [link to documentation]. Figure 4 The diagram shows a schematic representation of a data synchronization device (consumer cluster end) for a database according to an embodiment of the present invention.

[0224] The apparatus is suitable for a consumer cluster of a distributed database, the consumer cluster being communicatively connected to a producer cluster. As an example, the database data synchronization apparatus may include: The acquisition module 301 is used to acquire Binlog information sent by the producer cluster, wherein the Binlog information is information generated by the producer cluster by binding the monotonically increasing DR sequence number of each partition with the Binlog event; The identification module 302 is used to identify several missing sequence numbers in the corresponding partition based on the sequence number of the Binlog information; The synchronization module 303 is used to obtain supplementary data from the producer cluster based on the several missing sequence numbers, and use the supplementary data to update the data so that the partition storage data is synchronized with the data stored in the producer cluster.

[0225] Optionally, identifying several missing sequence numbers within the corresponding partition based on the sequence numbers in the Binlog information includes: Extract the serial number to be updated and the partition ID from the Binlog information. The serial number to be updated is the serial number that has completed the audit and verification. The audit and verification verifies the value of the serial number, the maximum serial number, and the range of the serial number. Obtain the real-time sequence range from the built-in tracker of the partition ID, wherein the built-in tracker is a mapping table of partition IDs and sequence number ranges for the producer cluster; Traverse all intervals of the real-time sequence range and determine the attribution relationship between the sequence number to be updated and all intervals of the real-time sequence range; Based on the attribution relationship, several sequence extension ranges are determined, and the sequence number of each sequence extension range is obtained, resulting in several missing sequence numbers.

[0226] Those skilled in the art will understand that, for ease of description and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0227] Furthermore, this application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the database data synchronization method as described in the above embodiments.

[0228] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program for causing a computer to execute the database data synchronization method as described in the above embodiments.

[0229] In the description of the embodiments of the present invention, it should be noted that the terms "above," "below," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. When an element such as a layer, region, or substrate is referred to as being "above" or "on top of" another element, it may be directly on the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly on" or "above" another element, there is no intermediate element. It should also be understood that when an element is referred to as being "below" or "under" another element, it may be directly below or under the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly below" or "under" another element, there is no intermediate element. Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0230] Those skilled in the art will understand that embodiments of this application may also include computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0231] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), devices, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0232] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0233] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0234] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A database data synchronization method, characterized in that, The method is applicable to a producer cluster of a distributed database, wherein the producer cluster is communicatively connected to several consumer clusters, and the method includes: Construct Binlog information, wherein the Binlog information is generated by binding the monotonically increasing DR sequence number of each partition with Binlog events; The Binlog information is sent in parallel to several consumer clusters, so that each consumer cluster can identify several missing sequence numbers in the corresponding partition based on the sequence number of the Binlog information. Based on the completion request uploaded by each consumer cluster, corresponding completion data is sent to each consumer cluster for the consumer cluster to use for data synchronization. The completion request is a request constructed by the consumer cluster based on several missing sequence numbers of the identified partition.

2. The database data synchronization method according to claim 1, characterized in that, The constructed Binlog information includes: Determine the Binlog event of the target partition, which is either an extended partition or a partition where a write transaction was executed; The DR sequence number corresponding to the target partition is obtained based on the Binlog event. The DR sequence number is a range of sequence numbers constructed by monotonically increasing the counter established when the target partition is created or initialized based on the executed events. Bind the DR sequence number with the Binlog event to obtain the Binlog information.

3. The database data synchronization method according to claim 2, characterized in that, The step of obtaining the DR sequence number corresponding to the target partition based on the Binlog event includes: The corresponding Binlog entry value is determined based on the transaction type of the Binlog event, and the real-time count value of the counter for the target partition is obtained. The initial sequence number range is determined using the Binlog line values ​​and the real-time count values; After dynamically adjusting and updating the initial sequence number range, a DR sequence number is obtained. The dynamic adjustment is to adjust the maximum value of the initial sequence number range according to a preset reserved buffer or a preset allocation step size. The update verification is to check the validity of the sequence number range after updating the counter of the target partition with the initial sequence number range.

4. The database data synchronization method according to claim 1, characterized in that, Sending the Binlog information in parallel to several of the consumer clusters includes: The Binlog information is allocated to the corresponding partition tasks according to the number of database partitions, and the threads corresponding to the partition tasks are matched in a preset thread pool so that each thread performs batch reading and sequence number packaging of the Binlog information of the partition task to obtain Binlog data packets. The Binlog data packet is sent to several consumer clusters by calling a preset network transmission interface. After receiving the confirmation signal from the consumer cluster, the cache is cleared. The confirmation signal is a signal returned by the consumer cluster to the corresponding producer cluster after receiving the Binlog data packet and verifying the continuity of the sequence number of the Binlog data packet.

5. A database data synchronization method, characterized in that, The method is applicable to consumer clusters in a distributed database, wherein the consumer cluster communicates with a producer cluster, and the method includes: Obtain the Binlog information sent by the producer cluster, wherein the Binlog information is generated by the producer cluster binding the monotonically increasing DR sequence number of each partition with the Binlog event; Based on the sequence number of the Binlog information, identify several missing sequence numbers within the corresponding partition; Based on the missing sequence numbers, complete data is obtained from the producer cluster, and the complete data is used to update the data so that the partition storage data is synchronized with the data stored in the producer cluster.

6. The database data synchronization method according to claim 5, characterized in that, The step of identifying several missing sequence numbers in the corresponding partition based on the sequence numbers of the Binlog information includes: Extract the serial number to be updated and the partition ID from the Binlog information. The serial number to be updated is the serial number that has completed the audit and verification. The audit and verification verifies the value of the serial number, the maximum serial number, and the range of the serial number. Obtain the real-time sequence range from the built-in tracker of the partition ID, wherein the built-in tracker is a mapping table of partition IDs and sequence number ranges for the producer cluster; Traverse all intervals of the real-time sequence range and determine the attribution relationship between the sequence number to be updated and all intervals of the real-time sequence range; Based on the attribution relationship, several sequence extension ranges are determined, and the sequence number of each sequence extension range is obtained, resulting in several missing sequence numbers.

7. A data synchronization device for a database, characterized in that, The apparatus is suitable for a producer cluster of a distributed database, wherein the producer cluster is communicatively connected to several consumer clusters, and the apparatus includes: The construction module is used to construct Binlog information, wherein the Binlog information is generated by binding the monotonically increasing DR sequence number of each partition with Binlog events; A parallel sending module is used to send the Binlog information to several consumer clusters in parallel, so that each consumer cluster can identify several missing sequence numbers in the corresponding partition according to the sequence number of the Binlog information. Sending data is used to send corresponding completion data to each of the consumer clusters according to the completion request uploaded by each of the consumer clusters, so that the consumer clusters can use the completion data to perform data synchronization. The completion request is a request constructed by the consumer cluster based on several missing sequence numbers of the identified partition.

8. A data synchronization device for a database, characterized in that, The device is suitable for a consumer cluster of a distributed database, wherein the consumer cluster is communicatively connected to a producer cluster, and the device includes: The acquisition module is used to acquire Binlog information sent by the producer cluster, wherein the Binlog information is generated by the producer cluster binding the monotonically increasing DR sequence number of each partition with the Binlog event; The identification module is used to identify several missing sequence numbers in the corresponding partition based on the sequence number of the Binlog information; The synchronization module is used to obtain supplementary data from the producer cluster based on the missing sequence numbers, and use the supplementary data to update the data so that the partition storage data is synchronized with the data stored in the producer cluster.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the program, it implements a database data synchronization method as described in any one of claims 1-4 or a database data synchronization method as described in any one of claims 5-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform a database data synchronization method as described in any one of claims 1-4 or as described in any one of claims 5-6.

Images