Data synchronization method and device among clusters

Abstract

The application provides a data synchronization method and device between clusters, the method comprising: obtaining to-be-synchronized data and a current latest global version number in a source cluster; dividing the to-be-synchronized data to obtain a plurality of first data blocks; transmitting the plurality of first data blocks to a target cluster by using a plurality of threads; obtaining a first newly-added global version number after the current latest global version number in the source cluster and a data change event corresponding to the first newly-added global version number; and updating the plurality of first data blocks in the target cluster according to the data change event corresponding to the first newly-added global version number in the order of the first newly-added global version number. The efficiency of full data synchronization between clusters is improved. Moreover, the problem of missing synchronization or out-of-order writing of incremental data caused by network delay or processing interruption in the traditional synchronization scheme is effectively solved, the target cluster can still maintain the final consistency with the source cluster, and the accuracy and reliability of cross-cluster data synchronization are improved.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to a data synchronization method, a data synchronization device between clusters, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] etcd is a distributed key-value store system, typically deployed in a cluster. An etcd cluster contains multiple communicating nodes that collaborate using the Raft consensus algorithm to maintain a persistent, strongly consistent copy of the data. However, as business scales and disaster recovery requirements increase, a single etcd cluster may not be sufficient for cross-regional and cross-datacenter data deployments. In such cases, data synchronization between multiple etcd clusters becomes necessary.

[0003] Currently, data synchronization between multiple etcd clusters primarily utilizes the data synchronization tools provided by the official etcd documentation, such as the `make-mirror` function in `etcdctl`. During the synchronization process, this tool first uses a thread to perform a full data scan of the source etcd cluster, retrieving key-value pairs one by one in a linear traversal manner. For each retrieved data pair, the thread encapsulates it into a write request and transmits it over the network to the target etcd cluster. The target cluster then executes the write operation on each data pair received. This process continues until all key-value pairs have been processed, at which point the full synchronization is complete.

[0004] However, the aforementioned data synchronization tools rely on a single serial thread during data reading, transmission, and writing. The synchronization time is strictly linearly proportional to the total amount of data. When processing massive amounts of data, this significantly prolongs the full synchronization cycle, thereby reducing the data synchronization efficiency between etcd clusters. Summary of the Invention

[0005] The purpose of this application is to provide a data synchronization method, a data synchronization device, a computer device, a computer-readable storage medium, and a computer program product for inter-cluster data synchronization, so as to improve the efficiency of data synchronization between clusters.

[0006] To address the aforementioned technical problems, this application provides the following technical solutions: The first aspect of this application provides a data synchronization method between clusters. The method is applied to a data synchronization device, which is connected to a source cluster and a target cluster respectively. The source cluster records data change events and corresponding global version numbers according to time. The method includes: obtaining the data to be synchronized and the latest global version number in the source cluster; dividing the data to be synchronized into multiple first data blocks; using multiple threads to transmit the multiple first data blocks to the target cluster; obtaining the first newly added global version number after the latest global version number in the source cluster and the data change event corresponding to the first newly added global version number; and updating the multiple first data blocks in the target cluster according to the order of the first newly added global version number and based on the data change event corresponding to the first newly added global version number.

[0007] Compared to existing technologies, the data synchronization method for inter-cluster data provided in the first aspect of this application improves the efficiency of full data synchronization between clusters by using multi-threaded concurrent transmission of data to be synchronized in the source cluster. Furthermore, after completing the full data transmission, by obtaining the first newly added global version number after the latest global version number in the source cluster and the corresponding data change event, and strictly updating the synchronized data in the target cluster according to the version number order, even if the source cluster data changes in real time during the synchronization process, versioned event listening and sequential processing can still achieve accurate tracking and orderly synchronization of incremental data. This effectively solves the problem of missed synchronization or out-of-order writing of incremental data caused by network latency or processing interruptions in traditional synchronization schemes, ensuring that the target cluster can still maintain eventual consistency with the source cluster, thus improving the accuracy and reliability of cross-cluster data synchronization.

[0008] In other embodiments provided in this application, the data to be synchronized is a portion of all data synchronized from the source cluster to the target cluster; the method further includes: obtaining other synchronized data besides the data to be synchronized and the latest global version number from the source cluster; dividing the other synchronized data to obtain multiple second data blocks; using multiple threads to transmit the multiple second data blocks to the target cluster; obtaining a second newly added global version number after the latest global version number in the source cluster and the data change event corresponding to the second newly added global version number; modifying the multiple second data blocks in the target cluster according to the order of the second newly added global version number and based on the data change event corresponding to the second newly added global version number, until all data in the source cluster is transmitted to the target cluster.

[0009] By acquiring and processing portions of the data to be synchronized in batches, the memory usage per transaction is effectively controlled, avoiding memory overflow or sudden drop in system performance caused by loading all data at once, thus ensuring the stable operation of the large-scale data synchronization process.

[0010] In other embodiments provided in this application, each piece of data in all data has a corresponding number, which is used to indicate that each piece of data is read sequentially; the method further includes: obtaining the first number of the last piece of data in the data to be synchronized; obtaining other synchronized data other than the data to be synchronized and the latest global version number at this time from the source cluster, including: determining the next number of the first number; starting from the next number in the source cluster, obtaining a preset number of data to obtain other synchronized data, and obtaining the latest global version number at this time.

[0011] Sequential pagination based on numbering ensures that no data is missed or duplicated during the full data retrieval process. Furthermore, it allows for precise location recovery after synchronization interruption, avoiding resource waste and time delays caused by full resynchronization.

[0012] In other embodiments provided in this application, multiple threads are used to transmit multiple first data blocks to the target cluster, including: when a thread among the multiple threads fails, determining the first data block corresponding to the failed thread, and transmitting the first data block corresponding to the failed thread to the target cluster using a normal thread among the multiple threads.

[0013] By using thread-level fault detection and task reallocation, it can be ensured that an abnormality in a single thread does not affect the overall synchronization process, thereby improving the overall completion rate of synchronization tasks.

[0014] In other embodiments provided in this application, the data synchronization device records the numbers of data successfully transmitted to the target cluster, and the numbers are used to indicate that the data to be synchronized is transmitted in sequence; determining the first data block corresponding to the faulty thread includes: searching in the data synchronization device for a number range where a preset number of numbers are missing, the preset number being equal to or less than the number of data items transmitted by a thread in one go among multiple threads; and determining the data corresponding to the number range as the first data block corresponding to the faulty thread.

[0015] Based on missing data detection within a number range, it can quickly and accurately locate unsynchronized data blocks caused by thread failures, enabling automated fault identification and task reallocation, avoiding manual intervention, and improving the self-healing capability and reliability of data synchronization.

[0016] In other embodiments provided in this application, before updating multiple first data blocks in the target cluster based on the data change events corresponding to the first newly added global version number in the order of the first newly added global version number, the method further includes: writing the data change events into a normal queue in the order of the first newly added global version number, and writing the abnormal events that occur during transmission into an abnormal queue; updating multiple first data blocks in the target cluster based on the data change events corresponding to the first newly added global version number in the order of the first newly added global version number includes: reading the data change events from the normal queue in a first-in-first-out order, and updating multiple first data blocks in the target cluster based on the read data change events.

[0017] By isolating the normal event stream and the exception handling stream with dual queues, the sequential consumption and consistent writing of incremental data change events are guaranteed, while the classification buffering and asynchronous processing of exception events are realized, thereby improving the orderliness and reliability of data synchronization.

[0018] In other embodiments provided in this application, the abnormal event includes a connection loss event with the source cluster; the method further includes: reading the connection loss event from the abnormal queue; re-establishing a connection with the source cluster based on the connection loss event; obtaining a data change event from the source cluster according to the latest global version number recorded before the connection loss; writing the obtained data change event and the corresponding global version number into the normal queue; if a Multi-Version Concurrency Control (MVCC) compression abnormality occurs, writing the data change event corresponding to the earliest recorded global version number in the source cluster and the final data state of its previous global version number into the normal queue.

[0019] By automatically reconnecting after a connection loss and reconstructing the data state in case of MVCC compression anomalies, the synchronization process can be automatically restored and continue from the correct point in extreme abnormal scenarios. This effectively avoids data loss or synchronization stagnation caused by network interruptions or storage compression, ensuring the continuity of cross-cluster data synchronization.

[0020] A second aspect of this application provides a data synchronization device for inter-cluster operations. The data synchronization device is connected to a source cluster and a target cluster, respectively. The source cluster records data change events and corresponding global version numbers in a time-based manner. The data synchronization device includes: an acquisition module for acquiring the data to be synchronized and the latest global version number in the source cluster; a partitioning module for partitioning the data to be synchronized into multiple first data blocks; a first synchronization module for transmitting the multiple first data blocks to the target cluster using multiple threads; an incremental module for acquiring the first newly added global version number after the latest global version number in the source cluster and the data change event corresponding to the first newly added global version number; and a second synchronization module for updating the multiple first data blocks in the target cluster according to the order of the first newly added global version number and based on the data change event corresponding to the first newly added global version number.

[0021] A third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of the first aspect.

[0022] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect.

[0023] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of the first aspect.

[0024] The data synchronization apparatus for inter-cluster communication provided in the second aspect of this application, the computer equipment provided in the third aspect, the computer-readable storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect have the same or similar beneficial effects as the data synchronization method for inter-cluster communication provided in the first aspect. Attached Figure Description

[0025] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein: Figure 1 This is a schematic diagram illustrating an application scenario of the data synchronization method between clusters in this application embodiment; Figure 2 This is a flowchart illustrating the data synchronization method between clusters in this application embodiment. Figure 1 ; Figure 3 This is a flowchart illustrating the data synchronization method between clusters in this application embodiment. Figure 2 ; Figure 4This is a schematic diagram of the data synchronization device between clusters in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0026] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0027] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0028] It should be noted that the user data, data acquisition, and / or use involved in the embodiments of this application strictly comply with the laws, regulations, and industry standards of relevant countries and regions. The collection and acquisition of data involved in the embodiments of this application are all done in advance by actively prompting or prominently displaying information to inform users and obtaining authorization, or by obtaining full authorization from all parties. The processing, manipulation, forwarding, and use of data involved in the embodiments of this application are all carried out on the premise that the user or relevant party is fully informed and authorized. In implementing the various embodiments of this application, the types of data or information, scope of use, and usage scenarios that may be involved are informed to users or relevant parties and authorization is obtained through appropriate means. The specific methods of notification and authorization may vary according to the actual situation, and this application is not limited in this regard. The processing of personal information involved in the embodiments of this application is carried out under the premise of having a legal basis (such as obtaining the consent of the personal information subject or being necessary for the performance of a contract), and is only processed within the prescribed or agreed scope. Sensitive personal information such as biometric information, medical and health information, financial account information, and precise location information involved in the embodiments of this application are all processed under the premise of having a specific purpose and sufficient necessity, and with the separate authorization and consent of the user or relevant party. In some embodiments of this application, if the user or related party refuses to process personal information other than the information necessary for the basic functions, it will not affect the use of the basic functions of the embodiments of this application.

[0029] Current technical solutions for cross-cluster data synchronization mainly rely on the official single-threaded full synchronization device provided by the cluster. When processing massive amounts of data, this solution suffers from low synchronization efficiency and prolonged cycle due to serial operations, making it difficult to meet the requirements for high availability and high performance cross-data center data synchronization.

[0030] Even with multi-threaded concurrent transmission, a technical problem remains during cross-cluster data synchronization. Specifically, during full synchronization, data in the source cluster that has been read but not yet fully transmitted may be updated. These updates cannot be captured and transmitted in a timely manner during the synchronization process, resulting in the target cluster retaining only a snapshot of the data at the start of synchronization, rather than the latest data state. This leads to the loss of data updates generated within the synchronization window, thereby reducing the accuracy of inter-cluster data synchronization.

[0031] In view of this, embodiments of this application provide a data synchronization method, a data synchronization device, a computer device, a computer-readable storage medium, and a computer program product for inter-cluster data synchronization, which records and tracks incremental changes based on a global version number during the full synchronization process. This enables seamless integration of full and incremental synchronization and orderly updates of data status, ensuring that the target cluster ultimately maintains real-time data consistency with the source cluster, thus improving the accuracy and consistency of inter-cluster data synchronization.

[0032] First, the application scenarios of the data synchronization method between clusters provided in the embodiments of this application will be described.

[0033] In some embodiments, the data synchronization device is a data synchronization tool. Figure 1 This is a schematic diagram illustrating the application scenario of the data synchronization method between clusters in this application embodiment. See [link / reference] Figure 1 As shown, the scenario may include: source cluster 11, target cluster 12, and data synchronization tool 13.

[0034] In practical applications, the source cluster 11 and the target cluster 12 can be a distributed key-value storage cluster built on etcd, or a distributed database cluster, distributed configuration center cluster, or distributed coordination service cluster implemented based on other distributed consensus protocols (such as Raft, Paxos, ZAB, etc.), such as ZooKeeper cluster, Consul cluster, or Apache Kafka cluster.

[0035] In distributed key-value storage systems like etcd, all data modification operations (such as adding, deleting, and modifying key-value pairs) are persistently recorded as a data change event and assigned a globally monotonically increasing unique integer as a global version number. The global version number represents the logical order and timeline of the event within the entire cluster. For example, when a client executes `put key1 value1`, `delete key2`, and `put key3 value3` operations sequentially on the etcd cluster, the cluster will generate three data change events in sequence and assign them global version numbers (assuming they start from 100) of 100, 101, and 102 respectively. Each event and its corresponding global version number are persistently stored, forming a complete change history.

[0036] The data synchronization tool 13 can be deployed on a dedicated synchronization node or intermediate server independent of the source cluster 11 and the target cluster 12, communicating with the source cluster 11 and the target cluster 12 via standard network protocols (such as HTTP / gRPC). The data synchronization tool 13 can also be implemented as a containerized application, deployed in orchestration platforms such as Kubernetes, or attached to the business application container as a sidecar proxy. In some integrated solutions, the functionality of the data synchronization tool 13 can also be embedded into a proxy service of the source cluster 11 or the target cluster 12.

[0037] When data needs to be synchronized from the source cluster 11 to the target cluster 12, the data synchronization tool 13 first obtains the data to be synchronized and its latest global version number from the source cluster 11. Then, it divides the data to be synchronized into multiple first data blocks and uses multi-threaded concurrency to transmit these data blocks to the target cluster 12, completing the initial synchronization of all data. Next, the data synchronization tool 13 continuously monitors the source cluster 11, obtaining any new global version numbers added after the previous latest global version number and their corresponding data change events. Strictly following the order of the newly added global version numbers, it applies the change events to the synchronized data blocks in the target cluster 12, thereby achieving precise and orderly updates of incremental changes and ultimately ensuring that the data between the source cluster 11 and the target cluster 12 reaches a consistent state.

[0038] Next, the data synchronization method between clusters provided in the embodiments of this application will be described in detail.

[0039] Figure 2 This is a flowchart illustrating the data synchronization method between clusters in this application embodiment. Figure 1 See Figure 2 As shown, the method may include: S21: Get the data to be synchronized and the latest global version number in the source cluster.

[0040] Data to be synchronized refers to the collection of all data in the source cluster that needs to be copied to the target cluster when the synchronization task starts. For example, when synchronizing all configuration items prefixed with / app / config / , the data to be synchronized includes all keys in the source cluster that match that prefix and their corresponding latest values.

[0041] The latest global version number refers to the largest global version number corresponding to all committed data change events in the source cluster at the moment the data to be synchronized is obtained. For example, in an etcd cluster, if a client performs several operations and assigns global version numbers 100, 101, and 102, and no new changes occur, the latest global version number is 102.

[0042] In practice, data synchronization tools can obtain the data to be synchronized by initiating range query requests to the source cluster's Application Programming Interface (API). For example, they can use etcd's Range RPC call and specify a key prefix or range to retrieve a list of key-value pairs that meet the criteria in batches. Simultaneously, the data synchronization tool can obtain the current revision value of the cluster by querying the source cluster's metadata or status interface (such as etcd's / version or Status request). This value represents the latest global version number.

[0043] S22: Divide the data to be synchronized into multiple first data blocks.

[0044] In specific partitioning, the data to be synchronized can be divided into multiple first data blocks according to a preset data volume. For example, if each data block is set to not exceed 10MB, and the total amount of data to be synchronized is 95MB, it can be divided into 10 data blocks of approximately 9.5MB each, or 9 data blocks of 10MB each and 1 data block of 5MB each.

[0045] Alternatively, the data to be synchronized can be divided into multiple first data blocks according to a preset number. For example, if each data block contains 1000 key-value pairs, and the data to be synchronized contains a total of 5000 key-value pairs, it can be divided into 6 data blocks, each containing 1000 key-value pairs. As another example, if the preset number is set to 5, and the data to be synchronized contains a total of 5500 key-value pairs, it can be divided into 5 data blocks, each containing 1100 key-value pairs.

[0046] S23: Use multiple threads to transmit multiple first data blocks to the target cluster.

[0047] If the number of the first data blocks is less than or equal to the number of multiple threads, one data block can correspond to one thread, and each data block can be transmitted to the target cluster using the corresponding thread.

[0048] If the number of first data blocks is greater than the number of multiple threads, the first data blocks equal to the number of multiple threads can be transmitted to the target cluster using the corresponding thread. After the thread is used up, the next data block can be transmitted using the same thread until all the first data blocks have been transmitted to the target cluster.

[0049] Multiple threads can be pre-configured in a thread pool. The data synchronization tool starts the thread pool, allocating an independent thread for each first data block. Each thread is responsible for sending its corresponding data block to the target cluster via a batch write interface (such as etcd's Txn transaction or Put batch operation). During transmission, the thread encapsulates the key-value pairs of the data block into network requests and submits them concurrently to the corresponding nodes in the target cluster via protocols such as gRPC. All threads execute independently, and the transmission progress is monitored uniformly by the data synchronization tool. When a thread completes transmission, its resources are released or used for subsequent tasks, thereby significantly improving the throughput and efficiency of full data synchronization through parallelization.

[0050] S24: Get the first newly added global version number after the latest global version number in the source cluster, and the data change event corresponding to the first newly added global version number.

[0051] The first newly added global version number after the current latest global version number refers to the global version number corresponding to the first logically immediate new data change event generated in the source cluster after the version on which the full synchronization was based. For example, if the current latest global version number recorded when the full synchronization is completed is 102, and the source cluster subsequently receives a put key4 value4 operation, the change event generated by this operation will be assigned the global version number 103. In this case, 103 is the first newly added global version number.

[0052] The significance of obtaining the first newly added global version number is to establish a precise logical boundary between full synchronization and incremental synchronization, ensuring that incremental monitoring can start from the first new change event after the full synchronization ends, thus avoiding data loss or duplicate synchronization.

[0053] A data change event is a complete unit of change recorded when data is created, updated, or deleted in a distributed key-value store system. It typically includes metadata such as the operation type, the key of the operation, the changed value (or deletion marker), a timestamp, and the corresponding global version number. For example, in etcd, when a client executes the operation `put / config / server timeout 30`, a data change event is generated. Its content might include: operation type PUT, key ` / config / server`, value `timeout 30`, and a global version number (e.g., `revision=105`).

[0054] In practice, the data synchronization tool obtains the first newly added global version number and its corresponding data change event by initiating a Watch request to the source cluster. This request uses the latest global version number from the full synchronization records as the starting watch version (start_revision = latest global version number + 1). When new data changes occur in the source cluster, the corresponding event stream is pushed to the data synchronization tool in real time via a gRPC long connection. The global version number carried by the first event pushed is the first newly added global version number, and the event content is the corresponding data change details.

[0055] S25: Update multiple first data blocks in the target cluster according to the order of the first newly added global version number and the data change event corresponding to the first newly added global version number.

[0056] After full synchronization is complete, the source cluster will continuously generate new data changes. Each change will generate a unique and incrementing global version number, thus forming an ordered sequence of version numbers. The global version number strictly reflects the logical sequence of data changes occurring in the source cluster. Only by applying these change events in sequence can we ensure that the final state of the updated data in the target cluster is completely consistent with that of the source cluster, avoiding key-value overwriting, state rollback, or data inconsistency issues caused by out-of-order updates.

[0057] During an update, the data synchronization tool reads data change events from the event queue in global version number order and performs corresponding operations based on the event type. For PUT events, the tool sends a write request to the target cluster to update or create the corresponding data with the same key and value. For DELETE events, the tool sends a request to the target cluster to delete the specified key. This process continues to ensure that each incremental change is applied sequentially and accurately to the corresponding data blocks in the target cluster, thus maintaining eventual data consistency across multiple clusters even in concurrent update scenarios.

[0058] As described above, the data synchronization method between clusters provided in this application improves the efficiency of full data synchronization between clusters by using multi-threaded concurrent transmission of data to be synchronized in the source cluster. Furthermore, after completing the full data transmission, by obtaining the first newly added global version number after the latest global version number in the source cluster and the corresponding data change event, and strictly updating the synchronized data in the target cluster according to the version number order, even if the source cluster data changes in real time during the synchronization process, versioned event listening and sequential processing can still achieve accurate tracking and orderly synchronization of incremental data. This effectively solves the problem of missed synchronization or out-of-order writing of incremental data caused by network latency or processing interruptions in traditional synchronization schemes, ensuring that the target cluster can still maintain eventual consistency with the source cluster, thus improving the accuracy and reliability of cross-cluster data synchronization.

[0059] Furthermore, as a response to Figure 2 In a refinement and extension of the method shown, this application embodiment also provides a data synchronization method between clusters.

[0060] Figure 3 This is a flowchart illustrating the data synchronization method between clusters in this application embodiment. Figure 2 See Figure 3 As shown, the method may include: S31: Get the data to be synchronized and the latest global version number in the source cluster.

[0061] The data to be synchronized here includes not only all the data to be synchronized from the source cluster to the target cluster, but also a portion of the data from all the data to be synchronized from the source cluster to the target cluster. In other words, the data in the source cluster can be divided into multiple batches based on a preset batching strategy (such as by data volume, key range, or number of entries), and each batch is read, processed, and transmitted sequentially or in parallel as the data to be synchronized in a synchronization task.

[0062] This batch synchronization mechanism effectively controls the memory usage and network load of a single synchronization task, supports incremental synchronization of large-scale datasets, and allows incremental monitoring to capture and process new changes occurring within the data being read in batches during the synchronization process. This improves synchronization manageability while ensuring data real-time performance and consistency.

[0063] For example, when synchronizing an etcd cluster containing millions of key-value pairs, a full load at once might cause a memory overflow. The data synchronization tool can adopt a strategy of batching by key prefix range. The first batch of data to be synchronized consists of keys with prefixes from / a to / e (approximately 200,000 records). After the full and incremental monitoring of this batch is completed, the next batch of keys with prefixes from / f to / j is processed until all data with all prefix ranges is synchronized.

[0064] S32: Divide the data to be synchronized into multiple first data blocks.

[0065] The division process here is the same as the division method in step S22 of the aforementioned embodiment. Please refer to the relevant description in the aforementioned embodiment, which will not be repeated here.

[0066] S33: In the process of using multiple threads to transmit multiple first data blocks to the target cluster, when a thread in the multiple threads fails, the first data block corresponding to the failed thread is determined, and the first data block corresponding to the failed thread is transmitted to the target cluster using a normal thread in the multiple threads.

[0067] After obtaining multiple first data blocks, the data synchronization tool can transmit these first data blocks to the target cluster in one go or in batches and in parallel using multiple threads.

[0068] A momentary network interruption can cause the connection between the thread and the target cluster to be abnormally broken. The temporary unavailability of the target cluster nodes can cause the thread's write request to time out or fail. Insufficient resources on the host where the data synchronization tool is located (such as memory exhaustion or CPU overload) can cause the thread to exit unexpectedly. During long-term synchronization, external interventions such as system scheduling and process recycling can cause the thread to be forcibly terminated. All of these faults may cause the thread to be unable to complete its assigned data block transmission task, thus resulting in thread failure.

[0069] When one or more threads fail, the first data block corresponding to the failed thread needs to be determined. This data block was not synchronized due to the thread failure and therefore needs to be resynchronized.

[0070] When resynchronizing, you can use any of the normal threads other than the faulty thread, or you can restart the device where the tool is located to restore the faulty thread to normal, and then use the restored thread to synchronize the data blocks that were not synchronized before.

[0071] In some embodiments, to determine the first data block corresponding to the faulty thread, it can be pre-recorded in the log of the data synchronization tool. When synchronizing each first data block, after all threads have been allocated, the correspondence between the first data block and the thread is recorded in the log. Therefore, when a thread fails, the first data block corresponding to the faulty thread can be queried through the log.

[0072] In order to save storage space for data synchronization tools and quickly locate the range of unsynchronized data caused by thread failure, the missing number interval can be identified by comparison based on the recorded consecutive success numbers, and the corresponding first data block can be determined through the interval.

[0073] Specifically, the data synchronization tool records the numbers of the data successfully transmitted to the target cluster. These numbers are used to indicate that the data to be synchronized is transmitted in sequence.

[0074] During data synchronization, the data synchronization tool assigns a sequentially increasing number to each data block to be synchronized. This number reflects the logical order in which the data blocks are processed during the synchronization task. After each successful write of a data block to the target cluster, the data synchronization tool records the corresponding number of that data block in its local log. For example, if the data to be synchronized is divided into data blocks numbered 1 to 100, and the data synchronization tool has already recorded successful entries for numbers 1-50 and 52-100, then the missing range numbered 51 can be identified as a data block that was not successfully transmitted, thus triggering a reallocation and transmission of that data block.

[0075] Step S33 above may include: S33a: In the data synchronization tool, find the number range where a preset number of numbers are missing. The preset number is equal to or less than the number of data entries transmitted by each thread in multiple threads at one time.

[0076] S33b: Identify the data corresponding to the number range as the first data block corresponding to the faulty thread.

[0077] Because the number of data entries transmitted by a single thread in a single task is fixed and continuous, setting the preset number to be equal to or less than this number ensures that the missing number interval corresponds exactly to the transmission batch that a thread has not completed, thereby accurately locating the range of data blocks that the faulty thread is responsible for and avoiding misjudgment.

[0078] For example, suppose a data synchronization tool transmits 10 data entries per thread at a time (corresponding to a numbered interval of length 10), and all successfully transmitted records numbered 1-100 are marked. However, during inspection, it is found that the numbered interval 71-80 is completely missing. Since the preset quantity is set to 10 (equal to the number of records transmitted per thread per batch), the data synchronization tool can accurately identify the failed transmission batch corresponding to this missing interval for one thread. Based on this, the data synchronization tool identifies the 10 data entries corresponding to numbers 71 to 80 as the first data block that the faulty thread failed to complete and reassigns it to other normal threads for retransmission.

[0079] In some embodiments, in order to reduce the storage overhead of status records and simplify the missing detection logic, only the end number of each successfully transmitted data interval can be recorded. By comparing whether the end numbers of consecutive intervals are increasing and without gaps, the range of missing data intervals can be quickly identified.

[0080] Specifically, after successfully transmitting a data block (e.g., containing data numbered 1 to 10), the data synchronization tool only records the largest number in that interval, 10, as a success point. When it needs to detect missing intervals, the data synchronization tool iterates through all recorded largest numbers, checking if they form a continuous increasing sequence (e.g., 10, 20, 30…). If a jump is found in the records (e.g., the recorded largest numbers are 10 and 30, but 20 is missing), then the data blocks corresponding to numbers 11 to 20 are determined to be missing intervals.

[0081] By using this lightweight recording method based on interval boundaries, data synchronization tools can efficiently locate unsynchronized data blocks caused by thread failures while significantly reducing metadata management overhead.

[0082] For the first data block corresponding to the faulty thread, continue to use the normal thread among multiple threads for synchronization.

[0083] In the process of synchronizing multiple first data blocks through multiple threads, in high-concurrency incremental synchronization scenarios, the reception speed of data change events may be faster than the processing speed. To avoid event loss, out-of-order processing, or synchronization blockage, events can be buffered in order to a normal queue and processed sequentially by consumers. Meanwhile, in incremental listening scenarios, if network jitter or cluster anomalies cause listening interruption, abnormal events will directly block the normal event stream. To avoid normal data updates being blocked or out of order, abnormal events can be classified and written to an abnormal queue for asynchronous processing, while normal events continue to be consumed sequentially by the normal queue, thereby ensuring high availability and data consistency of the synchronization process.

[0084] S34: Write data change events into the normal queue in the order of the first newly added global version number, and write abnormal events that occur during transmission into the abnormal queue.

[0085] S35: Read data change events from the normal queue in first-in-first-out order, and update multiple first data blocks in the target cluster based on the read data change events.

[0086] When the data synchronization tool receives a data change event pushed by the source cluster through Watch, it extracts the global version number from the data change event and appends the data change event to the end of the normal queue in ascending order of the global version number.

[0087] The normal queue maintains an ordered buffer of global version numbers to ensure that the order of data change events in the normal queue is completely consistent with their occurrence logic in the source cluster. For example, after receiving data change events with global version numbers 103, 104, and 105, the corresponding data change events are written to the normal queue in the order of 103→104→105. Subsequent consumer threads will then retrieve the data change events from the head of the normal queue in sequence for processing.

[0088] Abnormal events refer to abnormal states or errors encountered during data synchronization that may hinder the continued execution of the synchronization process. For example: (1) Network connection loss: The gRPC long connection with the source cluster or the target cluster is unexpectedly interrupted; (2) MVCC compression error: The starting version number being monitored has been cleared due to the source cluster performing historical compression; (3) Write conflict: The writing of data to the target cluster fails due to version mismatch or key lock conflict; (4) Resource overrun: System-level abnormalities such as thread pool exhaustion, insufficient memory, or request rate exceeding the limit.

[0089] Abnormal events are captured, categorized, and written to an exception queue. The data synchronization tool starts a dedicated exception handling daemon thread that continuously monitors the exception queue. When an exception event is detected in the queue, the daemon thread takes appropriate recovery measures based on the event's category. For temporary exceptions such as network disconnections, the daemon thread attempts to re-establish the connection and resume listening from the breakpoint. For MVCC compression exceptions, the daemon thread queries the latest compression version number of the source cluster and re-initiates listening from that version. For write conflicts, the daemon thread may perform a limited number of retries or trigger a manual intervention alarm. After processing, the recovered listening events or reconstructed data state are reinjected into the normal queue, ensuring that exceptions are isolated and handled without affecting the continuity and order of the normal synchronization process.

[0090] In some embodiments, during long-running incremental synchronization, the source cluster may trigger MVCC compression due to storage space optimization, i.e., cleaning up historical change events earlier than a certain global version number. If the data synchronization tool is still listening to the compressed older version number at this time, it will trigger an MVCC compression anomaly, causing the listening stream to be interrupted and unable to be directly recovered from the original point. To ensure that synchronization continues after compression and maintains data integrity, the earliest available version number after compression is actively queried, and listening resumes from this version number. Simultaneously, to compensate for the loss of historical events caused by compression, the data change event corresponding to that version number and the final data state of its previous version are obtained and written as a baseline state to the normal queue. This allows the target cluster to reconstruct the intermediate data state that may have been lost during compression, achieving continuous data synchronization and eventual consistency in abnormal scenarios.

[0091] S36: Read the connection disconnection event from the exception queue; re-establish the connection with the source cluster based on the connection disconnection event; retrieve the data change event from the source cluster according to the latest global version number recorded before the connection disconnection; write the retrieved data change event and the corresponding global version number into the normal queue; if an MVCC compression exception occurs, write the data change event corresponding to the earliest recorded global version number in the source cluster and the final data status of its previous global version number into the normal queue.

[0092] The final data state describes the current state (value or existence) of a specific key-value pair at a given global version number; it is a static snapshot. For example, at global version number 105, the value of the key / app / port is 8080, which is the final data state at that version.

[0093] Data change events, on the other hand, record the specific operations that caused the state change and their metadata; they are dynamic process records. For example, a PUT / app / port 8080 operation occurred at version number 105. This event contains information such as the operation type, key, value, and version number.

[0094] Data change events are the causes, describing how the state changes, while the final data state is the effect, describing the result after the change. Listening events are used for synchronization, while the state can be used to rebuild or repair data consistency.

[0095] When the data synchronization tool reads a connection loss event from the exception queue, it first attempts to re-establish the gRPC connection. After the connection is restored, the data synchronization tool sends a Watch request to the source cluster based on the latest global version number recorded locally before the disconnection (e.g., revision=150). It listens for new change events starting from version 151 and writes the received events into the normal queue in order. If the source cluster performs MVCC compression, causing historical events before version 160 to be cleared, the data synchronization tool will capture the MVCC compression exception and query the earliest available global version number of the source cluster (e.g., compact_revision=161). Then, it obtains the data change event (i.e., the current state of the key-value pair) corresponding to that version number (161) and simultaneously obtains the final data state of its previous version (160). It writes both of these as baseline state data into the normal queue.

[0096] The data synchronization tool sequentially retrieves baseline state data (data change events of version 161 and the final data state of version 160) and subsequent incremental events from the normal queue, and then updates the target cluster one by one according to the event type. For the baseline state data, the data synchronization tool writes the final data state as a historical snapshot to ensure data continuity, and synchronizes the data change events of version 161 to the target cluster. Subsequently, the data synchronization tool continues to consume incremental events of version 162 and later from the normal queue, thereby ensuring that the data state of the target cluster and the source cluster are re-consistent at the new historical starting point after MVCC compression anomaly recovery.

[0097] At this point, the first batch of data from the source cluster has been synchronized to the target cluster. The next step is to synchronize the second batch of data, the third batch of data, and so on, from the source cluster to the target cluster.

[0098] The significance of batch synchronization lies in effectively controlling system resource consumption, avoiding memory overflow and network congestion, and supporting the interruptibility and recovery of the synchronization process by splitting large-scale datasets into multiple small batches for sequential processing.

[0099] S37: Obtain other synchronization data besides the data to be synchronized from the source cluster, as well as the latest global version number at this time.

[0100] Other synchronized data refers to the remaining subset of source cluster data after the current batch that has not yet been synchronized. The latest global version number at this point refers to the largest global version number corresponding to all committed changes in the source cluster at the time this batch of data was acquired. This latest global version number serves as the timestamp base for this batch of data, used to connect with subsequent incremental monitoring.

[0101] For example, suppose there are 10,000 key-value pairs in the source cluster that need to be synchronized. The data synchronization tool first retrieves the first batch of data to be synchronized (e.g., the first 2,000 records) using the key prefix " / a" as the range, and records the latest global version number as 500. After completing the full and incremental processing of this batch, the data synchronization tool uses the key of the last record in the previous batch as the starting point (e.g., " / a / xxx") to continue retrieving the next batch of other synchronized data (e.g., the next 2,000 records, corresponding to the key range from " / a / xxx" to " / b"), and simultaneously retrieves the latest global version number (e.g., now changed to 520). This version number reflects the latest state of the source cluster when retrieving the second batch of data. Subsequent incremental monitoring will continue to track new changes based on this version number, thereby achieving batch-wise, version-sequential, continuous synchronization of the remaining data.

[0102] For each batch of data to be synchronized, it can be retrieved from the source cluster in numerical order, or it can be retrieved from the source cluster according to data storage location, node, etc. Here, the numerical designation can refer to the original key value of the data, or it can refer to the numerical designation assigned to the data when retrieving it from the source cluster.

[0103] In some embodiments, in order to achieve full synchronization of breakpoint resume and batch acquisition of data without duplication or omission, the number of the last data in the previous batch can be recorded, and the next number of the previous batch can be used as the starting point to acquire a preset number of subsequent data, thereby ensuring that the data in each batch is continuous and accurately connected.

[0104] Specifically, each piece of data in all the data has a corresponding number, which is used to indicate that each piece of data is read sequentially.

[0105] The numbering here can be a continuous and unique sequential identifier assigned to all data to be synchronized in the source cluster, used to explicitly indicate the logical order in which each piece of data is read and processed in the full synchronization task. This numbering is typically generated based on the data's storage location, the lexicographical order of the keys, or a custom indexing rule, ensuring that the synchronization tool can read the data one by one or in batches in ascending order of the numbers. For example, if there are 100 key-value pairs in the source cluster, the data synchronization tool can assign them numbers 1 to 100 in ascending alphabetical order. During the initial synchronization, data numbered 1-20 is read, and the last piece of data is recorded as number 20. During the next synchronization, subsequent data is read starting from number 21. This achieves ordered, non-repeating batch synchronization.

[0106] Step S37 above may include: S37a: Get the first number of the last data in the data to be synchronized.

[0107] S37b: Determine the next number after the first number.

[0108] S37c: Starting from the next number in the source cluster, retrieve a preset amount of data, obtain other synchronized data, and retrieve the latest global version number at this time.

[0109] After each batch of data to be synchronized is successfully transmitted to the target cluster, the data synchronization tool will iterate through all processed data items in that batch, determine the largest number based on a preset numbering rule (such as lexicographical order of keys or storage location index), and record it as the first number. For example, if the data numbers in a batch range from 101 to 150, after the data synchronization tool has completed the transmission and confirmation of all data in that batch, it will generate and store number 150 as the first number of the last data item.

[0110] The first ID of the last piece of data in the data to be synchronized can be stored in the local persistent state file of the data synchronization tool, such as a progress log or a metadata database.

[0111] When the data synchronization tool initiates a new batch synchronization, it reads the status file to obtain the last data entry number synchronized in the previous batch (e.g., entry number 20). The data synchronization tool then calculates its next entry number (i.e., 21), using this as the starting key or query offset for this data retrieval. It then sends a range query request to the source cluster to retrieve a preset number (e.g., 20) of other synchronized data starting from entry number 21, and simultaneously records the latest global version number of the source cluster at the time of this query, thus achieving seamless integration and progress tracking between batches.

[0112] S38: Divide the other synchronized data into multiple second data blocks.

[0113] S39: Use multiple threads to transfer multiple second data blocks to the target cluster.

[0114] S310: Get the second newly added global version number after the latest global version number in the source cluster, as well as the data change event corresponding to the second newly added global version number.

[0115] S311: Modify multiple second data blocks in the target cluster according to the order of the second newly added global version number, based on the data change event corresponding to the second newly added global version number, until all data in the source cluster is transmitted to the target cluster.

[0116] Steps S38-S311 here, which is the synchronization of data for subsequent batches, are the same as steps S22-S25 in the aforementioned embodiment, which is the synchronization of the first batch of data. Please refer to the relevant descriptions above, and they will not be repeated here.

[0117] The data synchronization tool first completes the full and incremental synchronization of the first batch of data to be synchronized (e.g., numbers 1-1000), and records the number of the last data entry (1000) and the global version number at the time of synchronization completion (e.g., 500). Then, the tool executes the second batch of synchronization, starting with number 1001, acquiring the next batch of other synchronized data (numbers 1001-2000), and recording the global version number at this time (e.g., 520). This batch of data is divided into multiple second data blocks and transmitted concurrently to the target cluster via multi-threading. Afterwards, starting from version 521, it listens for events corresponding to the newly added global version number and updates the corresponding data blocks in the target cluster sequentially. This process is executed cyclically, updating the number and version number after each batch is completed, until all batches (e.g., numbers 1-10000) are fully synchronized, thus achieving segmented, concurrent, and versioned synchronization of large-scale datasets.

[0118] This concludes the description of the data synchronization method between clusters provided in the embodiments of this application.

[0119] Based on the same inventive concept, embodiments of this application also provide a data synchronization device between clusters.

[0120] The data synchronization device between the clusters is connected to the source cluster and the target cluster respectively. The source cluster records data change events and corresponding global version numbers according to time.

[0121] In some embodiments, a specific example of the data synchronization device is a data synchronization tool. Figure 4 This is a schematic diagram of the data synchronization device between clusters in an embodiment of this application. See also... Figure 4 As shown, the device may include: Module 41 is used to obtain the data to be synchronized and the latest global version number in the source cluster.

[0122] The partitioning module 42 is used to partition the data to be synchronized into multiple first data blocks.

[0123] The first synchronization module 43 is used to transmit multiple first data blocks to the target cluster using multiple threads.

[0124] Incremental module 44 is used to obtain the first newly added global version number after the latest global version number in the source cluster, as well as the data change event corresponding to the first newly added global version number.

[0125] The second synchronization module 45 is used to update multiple first data blocks in the target cluster according to the order of the first newly added global version number and based on the data change event corresponding to the first newly added global version number.

[0126] Furthermore, when the data to be synchronized is a portion of all data synchronized from the source cluster to the target cluster, the acquisition module 41 is also used to acquire other synchronized data besides the data to be synchronized and the latest global version number from the source cluster.

[0127] The partitioning module 42 is also used to partition other synchronized data to obtain multiple second data blocks.

[0128] The first synchronization module 43 is also used to transmit multiple second data blocks to the target cluster using multiple threads.

[0129] Incremental module 44 is also used to obtain the second newly added global version number after the latest global version number in the source cluster, as well as the data change event corresponding to the second newly added global version number.

[0130] The second synchronization module 45 is also used to modify multiple second data blocks in the target cluster according to the order of the second newly added global version number, based on the data change event corresponding to the second newly added global version number, until all data in the source cluster is transmitted to the target cluster.

[0131] Furthermore, each piece of data in all the data has a corresponding number, which is used to indicate that each piece of data is read sequentially. The acquisition module 41 is also used to acquire the first number of the last piece of data in the data to be synchronized.

[0132] The acquisition module 41 is specifically used to determine the next number after the first number; starting from the next number in the source cluster, it acquires a preset number of data, obtains other synchronized data, and acquires the latest global version number at this time.

[0133] Furthermore, the first synchronization module 43 is specifically used to determine the first data block corresponding to the faulty thread when a thread among the multiple threads fails, and to transmit the first data block corresponding to the faulty thread to the target cluster using a normal thread among the multiple threads.

[0134] Furthermore, the data synchronization device records the numbers of the data successfully transmitted to the target cluster. The numbers are used to indicate that the data to be synchronized is transmitted in order. Specifically, the first synchronization module 43 is used to find the number intervals in the data synchronization device where a preset number of numbers are missing. The preset number is equal to or less than the number of data items transmitted by a thread in multiple threads at one time. The data corresponding to the number interval is determined as the first data block corresponding to the faulty thread.

[0135] Furthermore, the incremental module 44 is also used to write data change events into the normal queue in the order of the first newly added global version number, and to write abnormal events that occur during transmission into the abnormal queue.

[0136] The second synchronization module 45 is specifically used to read data change events from the normal queue in a first-in-first-out order, and update multiple first data blocks in the target cluster based on the read data change events.

[0137] Furthermore, in the event of an abnormal event including a disconnection event from the source cluster, the incremental module 44 is also used to read the disconnection event from the abnormal queue; re-establish a connection with the source cluster based on the disconnection event; obtain the data change event from the source cluster according to the latest global version number recorded before the disconnection; write the obtained data change event and the corresponding global version number into the normal queue; if a multi-version concurrency control (MVCC) compression abnormality occurs, the data change event corresponding to the earliest recorded global version number in the source cluster and the final data state of its previous global version number are written into the normal queue.

[0138] It should be noted that the descriptions of the tool embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the tool embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0139] Based on the same inventive concept, this application also provides a computer device.

[0140] Figure 5 This is a schematic diagram of the structure of the computer device in an embodiment of this application. See also... Figure 5 As shown, the computer device may include: a memory 51, a processor 52, and a computer program stored on the memory 51, wherein the processor 52 executes the computer program to implement the methods described in the foregoing embodiments.

[0141] It should be noted that the description of the above computer device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the computer device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0142] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the methods described in the foregoing embodiments.

[0143] It should be noted that the description of the above computer-readable storage medium embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the computer-readable storage medium embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0144] Based on the same inventive concept, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the methods described in the foregoing embodiments.

[0145] It should be noted that the descriptions of the above computer program product embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the computer program product embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0146] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data synchronization method between clusters, characterized in that, The method is applied to a data synchronization device, which is connected to a source cluster and a target cluster respectively. The source cluster records data change events and corresponding global version numbers according to time. The method includes: Obtain the data to be synchronized and the latest global version number from the source cluster; The data to be synchronized is divided into multiple first data blocks; Multiple threads are used to transmit the multiple first data blocks to the target cluster; Obtain the first newly added global version number after the current latest global version number in the source cluster, and the data change event corresponding to the first newly added global version number; In accordance with the order of the first newly added global version number, the plurality of first data blocks in the target cluster are updated based on the data change events corresponding to the first newly added global version number.

2. The method according to claim 1, characterized in that, The data to be synchronized is a portion of all data synchronized from the source cluster to the target cluster; the method further includes: Obtain other synchronization data besides the data to be synchronized and the latest global version number from the source cluster; The other synchronized data is divided into multiple second data blocks; Multiple threads are used to transmit the multiple second data blocks to the target cluster; Obtain the second newly added global version number after the latest global version number in the source cluster, and the data change event corresponding to the second newly added global version number; Following the order of the second newly added global version number, the plurality of second data blocks in the target cluster are modified based on the data change events corresponding to the second newly added global version number until all data in the source cluster is transmitted to the target cluster.

3. The method according to claim 2, characterized in that, Each piece of data in all the data is assigned a number, which indicates that each piece of data is read sequentially; the method further includes: Obtain the first ID of the last data item in the data to be synchronized; The step of obtaining other synchronization data besides the data to be synchronized and the latest global version number from the source cluster includes: Determine the next number after the first number; Starting from the next number in the source cluster, a preset number of data is obtained, other synchronized data is obtained, and the latest global version number at this time is obtained.

4. The method according to claim 1, characterized in that, The step of using multiple threads to transmit the multiple first data blocks to the target cluster includes: When a thread among the multiple threads fails, the first data block corresponding to the failed thread is determined, and the first data block corresponding to the failed thread is transmitted to the target cluster using a normal thread among the multiple threads.

5. The method according to claim 4, characterized in that, The data synchronization device records the numbers of data successfully transmitted to the target cluster, and these numbers are used to indicate that the data to be synchronized is transmitted in sequence; the first data block corresponding to the faulty thread includes: The data synchronization device searches for a number range where a preset number of numbers are missing, where the preset number is equal to or less than the number of data entries transmitted by a thread in one of the multiple threads. The data corresponding to the number range is determined as the first data block corresponding to the faulty thread.

6. The method according to any one of claims 1 to 5, characterized in that, Before updating the plurality of first data blocks in the target cluster according to the order of the first newly added global version number and based on the data change event corresponding to the first newly added global version number, the method further includes: The data change events are written into the normal queue in the order of the first newly added global version number, and the abnormal events that occur during the transmission process are written into the abnormal queue. The step of updating the plurality of first data blocks in the target cluster according to the order of the first newly added global version number, based on the data change event corresponding to the first newly added global version number, includes: The data change events are read from the normal queue in a first-in-first-out order, and the plurality of first data blocks in the target cluster are updated based on the read data change events.

7. The method according to claim 6, characterized in that, The abnormal events include events that disconnect from the source cluster; The method further includes: Read the connection disconnection event from the exception queue; Based on the connection disconnection event, a new connection is established with the source cluster; Data change events are retrieved from the source cluster based on the latest global version number recorded before the connection was disconnected; Write the acquired data change event and the corresponding global version number into the normal queue; If a Multi-Version Concurrency Control (MVCC) compression anomaly occurs, the data change event corresponding to the earliest global version number currently recorded in the source cluster and the final data status of its previous global version number are written into the normal queue.

8. A data synchronization device for inter-cluster data, characterized in that, The data synchronization device is connected to both the source cluster and the target cluster. The source cluster records data change events and their corresponding global version numbers according to time. The data synchronization device includes: The acquisition module is used to acquire the data to be synchronized and the latest global version number in the source cluster; The partitioning module is used to partition the data to be synchronized into multiple first data blocks; The first synchronization module is used to transmit the plurality of first data blocks to the target cluster using multiple threads; The incremental module is used to obtain the first newly added global version number after the current latest global version number in the source cluster, as well as the data change event corresponding to the first newly added global version number. The second synchronization module is used to update the plurality of first data blocks in the target cluster according to the order of the first newly added global version number and based on the data change event corresponding to the first newly added global version number.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images