Data synchronization methods, devices, equipment, media, and software products based on OTS

By using an OTS-based data synchronization method, which utilizes configuration file parsing and concurrent thread pools to process data, combined with recursive partitioning and retry mechanisms and verification mechanisms, the dual requirements of cross-regional data synchronization and verification are solved, achieving efficient and reliable data consistency and accuracy.

CN123115154APending Publication Date: 2026-11-17BEIJING CHINA POWER INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611165958.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-03
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies have not yet been able to effectively address the dual requirements of cross-regional OTS data synchronization and verification, especially in regional load architectures where the accuracy and robustness of data synchronization are insufficient.

Method used

The data synchronization method based on OTS is adopted. The synchronization task parameters are parsed by obtaining the configuration file, the target data is read by the interface of the source OTS and stored in the memory queue, the target table is written asynchronously using a concurrent thread pool, and recursive splitting and retry are performed when the write fails. Finally, the data volume and content are verified.

Benefits of technology

It achieves efficient and reliable cross-regional data synchronization, ensuring data consistency and availability, reducing network transmission volume and synchronization time, and improving the system's flexibility and robustness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115154A_ABST
    Figure CN123115154A_ABST
Patent Text Reader

Abstract

This invention provides a data synchronization method, apparatus, device, medium, and program product based on OTS (Optical Time Synchronization), comprising: obtaining a configuration file and parsing it to obtain synchronization task parameters; reading target data that meets filtering conditions from a source table using a first preset interface of the source OTS; asynchronously obtaining the target data using a thread pool containing multiple concurrent worker threads; writing the target data into a target table using a second preset interface of the target OTS; in response to a write failure, performing a recursive segmentation and retry operation on the target data until the number of write failures for a single row of segmented data exceeds a threshold, generating a synchronization completion status signal; and in response to receiving the synchronization completion status signal, performing data volume verification and content sampling verification on the source and target tables to obtain verification results. This provides an efficient and reliable OTS data synchronization method, thereby meeting the accuracy and robustness requirements of regional load balancing for data synchronization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and in particular to a data synchronization method, apparatus, device, medium and program product based on OTS. Background Technology

[0002] As State Grid's "State Grid Online" business evolves towards a regional load architecture, in order to achieve business collaboration and disaster recovery capabilities between the first-location data center and the second-location data center, it is necessary to efficiently and completely synchronize the data from the first-location side to the second-location side, and to perform strict data verification after synchronization to ensure the consistency and availability of data at both ends.

[0003] However, there is currently no mature and feasible method in the industry that can simultaneously meet the dual requirements of cross-regional OTS data synchronization and verification. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data synchronization method, apparatus, device, medium and program product based on OTS.

[0005] To achieve the above objectives, the first aspect of the present invention provides a data synchronization method based on OTS, comprising:

[0006] Obtain the configuration file, parse the configuration file to obtain the synchronization task parameters, the synchronization task parameters include: the source table of the source OTS, the target table of the target OTS, and the filtering conditions; Using the first preset interface of the source end OTS, target data that meets the filtering conditions is read from the source table, and the target data is stored in the memory queue; Using a thread pool containing multiple concurrent worker threads, the target data is asynchronously retrieved from the memory queue, and the target data is written into the target table using the second preset interface of the target end OTS to obtain the data writing result; In response to determining that the data writing result is a writing failure, a recursive segmentation and retry operation is performed on the target data until the number of writing failures of the segmented single row data is greater than the number threshold, and a synchronization completion status signal is generated. In response to receiving the synchronization completion status signal, data volume verification and content sampling verification are performed on the source table and the target table to obtain the verification results.

[0007] In some exemplary embodiments, the step of performing a recursive segmentation and retry operation on the target data in response to determining that the data write result is a write failure, until the number of write failures of the segmented single row data is greater than a threshold, and generating a synchronization completion status signal, includes: The target data is divided into a first sub-batch and a second sub-batch based on the number of rows. Call the second preset interface to retry writing the first sub-batch and the second sub-batch respectively; If either the first sub-batch or the second sub-batch fails to write, it is split into two halves according to the number of rows as a new failed batch and recursively retried until the split batch contains only a single row of data. In response to the number of write failures of the single row of data exceeding a threshold, the synchronization completion status signal is generated.

[0008] In some exemplary embodiments, the step of performing data volume verification on the source table and the target table includes: The total number of records in the source table that meet the filtering conditions and the total number of records in the target table that meet the filtering conditions are counted using a paginated scanning method. Compare the total number of records in the source table with the total number of records in the target table to see if they are the same, and calculate the difference value and the difference percentage.

[0009] In some exemplary embodiments, content sampling verification is performed on the source table and the target table, including: Scan each page of the source table according to the filtering conditions and collect a preset number of row records as a sample set. For each sample record in the sample set, a point query is performed in the target table using the primary key of the sample record as the query condition to obtain the corresponding target row data. The consistency between the sample record and the target row data is compared field by field.

[0010] In some exemplary embodiments, during the process of reading target data that meets the filtering conditions from the source table, the method further includes: Record the primary key of the next scan after each read operation is completed as the synchronization point, and persist the synchronization point. In response to a restart after a synchronization task is interrupted, the synchronization point is read, and the operation of reading the target data that meets the filtering conditions from the source table is continued with the synchronization point as the starting position.

[0011] In some exemplary embodiments, the filtering conditions include time range conditions and / or attribute column content keyword conditions; The step of reading target data that meets the filtering conditions from the source table using the first preset interface of the source-end OTS includes: When the first preset interface is called, a column condition containing the time range condition and / or the attribute column content keyword condition is constructed in the interface call parameters, and the filtering condition is passed to the source OTS server so that the data filtering is completed at the source OTS server and only the data rows that meet the filtering condition are returned as the target data.

[0012] A second aspect of the present invention provides a data synchronization device based on OTS, comprising: The synchronization task parameter determination module is configured to obtain a configuration file, parse the configuration file to obtain synchronization task parameters, and the synchronization task parameters include: the source table of the source OTS, the target table of the target OTS, and filtering conditions. The target data reading module is configured to use the first preset interface of the source end OTS to read target data that meets the filtering conditions from the source table and store the target data in a memory queue. The target data writing module is configured to asynchronously retrieve the target data from the memory queue using a thread pool containing multiple concurrent worker threads, and write the target data into the target table using the second preset interface of the target end OTS to obtain the data writing result; The recursive segmentation retry module is configured to perform a recursive segmentation retry operation on the target data in response to determining that the data writing result is a writing failure, until the number of writing failures of the segmented single row data is greater than the number threshold, and generate a synchronization completion status signal. The data verification module is configured to perform data volume verification and content sampling verification on the source table and the target table in response to receiving the synchronization completion status signal, and obtain the verification result.

[0013] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described in the first aspect.

[0014] A fourth aspect of the present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described in the first aspect.

[0015] The fifth aspect of the present invention provides a computer program product including computer program instructions that, when the computer program instructions are run on a computer, cause the computer to perform the method as described in the first aspect.

[0016] As can be seen from the above, the data synchronization method, apparatus, device, medium, and program product based on OTS provided by the present invention includes: obtaining a configuration file; parsing the configuration file to obtain synchronization task parameters, the synchronization task parameters including: a source table of the source OTS, a target table of the target OTS, and filtering conditions; using a first preset interface of the source OTS to read target data that meets the filtering conditions from the source table and storing the target data in a memory queue; using a thread pool containing multiple concurrent worker threads to asynchronously obtain the target data from the memory queue, and using a second preset interface of the target OTS to write the target data into the target table, obtaining a data writing result; in response to determining that the data writing result is a write failure, performing a recursive segmentation and retry operation on the target data until the number of write failures of the segmented single row data is greater than a threshold, generating a synchronization completion status signal; in response to receiving the synchronization completion status signal, performing data volume verification and content sampling verification on the source table and the target table, obtaining a verification result. This provides an efficient and reliable OTS data synchronization method, thereby meeting the accuracy and robustness requirements of regional load balancing for data synchronization. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 The diagram illustrates a flowchart of an exemplary OTS-based data synchronization method provided by an embodiment of the present invention.

[0019] Figure 2 A schematic diagram of an exemplary OTS-based data synchronization device provided in an embodiment of the present invention is shown.

[0020] Figure 3 A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the principles and spirit of the invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make the invention more thorough and complete, and to fully convey the scope of the invention to those skilled in the art.

[0023] It is important to understand in this article that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and that any naming is for distinction only and has no limiting meaning.

[0024] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in the embodiments of this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "a" or "an" preceding an element does not exclude the existence of multiple such elements.

[0025] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0026] With the rapid development of cloud computing and big data technologies, enterprises' needs for data storage and synchronization are becoming increasingly complex. Against the backdrop of the State Grid's "Online State Grid" and other businesses evolving towards regional load architectures, cross-regional data synchronization has become a key infrastructure component for achieving business collaboration and disaster recovery capabilities among multiple data centers.

[0027] Alibaba Cloud Table Store (OTS) is a NoSQL data storage service built on cloud computing technology, providing storage and management capabilities for massive amounts of structured data. Its data model is centered around two-dimensional tables. Unlike traditional relational databases, Table Store tables are sparse, with each row potentially having different columns. Attribute columns can be dynamically added or removed, and a strict schema definition is not required when creating tables. In cross-regional business scenarios, users need to synchronize OTS data from one instance in one region to another to meet business requirements such as disaster recovery, data backup, and accelerated access from the nearest location.

[0028] Currently, there are several OTS cross-regional data synchronization solutions in the industry, but these solutions have limitations in terms of flexibility, autonomy and controllability, data verification and anomaly handling, and have not yet formed a mature and comprehensive solution that can meet the dual requirements of data synchronization accuracy and robustness for regional load balancing.

[0029] Figure 1 The diagram illustrates a flowchart of an exemplary OTS-based data synchronization method provided by an embodiment of the present invention.

[0030] Data synchronization methods based on OTS include: S101. Obtain the configuration file and parse the configuration file to obtain the synchronization task parameters. The synchronization task parameters include: the source table of the source OTS, the target table of the target OTS, and the filtering conditions.

[0031] The configuration file can be pre-written according to actual business needs.

[0032] As an example, the configuration file is a text file in JSON, YAML, XML, or other formats that have the ability to describe structured data.

[0033] The synchronization task parameters are obtained by parsing the configuration file, including: The contents of each field in the configuration file are read using a JSON parser to extract all the parameter information required for the synchronization task.

[0034] As an example, the source OTS refers to an Alibaba Cloud Table Store (OTS) service instance deployed on the data sender, such as a data center at a first location. The source table of the source OTS refers to the source data table to be synchronized in this instance, i.e., the data source for this synchronization operation. The target OTS refers to an Alibaba Cloud Table Store service instance deployed on the data receiver, such as a data center at a second location. The target table of the target OTS refers to the target data table in this instance used to receive the synchronized data, i.e., the destination where the data is written in this synchronization operation.

[0035] As an example, the source table of the source OTS and the target table of the target OTS can have the same table name or different table names, depending on the mapping relationship in the configuration file.

[0036] Through the above embodiments, the present invention realizes configuration-driven synchronization task parameters, which allows the synchronization range, synchronization objects, and synchronization rules to be flexibly adjusted by modifying the configuration file without recompiling or deploying the program code.

[0037] The filtering conditions are used to limit the data range of this synchronization operation.

[0038] S102. Using the first preset interface of the source end OTS, read the target data that meets the filtering conditions from the source table and store the target data in the memory queue.

[0039] As an example, the first preset interface of the source OTS is the get_range interface provided by Alibaba Cloud Table Storage. This interface supports pagination scanning according to a specified primary key range, and each call returns a batch of data rows and the next row's primary key for the next call. First, a network connection is established with the source OTS based on the instance information of the source OTS, and the OTS client is initialized. Then, interface call parameters are constructed according to the filtering conditions, and the first preset interface is called repeatedly to read data rows that meet the filtering conditions from the source table in batches. In each round of calls, the next row's primary key returned by the previous call is passed as the starting primary key for the current call, and this process is repeated until all data rows that meet the filtering conditions have been read.

[0040] The above embodiments enable batch and ordered reading of massive amounts of data from the source table, avoiding the risk of memory overflow caused by fetching the entire table at once. Each set of data rows returned by the interface call is treated as an independent data batch, which serves as the carrier unit for the target data.

[0041] As an example, after each batch of target data is read, it is stored in a memory queue. This memory queue acts as a buffer between the producer and consumer, employing a first-in, first-out (FIFO) data structure to temporarily store data batches that have been read but not yet written to the target. The data reading operation at the source (producer) and the data writing operation at the target (consumer) are decoupled through this memory queue. The read operation simply places the data batch at the tail of the queue to continue reading the next batch without waiting for the write operation to complete; the write operation asynchronously retrieves data batches from the head of the queue and writes them.

[0042] The above embodiments effectively smooth out the read / write rate mismatch caused by latency fluctuations in cross-regional network environments, preventing read threads from being suspended for extended periods due to temporary blocking at the write end, thereby improving overall synchronization efficiency. The memory queue has a preset capacity limit; when the queue is full, read operations will block and wait until the queue releases storage space, thus preventing the unlimited consumption of memory resources.

[0043] In some exemplary embodiments, the filtering conditions include time range conditions and / or attribute column content keyword conditions; The step of reading target data that meets the filtering conditions from the source table using the first preset interface of the source-end OTS includes: When the first preset interface is called, a column condition containing the time range condition and / or the attribute column content keyword condition is constructed in the interface call parameters, and the filtering condition is passed to the source OTS server so that the data filtering is completed at the source OTS server and only the data rows that meet the filtering condition are returned as the target data.

[0044] As an example, the time range condition limits the synchronization window using the start time and end time fields defined in the configuration file. The system only synchronizes data rows that have changed within this time window. When the incremental synchronization flag is set to a valid value in the configuration file, the system automatically enables the incremental synchronization mode based on the time range condition. The attribute column content keyword condition is used to further filter data on non-primary key attribute column dimensions, supporting at least one matching method among string inclusion relation matching, prefix matching, or exact matching. For example, it can be configured to synchronize only data rows whose "business type" attribute column contains the keyword "order," thereby achieving refined data distribution by business dimension.

[0045] As an example, by including this condition in the interface call parameters, the source OTS server can directly perform data filtering at the storage layer when performing data scanning. This means that only data rows that meet the filtering conditions are returned on the server side; data rows that do not meet the conditions are filtered out on the server side and will not be retrieved to the client. After calling the first preset interface, the first preset interface only returns data rows that meet the filtering conditions; these data rows are the target data.

[0046] The above embodiments fully utilize the computing power of the OTS server, performing data filtering on the server side. This avoids the network bandwidth waste and memory overhead caused by pulling all data to the client for filtering. Compared to existing solutions that only support primary key range filtering, the filtering mechanism of this invention significantly improves the flexibility and efficiency of synchronization, enabling users to flexibly configure the subset of data to be synchronized according to different business scenarios.

[0047] In some exemplary embodiments, during the process of reading target data that meets the filtering conditions from the source table, the method further includes: Record the primary key of the next scan after each read operation is completed as the synchronization point, and persist the synchronization point. In response to a restart after a synchronization task is interrupted, the synchronization point is read, and the operation of reading the target data that meets the filtering conditions from the source table is continued with the synchronization point as the starting position.

[0048] In the process of reading target data that meets the filtering conditions from the source table, this invention also provides a breakpoint resumption mechanism. When reading target data that meets the filtering conditions from the source table in batches by repeatedly calling the first preset interface of the source OTS, after each round of calls is completed and a batch of target data is successfully returned, the return result of the first preset interface carries a next-row primary key for the next call. This primary key identifies the currently read data position; all data rows with primary keys less than this value have been successfully read, and subsequent reads should continue scanning from this primary key. This invention records this next-row primary key as a synchronization point and persistently stores the synchronization point in a storage medium, such as a local file system, database, or distributed configuration center. During normal synchronization, after each successful completion of a round of read operations, the persistent synchronization point is updated to ensure that the storage medium always stores the position of the most recently completed read.

[0049] When a synchronization task is unexpectedly interrupted due to network outages, temporary unavailability of the target terminal, abnormal program exit, or other reasons and needs to be resumed, this invention supports manual or automatic triggering of the breakpoint resumption process. In response to the restart of the synchronization task after an interruption, the system first reads the most recently persisted synchronization point from the persistent storage medium, i.e., the primary key for the next scan start recorded before the interruption. Then, using this synchronization point as the starting primary key, the first preset interface is called again to continue reading target data that meets the filtering conditions from the source table. Since the synchronization point identifies the data position that was read before the interruption, the system continues to perform the reading operation from this point, ensuring that no data rows that were not read before the interruption are missed, nor are data rows that were successfully read before the interruption and may have been written to the target terminal repeatedly.

[0050] Through the above embodiments, the synchronization task of the present invention does not need to start full synchronization from scratch due to unexpected interruption, which significantly reduces the amount of repetitive work and synchronization time, and improves the robustness and maintainability of cross-regional long-distance synchronization tasks.

[0051] S103. Using a thread pool containing multiple concurrent worker threads, asynchronously obtain the target data from the memory queue, and use the second preset interface of the target end OTS to write the target data into the target table to obtain the data writing result.

[0052] In this invention, the target end OTS refers to an Alibaba Cloud Table Storage Service instance deployed at the data receiver, such as a second-location data center, and the target table refers to the data table in that instance used to receive synchronized data. This invention employs a thread pool containing multiple concurrent worker threads as the consumption engine for the write end. The number of concurrent worker threads in the thread pool can be configured through the configuration file. These worker threads are running simultaneously, each independently and asynchronously retrieving batches of data from the memory queue. Here, "asynchronous" means that each worker thread in the thread pool does not need to wait for a batch to finish writing before retrieving the next batch from the queue; multiple threads can execute write operations in parallel, and the write operations do not block the read operations at the source end in step S102.

[0053] Through the above embodiments, the present invention can process the writing of multiple data batches in parallel, significantly improving the utilization rate of cross-regional network bandwidth and avoiding the synchronization efficiency bottleneck caused by network latency, i.e., the time required for data to be transmitted from the sender to the receiver, during single-threaded writing. The memory queue here acts as a buffer channel in the producer-consumer pattern. In step S102 (producer), the data batch is placed at the tail of the queue, and the thread pool (consumer) in this step retrieves the data batch from the head of the queue. When there is no data in the memory queue, the worker thread enters a waiting state until a new data batch is stored in the queue and is then awakened. Through the combination of the above queue buffering and asynchronous consumption, dynamic adaptation of the read and write rates is achieved, effectively smoothing out the instantaneous latency caused by network fluctuations.

[0054] For each batch of data retrieved, the worker thread calls the second preset interface of the target OTS to write each data row in the batch into the target table. The second preset interface is the `batch_write_row` interface provided by Alibaba Cloud Table Storage, which supports batch writing of multiple rows of data in a single call. When calling the `batch_write_row` interface, the worker thread passes all data rows in the current batch as parameters. After receiving the request, the target OTS server writes the data rows into the target table in batches. After writing is complete, the second preset interface returns the execution result of this write operation, which serves as the data write result. The data write result includes at least: whether the write operation was entirely successful, which rows failed to write when partially successful, and the reason for the failure. This data write result is used to trigger different branch logics in subsequent steps. When the data write result indicates that all data was written successfully, the next batch is processed directly; when the data write result indicates that some data failed, the recursive splitting and retrying operation in step S104 is triggered.

[0055] The above embodiments enable the system to instantly perceive the results of each write operation, providing a foundation for subsequent fault-tolerant processing.

[0056] S104. In response to determining that the data writing result is a writing failure, a recursive segmentation and retry operation is performed on the target data until the number of writing failures of the segmented single row data is greater than the number threshold, and a synchronization completion status signal is generated.

[0057] The data writing result refers to the execution result returned by the second preset interface after calling the second preset interface of the target OTS in step S103 to write each data batch. A write failure means that the data writing result indicates that at least some data rows in the current data batch failed to be successfully written to the target table of the target OTS. In cross-regional long-distance network environments, partial failures in batch write operations are common due to factors such as network packet loss, latency fluctuations, or temporary rate limiting by the target OTS server. To address these write failures, this invention does not adopt the simple strategy of discarding the entire batch of data or retrying the entire batch from scratch. Instead, it performs a recursive segmentation and retry operation on the failed data. The recursive segmentation and retry operation refers to dividing a large failed dataset layer by layer into smaller data subsets, retrying the writing of each subset, thereby gradually converging the failure range to the small amount of data that truly has problems. Successfully written portions are not retransmitted. During each recursive segmentation and retry process, if a sub-batch is successfully written, further segmentation of that sub-batch is terminated; if the sub-batch still fails to write, segmentation continues. This recursive process continues until the segmented batch contains only a single row of data. For each split row of data, if the number of write failures exceeds a preset threshold (e.g., 3 times), retries for that row will stop, indicating that the row may fail to write due to hard errors such as incompatible primary key formats or excessively long fields. After the recursive splitting and retrying operation is completed, a synchronization completion status signal is generated regardless of whether all data has been successfully written, to notify the subsequent verification process to begin.

[0058] The above embodiments ensure that the synchronization process is not blocked by a very small number of abnormal data, while minimizing network retransmission overhead.

[0059] In some exemplary embodiments, the step of performing a recursive segmentation and retry operation on the target data in response to determining that the data write result is a write failure, until the number of write failures of the segmented single row data is greater than a threshold, and generating a synchronization completion status signal, includes: The target data is divided into a first sub-batch and a second sub-batch based on the number of rows. Call the second preset interface to retry writing the first sub-batch and the second sub-batch respectively; If either the first sub-batch or the second sub-batch fails to write, it is split into two halves according to the number of rows as a new failed batch and recursively retried until the split batch contains only a single row of data. In response to the number of write failures of the single row of data exceeding a threshold, the synchronization completion status signal is generated.

[0060] As an example, when it is determined that a batch of data, i.e., a batch of target data, has failed to be written, the target data is split in half according to the number of rows, into a first sub-batch and a second sub-batch. For example, if the target data contains 20,000 rows, the first and second sub-batches after splitting each contain approximately 10,000 rows. Then, the second preset interface is called to retry writing the first and second sub-batches respectively. The second preset interface returns the writing result of each sub-batch after performing the write operation. If either the first or second sub-batch fails to be written again, that sub-batch is treated as a new failed batch, and the process of splitting it in half according to the number of rows and recursively retrying continues in the same manner. For example, if the first sub-batch of 10,000 rows fails to be written, it is split in half again into two smaller sub-batches of 5,000 rows each, and the writing is retried for each. This recursive splitting continues until the split batch contains only a single row of data. During the recursive segmentation and retry process described above, if a call to the second preset interface returns success for writing a sub-batch, it indicates that all data rows in that sub-batch have been successfully written to the target table. This sub-batch will not participate in further segmentation and retries; that is, successfully written sub-batch will not be transmitted repeatedly. Simultaneously, before each retry call to the second preset interface, a waiting time is calculated based on the accumulated number of retries for the current batch using an exponential backoff strategy. For example, the baseline interval for exponential backoff is 1 second, with a maximum interval not exceeding 10 seconds. The retry is then performed after the specified waiting time to avoid frequent retries exacerbating system load during network congestion or server overload.

[0061] When the recursive partitioning and retrying operation reaches a point where a branch has been partitioned into a single row of data, if the write operation to that single row of data still fails when calling the second preset interface, the failure count of that single row of data is recorded, and the single row of data is retried. When the number of write failures for that single row of data exceeds a preset threshold, such as 3 times, the retry operation for that single row of data is stopped, the single row of data is recorded in the failure log, and the next batch of data is processed. After all the recursive partitioning and retrying operations are completed, that is, all batches of data have been processed, the batches that were successfully written have been written to the target table, and the single rows of data with failure counts exceeding the threshold have been recorded in the failure log, the synchronization completion status signal is generated. The synchronization completion status signal is used to trigger the verification process in step S105.

[0062] Through the above embodiments, the present invention only needs to retransmit a small number of truly problematic data rows, rather than retransmitting a large batch containing tens of thousands of rows as a whole. This effectively reduces the amount of data transmitted across regional networks, while ensuring that the synchronization process is not blocked by a small amount of abnormal data, and significantly improves the efficiency and robustness of data synchronization in cross-regional long-distance network environments.

[0063] S105. In response to receiving the synchronization completion status signal, perform data volume verification and content sampling verification on the source table and the target table to obtain the verification result.

[0064] In response to receiving a synchronization completion status signal, this invention initiates a two-level data consistency verification mechanism: data volume verification and content sampling verification. The data volume verification performs a macro-level comparison of the source and target data. Specifically, it counts the total number of records in the source table that meet the filtering conditions and the total number of records in the target table that meet the same filtering conditions. The consistency between the two is used to determine if there is significant overall data loss or redundancy. The content sampling verification performs a precise comparison of the two sets of data from a micro-level content perspective. A certain number of rows are sampled from the source table according to the filtering conditions. For each sample record, the primary key of that sample record is used as the query condition to find the corresponding row data in the target table. Then, the content of the two sets of data is compared field by field to determine consistency, including the comparison of the primary key value and the values ​​of each attribute column.

[0065] Through the above embodiments, this invention verifies the consistency of synchronized data from two dimensions: macro-level quantity and micro-level content. This avoids the excessively high network transmission and computational costs of full-table content comparison in cross-regional scenarios, while also enabling the detection of data inconsistencies with configurable confidence levels. The verification results include at least the percentage difference in data volume verification, the consistency rate of content sampling verification, and details of specific records with discrepancies. This invention provides the verification results as the final output for operation and maintenance personnel or automated operation and maintenance systems to review and process, thereby achieving closed-loop verification of synchronization quality and meeting the data quality acceptance requirements of regional load architectures.

[0066] In some exemplary embodiments, the step of performing data volume verification on the source table and the target table includes: The total number of records in the source table that meet the filtering conditions and the total number of records in the target table that meet the filtering conditions are counted using a paginated scanning method. Compare the total number of records in the source table with the total number of records in the target table to see if they are the same, and calculate the difference value and the difference percentage.

[0067] The filtering conditions are consistent with those in the synchronization task parameters parsed in step S101, ensuring precise alignment between the data range for data volume verification and the data range for synchronization operations. The pagination scanning method refers to calling the OTS data scanning interface during the statistical process to read a certain number of data rows, for example, 1000 rows per page. The client counts and accumulates the read rows, then continues calling the interface with the starting primary key of the next page for the next round of scanning, until all data rows meeting the filtering conditions have been read and counted. Using pagination scanning instead of loading the entire table at once avoids the risk of memory overflow due to excessive data volume, while ensuring stable execution of the statistical operation, unaffected by the actual data size of the source and target tables. After the statistical analysis is completed, the total number of records in the source table and the total number of records in the target table are obtained. Then, the total number of records in the source table and the total number of records in the target table are compared. If they are consistent, it indicates that the data volume at both ends is consistent at the macro level, with no significant row loss or redundancy. If they are inconsistent, the difference value and percentage are calculated.

[0068] The above embodiments allow for the rapid quantification of the degree of deviation in data volume between the two ends. Furthermore, statistical results such as the total number of records in the source table, the total number of records in the target table, the difference value, the difference percentage, and the verification time are written into a database table for subsequent auditing and traceability. If the difference percentage exceeds a preset threshold, such as 1%, an alarm is issued, prompting maintenance personnel to intervene and verify. Through the above data volume verification, this invention can quickly determine whether there is significant overall data loss or redundancy after synchronization, ensuring data consistency from a macro-level quantitative perspective.

[0069] In some exemplary embodiments, content sampling verification is performed on the source table and the target table, including: Scan each page of the source table according to the filtering conditions and collect a preset number of row records as a sample set. For each sample record in the sample set, a point query is performed in the target table using the primary key of the sample record as the query condition to obtain the corresponding target row data. The consistency between the sample record and the target row data is compared field by field.

[0070] The filtering conditions are consistent with those used in the synchronization operation, ensuring precise alignment between the data range for sampling verification and the synchronization range. This means that content verification is performed only on the data rows that are actually synchronized, avoiding false alarms due to inconsistencies in range. The page-by-page scanning is implemented by calling the OTS data scanning interface to read data rows from the source table in batches according to the filtering conditions. Each time, one page is read, for example, 1000 rows per page. The read row records are added to the sample set, and the scanning continues from the starting primary key of the next page, accumulating the collection until the total number of collected row records reaches the preset sample size, for example, 1000. The preset sample size can be configured through a configuration file, allowing users to flexibly adjust this number based on their overall requirements for verification confidence and time consumption. During the sample set collection process, users can further configure content filtering conditions, i.e., perform targeted sampling for specific attribute columns, such as only extracting data rows whose "business type" attribute column contains the keyword "order" as samples. This concentrates verification resources on key business data, improving the targeting and efficiency of the verification.

[0071] The point lookup, or precise primary key lookup, differs from the range scan in step S102. Point lookup directly locates the row of data in the OTS index on the target side by specifying the complete primary key value. Its query speed is significantly faster than range scan, making it particularly suitable for scenarios requiring precise retrieval of a small number of specific rows. Since the primary key of each sample record in the sample set uniquely identifies a row of data in the source table, the row of data obtained by point lookup using the same primary key in the target table is the synchronized data corresponding to the source table sample on the target side.

[0072] In the OTS sparse table data model, each row of the data table can have a different set of attribute columns, the number of attribute columns in different rows can vary, and attribute columns can be dynamically added or removed. Therefore, the comparison should include at least the following three aspects: comparing whether the primary key values ​​are consistent (i.e., whether the primary keys of the row in the source table and the target table are exactly the same); comparing whether the values ​​of each attribute column are consistent (i.e., for each attribute column existing in the source table, check whether the corresponding attribute column exists in the target table and has the same value); and determining whether the attribute column existing in the source table exists in the corresponding row in the target table (i.e., checking whether there are any missing attribute columns in the target table). If, after field-by-field comparison, inconsistent field values, missing fields, or missing entire rows are found (i.e., the row data corresponding to the primary key cannot be found in the target table), the difference details should be recorded, including at least the difference type, the difference field name, the field value in the source table, the field value in the target table, and the primary key information, and stored in the audit results table for subsequent manual verification or automated repair.

[0073] Through the above embodiments, the present invention achieves accurate comparison of synchronized data content with configurable confidence levels while controlling verification costs.

[0074] It should be noted that the method of this embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this embodiment, and the multiple devices will interact with each other to complete the method described.

[0075] It should be noted that the above description describes some embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] Based on the same inventive concept, and corresponding to any of the above embodiments, the present invention also provides a data synchronization device based on OTS.

[0077] Figure 2 A schematic diagram of an exemplary OTS-based data synchronization device provided in an embodiment of the present invention is shown.

[0078] OTS-based data synchronization devices include: The synchronization task parameter determination module 11 is configured to obtain a configuration file, parse the configuration file to obtain synchronization task parameters, and the synchronization task parameters include: the source table of the source OTS, the target table of the target OTS, and filtering conditions. The target data reading module 12 is configured to use the first preset interface of the source end OTS to read target data that meets the filtering conditions from the source table and store the target data in a memory queue. The target data writing module 13 is configured to asynchronously obtain the target data from the memory queue using a thread pool containing multiple concurrent worker threads, and write the target data into the target table using the second preset interface of the target end OTS to obtain the data writing result; The recursive segmentation retry module 14 is configured to perform a recursive segmentation retry operation on the target data in response to determining that the data writing result is a writing failure, until the number of writing failures of the segmented single row data is greater than the number threshold, and generate a synchronization completion status signal. The data verification module 15 is configured to perform data volume verification and content sampling verification on the source table and the target table in response to receiving the synchronization completion status signal, and obtain the verification result.

[0079] In some exemplary embodiments, the recursive segmentation retry module 14 is configured as follows: The target data is divided into a first sub-batch and a second sub-batch based on the number of rows. Call the second preset interface to retry writing the first sub-batch and the second sub-batch respectively; If either the first sub-batch or the second sub-batch fails to write, it is split into two halves according to the number of rows as a new failed batch and recursively retried until the split batch contains only a single row of data. In response to the number of write failures of the single row of data exceeding a threshold, the synchronization completion status signal is generated.

[0080] In some exemplary embodiments, the data verification module 15 is configured as follows: The total number of records in the source table that meet the filtering conditions and the total number of records in the target table that meet the filtering conditions are counted using a paginated scanning method. Compare the total number of records in the source table with the total number of records in the target table to see if they are the same, and calculate the difference value and the difference percentage.

[0081] In some exemplary embodiments, the data verification module 15 is configured as follows: Scan each page of the source table according to the filtering conditions and collect a preset number of row records as a sample set. For each sample record in the sample set, a point query is performed in the target table using the primary key of the sample record as the query condition to obtain the corresponding target row data. The consistency between the sample record and the target row data is compared field by field.

[0082] In some exemplary embodiments, the target data reading module 12 is configured to: Record the primary key of the next scan after each read operation is completed as the synchronization point, and persist the synchronization point. In response to a restart after a synchronization task is interrupted, the synchronization point is read, and the operation of reading the target data that meets the filtering conditions from the source table is continued with the synchronization point as the starting position.

[0083] In some exemplary embodiments, the filtering conditions include time range conditions and / or attribute column content keyword conditions; Target data reading module 12 is configured as follows: When the first preset interface is called, a column condition containing the time range condition and / or the attribute column content keyword condition is constructed in the interface call parameters, and the filtering condition is passed to the source OTS server so that the data filtering is completed at the source OTS server and only the data rows that meet the filtering condition are returned as the target data.

[0084] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this invention, the functions of each module can be implemented in one or more software and / or hardware components.

[0085] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0086] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0087] Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0088] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0089] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0090] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0091] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0092] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0093] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0094] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0095] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0096] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0097] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0098] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to perform the method. Corresponding to the execution entity for each step in each embodiment of the method, the processor performing the corresponding step may belong to the corresponding execution entity.

[0099] The computer program products of the above embodiments are used to cause a processor to execute the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0100] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, the present invention can be specifically implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0101] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0102] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0103] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0104] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0106] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0107] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0108] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0110] It should be noted that although several modules or units of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0111] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of the invention as described above, which are not provided in the details for the sake of brevity.

[0112] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of the invention, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of the invention, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of the invention will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of the invention, it will be apparent to those skilled in the art that the embodiments of the invention may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0113] Although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0114] The embodiments of this invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this invention should be included within the protection scope of this invention.

Claims

1. A data synchronization method based on OTS, comprising: Obtain the configuration file, parse the configuration file to obtain the synchronization task parameters, the synchronization task parameters include: the source table of the source OTS, the target table of the target OTS, and the filtering conditions; Using the first preset interface of the source end OTS, target data that meets the filtering conditions is read from the source table, and the target data is stored in the memory queue; Using a thread pool containing multiple concurrent worker threads, the target data is asynchronously retrieved from the memory queue, and the target data is written into the target table using the second preset interface of the target end OTS to obtain the data writing result; In response to determining that the data writing result is a writing failure, a recursive segmentation and retry operation is performed on the target data until the number of writing failures of the segmented single row data is greater than the number threshold, and a synchronization completion status signal is generated. In response to receiving the synchronization completion status signal, data volume verification and content sampling verification are performed on the source table and the target table to obtain the verification results.

2. The method according to claim 1, wherein in response to determining that the data write result is a write failure, a recursive segmentation and retry operation is performed on the target data until the number of write failures of the segmented single row data is greater than a threshold, and a synchronization completion status signal is generated, the method includes: The target data is divided into a first sub-batch and a second sub-batch based on the number of rows. Call the second preset interface to retry writing the first sub-batch and the second sub-batch respectively; If either the first sub-batch or the second sub-batch fails to write, it is split into two halves according to the number of rows as a new failed batch and recursively retried until the split batch contains only a single row of data. In response to the number of write failures of the single row of data exceeding a threshold, the synchronization completion status signal is generated.

3. The method according to claim 1, wherein the data volume verification of the source table and the target table includes: The total number of records in the source table that meet the filtering conditions and the total number of records in the target table that meet the filtering conditions are counted using a paginated scanning method. Compare the total number of records in the source table with the total number of records in the target table to see if they are the same, and calculate the difference value and the difference percentage.

4. The method according to claim 1, wherein content sampling verification is performed on the source table and the target table, comprising: Scan each page of the source table according to the filtering conditions and collect a preset number of row records as a sample set. For each sample record in the sample set, a point query is performed in the target table using the primary key of the sample record as the query condition to obtain the corresponding target row data. The consistency between the sample record and the target row data is compared field by field.

5. The method according to claim 1, wherein during the process of reading target data that meets the filtering conditions from the source table, the method further includes: Record the primary key of the next scan after each read operation is completed as the synchronization point, and persist the synchronization point. In response to a restart after a synchronization task is interrupted, the synchronization point is read, and the operation of reading the target data that meets the filtering conditions from the source table is continued with the synchronization point as the starting position.

6. The method according to claim 1, wherein the filtering conditions include time range conditions and / or attribute column content keyword conditions; The step of reading target data that meets the filtering conditions from the source table using the first preset interface of the source-end OTS includes: When the first preset interface is called, a column condition containing the time range condition and / or the attribute column content keyword condition is constructed in the interface call parameters, and the filtering condition is passed to the source OTS server so that the data filtering is completed at the source OTS server and only the data rows that meet the filtering condition are returned as the target data.

7. A data synchronization device based on OTS, comprising: The synchronization task parameter determination module is configured to obtain a configuration file, parse the configuration file to obtain synchronization task parameters, and the synchronization task parameters include: the source table of the source OTS, the target table of the target OTS, and filtering conditions. The target data reading module is configured to use the first preset interface of the source end OTS to read target data that meets the filtering conditions from the source table and store the target data in a memory queue. The target data writing module is configured to asynchronously retrieve the target data from the memory queue using a thread pool containing multiple concurrent worker threads, and write the target data into the target table using the second preset interface of the target end OTS to obtain the data writing result; The recursive segmentation retry module is configured to perform a recursive segmentation retry operation on the target data in response to determining that the data writing result is a writing failure, until the number of writing failures of the segmented single row data is greater than the number threshold, and generate a synchronization completion status signal. The data verification module is configured to perform data volume verification and content sampling verification on the source table and the target table in response to receiving the synchronization completion status signal, and obtain the verification result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterised in that, It includes computer program instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.