Data synchronization framework program

Abstract

The invention discloses a data synchronization framework program, which comprises a set of API (Application Program Interface) for data synchronization processing and a core computing engine, the core computing engine is composed of a CoordinatorService, a TaskExecution Service and a Slot Service, the CoordinatorService is a Master service of a cluster, provides a generation process of each job from LogicalDag to Execution Dag and then to PhysicalDag, and finally creates a JobMaster of the job to carry out scheduling execution and state monitoring of the job, so that the data synchronization framework program can be used for realizing data synchronization processing. The TaskExecution Service is a Worker service of the cluster and provides a real runtime environment of each Task in a job, the TaskExecution Service uses a DynamicThread Sharing technology to reduce the use of a CPU (Central Processing Unit), and the Slot Service can run on each node of the cluster and is mainly responsible for division, application and recovery of resources on the nodes. According to the method, the pressure influence on a data source is reduced, data delay possibly caused by hardware problems such as node downtime is solved, and more table synchronization is completed with fewer resources.

Description

Technical Field

[0001] The present invention relates to the field of big data and database technology, and in particular to a data synchronization framework program. Background Art

[0002] The data synchronization function helps users achieve real-time data synchronization between two data sources. It can be applied to various business scenarios, including remote multi-active deployment, remote data disaster recovery, local data disaster recovery, remote multi-active deployment, cross-border data synchronization, query and report diversion, cloud BI, and real-time data warehouses. Currently, existing data synchronization framework programs / engines have low resource utilization, high database connector counts put pressure on database servers, and there is no guarantee that writing to and reading data from the cache are transactional, which may lead to data inconsistencies. The current table schema cannot be changed to adapt to data that changes over time. Fault tolerance is low, downstream operators are highly dependent on the calculation results of upstream operators, and the data synchronization speed is uncontrollable, which excessively impacts the source or target database. Therefore, we propose a data synchronization framework program to address these issues. Summary of the Invention

[0003] The present invention aims to address existing issues in data synchronization framework programs / engines, including low resource utilization, high database server pressure caused by high database connector requirements, inability to ensure transactional data writes and reads from the cache, which can lead to data inconsistencies, inability to change the current table schema to adapt to time-varying data, low fault tolerance, downstream operators being highly dependent on the calculation results of upstream operators, and uncontrollable data synchronization speed, which can excessively impact the source or target database. To achieve the above objectives, the present invention adopts the following technical solution: a data synchronization framework program comprising a set of data synchronization processing APIs and a core computing engine, the core computing engine consisting of a CoordinatorService, a TaskExecutionService, and a SlotService. Preferably, the CoordinatorService is the cluster's Master service, providing the generation process for each job from LogicalDag to ExecutionDag, and then to PhysicalDag, and ultimately creating the job's JobMaster to schedule, execute, and monitor its status. Preferably, the TaskExecutionService is the Worker service of the cluster, which provides the real runtime environment for each Task in the job. TaskExecutionService uses DynamicThreadSharing technology to reduce CPU usage. Preferably, the SlotService runs on each node in the cluster and is mainly responsible for the division, application and recycling of resources on the node. Preferably, the Runtime Environment includes Local, Standalone, Yarn and K8s. Preferably, the data sources include HDFS, Hive, Mysql, Oracle, lotDB, Kafka, LocalFile, Socket and CDC. Preferably, the Data Persistence includes JDBC and File Svstem, the JDBC includes MySQL and PG, and the File Svstem includes Local, HDFS, S3 and OSS. Preferably, the Hazelcast IMDG includes CheckpointSnapshot, JobStatus and Distributed Data Structures.The beneficial effects of the present invention are as follows: 1. Autonomous cluster: realize autonomous cluster (decentralization), in order to achieve cluster autonomy and job fault tolerance without relying on third-party service components (such as Zookeeper), even if the entire cluster crashes and restarts, the runtime information of the job can be repaired; 2. Data cache: natural support for cache. When there are multiple synchronization jobs in the cluster sharing a data source, the SeaTunnel engine will automatically enable data cache, thereby reducing the reading pressure of the data source and reducing the impact of data synchronization on the data source; 3. Controllable speed: support speed limit during data synchronization, which can not only ensure that data synchronization is completed on time, but also minimize the pressure on the data source; 4. Shared connection pool: use a shared connection pool to ensure that multiple tables can share JDBC connections, thereby reducing the use of database connections, avoiding excessive database connections that lead to excessive database pressure and reduced read and write delays and stability; 5. Support incremental and full breakpoint resumption: continuously save the state (checkpoint) during offline synchronization, and when the job hangs and is rerun, it will continue to run from the last checkpoint, which effectively solves the data delay caused by hardware problems such as node downtime; 6. Support schema evolution: allows customers to easily change the current schema of the table to adapt to the function of data that changes over time; 7. Finer-grained fault-tolerant design: compared with the job-level fault tolerance and rollback of other synchronization framework programs, this framework program supports job calling, fault tolerance and rollback at the pipeline granularity; 8. Dynamic shared threads: before starting the thread, the data size is judged to automatically optimize thread usage. Jobs with small data volumes share threads, and jobs with large data volumes have exclusive threads, making it possible to run thousands of tables on the same node and complete the synchronization of more tables with fewer resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] FIG1 is a schematic diagram of the structure of a data synchronization framework program proposed by the present invention. DETAILED DESCRIPTION

[0005] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0006] Example 1 Referring to Figure 1, a data synchronization framework program includes a set of data synchronization processing APIs and a core computing engine. The core computing engine consists of CoordinatorService, TaskExecutionService and SlotService. CoordinatorService is the Master service of the cluster, which provides the generation process of each job from LogicalDag to ExecutionDag and then to PhysicalDag, and finally creates the JobMaster of the job to schedule, execute and monitor the status of the job. TaskExecutionService is the Worker service of the cluster, which provides the real runtime environment for each Task in the job. TaskExecutionService uses DynamicThreadSharing technology to reduce CPU usage. SlotService runs on each node of the cluster and is mainly responsible for the division, application and recycling of resources on the node. Runtime Environment includes Local and Standalone. Data sources include HDFS, Hive, Mysql, Oracle, lotDB and Kafka. Data Persistence includes JDBC and File Svstem. JDBC includes MySQL and PG. File Svstem includes Local, HDFS and S3. Like OSS, Hazelcast IMDG includes CheckpointSnapshot, JobStatus, and Distributed DataStructures.

[0007] In this invention, autonomous clusters are implemented (decentralized) to achieve cluster autonomy and job fault tolerance without relying on third-party service components (such as Zookeeper). Even if the entire cluster crashes and restarts, job runtime information can be restored. Data caching is natively supported. When multiple synchronization jobs in a cluster share a data source, the data engine automatically enables data caching, reducing the read pressure on the data source and minimizing the impact of data synchronization on the data source. Speed control is supported: Speed limits are supported during data synchronization, ensuring that data synchronization is completed on time while minimizing the pressure on the data source. Shared connection pools are used to ensure that multiple tables can share JDBC connections, reducing database connection usage and preventing excessive database connection pressure, which can lead to reduced read and write latency and stability. Incremental and full data resumable uploads are supported: During offline synchronization, state is continuously saved (checkpoints are performed). When a job fails and is rerun, it resumes from the last checkpoint, effectively resolving data delays caused by hardware issues such as node failures. Schema evolution is supported: Allows customers to easily change the current schema of a table to adapt to data changes over time. Finer-grained fault-tolerant design: Compared with the job-level fault tolerance and rollback of other synchronization framework programs, this framework program supports job calling, fault tolerance and rollback at the pipeline granularity. Dynamic shared threads: Before starting the thread, the amount of data is determined to automatically optimize thread usage. Jobs with small amounts of data share threads, while jobs with large amounts of data have exclusive threads, which makes it possible to run thousands of tables on the same node and complete the synchronization of more tables with fewer resources. The above is only a preferred specific implementation method of the present invention, but the scope of protection of the present invention is not limited to this. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, shall make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention, which shall be covered within the scope of protection of the present invention.

Claims

1. A data synchronization framework program, including a set of data synchronization processing APIs and a core computing engine, characterized by: The core computing engine consists of CoordinatorService, TaskExecutionService and SlotService.

2. A data synchronization framework program according to claim 1, characterized in that: The CoordinatorService is the Master service of the cluster, which provides the generation process of each job from LogicalDag to ExecutionDag and then to PhysicalDag, and finally creates the JobMaster of the job to schedule, execute and monitor the status of the job.

3. A data synchronization framework program according to claim 1, characterized in that: The TaskExecutionService is a cluster worker service that provides the actual runtime environment for each task in the job. The TaskExecutionService uses DynamicThreadSharing technology to reduce CPU usage.

4. A data synchronization framework program according to claim 1, characterized in that: The SlotService runs on each node in the cluster and is mainly responsible for the division, application and recycling of resources on the node.

5. A data synchronization framework program according to claim 1, characterized in that: The RuntimeEnvironment includes Local, Standalone, Yarn, and K8s.

6. A data synchronization framework program according to claim 1, characterized in that: The data sources include HDFS, Hive, Mysql, Oracle, lotDB, Kafka, LocalFile, Socket and CDC.

7. A data synchronization framework program according to claim 1, characterized in that: The DataPersistence includes JDBC and File Svstem, the JDBC includes MySQL and PG, and the File Svstem includes Local, HDFS, S3 and OSS.

8. A data synchronization framework program according to claim 1, characterized in that: The HazelcastIMDG includes CheckpointSnapshot, Job Status, and Distributed Data Structures.

Images