Self-adaptive data synchronization system, control method and application

Abstract

The invention belongs to the technical field of big data processing, and discloses a self-adaptive data synchronization system, a control method and application, and the self-adaptive data synchronization system comprises a data source layer, a read plug-in layer, a frame component, a data layer, a write plug-in layer, a database and a running environment. And the data source layer comprises Mysql, MongoDB, Clickhome and the like. The data source layer comprises Mysql, MongoDB, Clickhome and the like. The framework assembly comprises self-adaptive thread scheduling, task scheduling, system setting, storage scheduling and a UI interface. The operation environment comprises an Ali cloud host, an independent server and a third-party virtual host. According to the invention, a scheduler for automatic thread allocation is designed, and the number of threads for data synchronization is allocated according to the number of cores of a current CPU (Central Processing Unit). According to the method, CPU resources can be fully utilized, under the condition of a large number of synchronous tasks, it is guaranteed that the total speed of writing in databases such as Hbase is roughly equal to the total speed of all threads used by one task, and the problem of thread dynamic allocation is solved.

Description

technical field [0001] The invention belongs to the technical field of big data processing, and in particular relates to an adaptive data synchronization system, a control method and an application. Background technique [0002] At present, in the big data platform, when other types of data are synchronized to the big data database HBase, when the number of synchronized threads is large, there will be thread contention, because threads frequently switch, compete for CPU resources, and thread switching It also requires time and other resources. The more threads, the greater the consumption of this part, and the fewer resources are left for the threads to run. [0003] Through the above analysis, the existing problems and defects of the prior art are: in the prior art, when the number of synchronized threads is large, there will be thread contention. [0004] The difficulty of solving the above problems and defects is: [0005] 1. How to abstract the performance configuratio...

AI Technical Summary

Problems solved by technology

[0003] Through the above analysis, the existing problems and defects of the prior art are: in the prior art, when the number of synchronized threads is large, there will be thread contention

Images