Data synchronization method and device capable of realizing rapid data loading

Abstract

The invention relates to the technical field of data synchronization, in particular to a data synchronization method and device capable of achieving rapid data loading, and the method comprises the steps that: data is read from a data source, and the read data is put into a cache; the producer thread generates a plurality of data files from the data in the cache in batches, and puts the generateddata files into a task queue; the consumer thread calls a special batch data loading command of the target database, and loads the to-be-loaded data files in the task queue to the target database oneby one, wherein the data file generation process corresponding to the producer thread and the data batch loading process corresponding to the consumer thread are carried out in parallel. According tothe method, a producer-consumer model is adopted, the data files are continuously generated in batches during loading, the loading commands special for the target database are called to load the data,the generation of the data files and the execution of the data loading commands can be performed in parallel, the performance loss during serial execution is reduced, and the synchronization efficiency is improved.

Description

【Technical field】 [0001] The invention relates to the technical field of data synchronization, in particular to a data synchronization method and device capable of realizing fast data loading. 【Background technique】 [0002] At present, a typical data synchronization process usually includes two steps: one is to read the data from the data source and put it in the cache, that is, data reading; the other is to read the data from the cache and load it into the destination database, that is, the data load. Assuming that the source is a relational database, the current typical data synchronization process is as follows figure 1 As shown in , the data loading process generally calls the JDBC (Java Data Base Connectivity) or ODBC (Open Database Connectivity) interface to realize the access to the target database. Both JDBC and ODBC are general-purpose database programming interfaces, which can provide a complete set of programming interfaces independent of any specific database,...

AI Technical Summary

Problems solved by technology

The current common data integration tools support the above forms of batch loading, all require that the loaded data files must already exist on the disk, and cannot directly perform batch loading of data from the data source to the destination table.

To use batch loading, the user must first export the source table data to a file, and then use the batch loading command to load the data. The two steps must be executed serially, which is cumbersome, inefficient and has a large performance loss.

Images