Method and device for concurrent loading of database data

Abstract

The invention discloses a concurrent loading method and device for database data. The method comprises: receiving, by a target end, data files sent by a source end; sequentially storing the data filesin different loading directories of the target end by exporting different tables from big to small data volumes of the tables based on bottom data blocks of a statistical table in a full-volume dataloading phase to realize multi-process / thread concurrent loading of the target end; and storing the data files in different loading directories of the target end from big to small data volumes based on the incremental data of the statistical table in an incremental data loading phase to realize multi-process / thread concurrent loading of the target end; wherein the entire process of the multi-process / thread concurrent loading of the target end is sequential loading in processes / threads and is concurrently executed between the processes / threads. The database data can be concurrently loaded to improve the loading efficiency and realize the real-time performance of data synchronization. By cooperating with a multi-queue equalization mechanism, the loading task of each queue is averaged, resources are utilized reasonably, and the overall loading performance is ensured.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of data processing, and in particular to a method and device for concurrent loading of database data. Background technique [0002] The database is the core component of the application business system, and the stability of the database and the continuity of the business are the most concerned issues in the application business system. [0003] In the process of data synchronization, the loading of massive data is often involved. The existing mass data loading methods have the following problems: [0004] 1) Use a data migration tool or a backup recovery tool to copy the data in the original database to the disk, and then copy it to the target database through the disk. This method has a slow data loading speed and cannot realize real-time synchronization of the original database and the target database. [0005] 2) The target end single-threadedly executes SQL (Structured Query L...

AI Technical Summary

Problems solved by technology

[0004] 1) Use data migration tools or backup recovery tools to copy the data in the original database to the disk, and then copy the data to the target database through the disk. This method is slow in loading data and cannot realize real-time synchronization of the original database and target database data.

[0005] 2) The target end single-threadedly executes SQL (Structured Query Language) statements for data loading, which ensures the consistency of transactions, but this method is inefficient for data loading, which easily leads to a backlog of massive data at the target end, making it difficult to achieve real-time synchronization

[0006] 3) Multiple queues on the target end load data concurrently, which improves loading efficiency, but it is difficult to ensure transaction consistency; on the other hand, due to the lack of a queue balancing mechanism, some queues are idle, while some queues still have a large number of data files Waiting for loading, resulting in waste of resources

Images