Incremental data extracting method, device and system

Abstract

The embodiment of the invention relates to an incremental data extracting method, an incremental data extracting device and an incremental data extracting system. The incremental data extracting method comprises the following steps of: acquiring primary key information of an incremental data from a standby database; according to the primary key information, inquiring the entire incremental data in a main database in data synchronism with the standby database; and inserting the inquired entire incremental data into a target database. After adoption of the method, the device and the system for incremental data extraction, a lot of time and system resources can be saved and the incremental data extracting efficiency is greatly improved.

Description

technical field [0001] The present application relates to the technical field of data transmission, in particular to a method, device and system for realizing incremental data extraction. Background technique [0002] With the rapid development of the Internet, the amount of data displayed on the website is increasing. At the same time, the amount of data transmission between the front-end website and the back-end data warehouse is also increasing; when the back-end data warehouse performs data calculations, it needs Extract data from the front-end website. [0003] At present, the traditional implementation solution is that the data warehouse uses the hash operation method to extract data; for example: suppose the front-end website has table a, the data volume of this table is about 100 million, and the incremental data per day is about 600W. Now the data warehouse The incremental data of the table needs to be extracted every day. The extraction process is as follows: A. F...

AI Technical Summary

Problems solved by technology

[0004] Obviously, in the above step A, it takes 2-3 hours to scan all the hundreds of millions of data in table a and then create the on-site table 1, and then it takes longer to transfer the data to the data warehouse through the network; and, in step C, perform Association operations are also very time-consuming

[0005] Therefore, if the traditional extraction method is used, since the scale of the incremental data is constantly expanding, for example, the data extraction of a large table on the above-mentioned front-end website can reach 5 hours, which not only consumes a lot of time and computing resources, but also Causes delays in data warehouse data calculations

Images