Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Incremental data extracting method, device and system

An incremental data and data technology, applied in the field of incremental data extraction, can solve the problems of delay, time-consuming, time-consuming and computing resources, etc., to save system resources, improve efficiency, and reduce work pressure. Effect

Active Publication Date: 2012-12-26
ALIBABA GRP HLDG LTD
View PDF2 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Obviously, in the above step A, it takes 2-3 hours to scan all the hundreds of millions of data in table a and then create the on-site table 1, and then it takes longer to transfer the data to the data warehouse through the network; and, in step C, perform Association operations are also very time-consuming
[0005] Therefore, if the traditional extraction method is used, since the scale of the incremental data is constantly expanding, for example, the data extraction of a large table on the above-mentioned front-end website can reach 5 hours, which not only consumes a lot of time and computing resources, but also Causes delays in data warehouse data calculations

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Incremental data extracting method, device and system
  • Incremental data extracting method, device and system
  • Incremental data extracting method, device and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0024] Based on the problems caused by extracting all the front-end data to the data warehouse in the existing traditional scheme, this application proposes to use the primary key information of the incremental data to obtain the changed data, and only send the changed data to the data warehouse for subsequent calculations. This saves a lot of time and system resources, and greatly improves the efficiency of incremental data extraction.

[0025] Among them, it should be noted that those of ordinary skill in the art can easily understand that the incremental data mentioned in the embodiment of the present application is the daily change data of the foreground website; of course, in the specific application process, the incremental data It can also be the change data of other applications and forms, and is not limited to the change data of the front-end website, nor is it limited to the change data of each day in terms of time, which will not be described in detail in this articl...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention relates to an incremental data extracting method, an incremental data extracting device and an incremental data extracting system. The incremental data extracting method comprises the following steps of: acquiring primary key information of an incremental data from a standby database; according to the primary key information, inquiring the entire incremental data in a main database in data synchronism with the standby database; and inserting the inquired entire incremental data into a target database. After adoption of the method, the device and the system for incremental data extraction, a lot of time and system resources can be saved and the incremental data extracting efficiency is greatly improved.

Description

technical field [0001] The present application relates to the technical field of data transmission, in particular to a method, device and system for realizing incremental data extraction. Background technique [0002] With the rapid development of the Internet, the amount of data displayed on the website is increasing. At the same time, the amount of data transmission between the front-end website and the back-end data warehouse is also increasing; when the back-end data warehouse performs data calculations, it needs Extract data from the front-end website. [0003] At present, the traditional implementation solution is that the data warehouse uses the hash operation method to extract data; for example: suppose the front-end website has table a, the data volume of this table is about 100 million, and the incremental data per day is about 600W. Now the data warehouse The incremental data of the table needs to be extracted every day. The extraction process is as follows: A. F...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F17/30563G06F11/1402G06F16/254G06F16/273
Inventor 范鑫
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com