Data acquisition method and ETL (Extraction-Transformation-Loading) assembly

A data acquisition and component technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve problems such as system freeze, memory and IO port occupation, and reduce data acquisition efficiency, so as to improve integrity, The effect of stabilizing data collection efficiency and reducing performance impact

Inactive Publication Date: 2016-08-10
LETV HLDG BEIJING CO LTD +1
View PDF4 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In practical applications, the inventors of the present invention have found that when the ETL component extracts the data in the full data table with a relatively large amount of data in the business system, the memory and IO ports (input and output ports) of the data acquisition system where the ETL component is located will be affected. ) has a great impact, resulting in too much memory and IO ports, and the system freezes, which seriously reduces the efficiency of data collection and interferes with other data collection tasks.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data acquisition method and ETL (Extraction-Transformation-Loading) assembly
  • Data acquisition method and ETL (Extraction-Transformation-Loading) assembly
  • Data acquisition method and ETL (Extraction-Transformation-Loading) assembly

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0037] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0038] Those skilled in the art will understand that unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the word "comprising" used in the description of the present invention refers ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a data acquisition method and an ETL (Extraction-Transformation-Loading) assembly. The method comprises the following steps: at the arrival of a data acquisition period, performing leap type incremental data acquisition once specific to a full data table in a service system; determining acquisition times of the full data table; calculating a starting timestamp according to a minimum timestamp in the full data table, preset first acquisition duration and the determined acquisition times; calculating an ending timestamp according to the starting timestamp and the first acquisition duration; and acquiring data within a time range marked by the starting timestamp and the ending timestamp in the full data table. Through application of the data acquisition method and the ETL assembly, performance influence on a data acquisition system in which the ETL assembly is positioned can be lowered, and the data acquisition efficiency is stabilized when data acquisition is performed specific to the full data table.

Description

technical field [0001] The embodiment of the present invention relates to the technical field of data collection, in particular to a data collection method and an ETL component. Background technique [0002] ETL (Extraction-Transformation-Loading, data extraction, transformation and loading) components are an extremely important part of building a data warehouse. According to pre-designed rules, it provides the functions of extracting, converting, and cleaning data from different data sources, integrates and purifies data from multiple different systems, and finally loads the converted data incrementally or completely into the target database . It obtains unified target data from different heterogeneous data sources and serves as the basis for online analytical processing and data mining. [0003] Usually, the objects for data extraction by ETL components include full data tables or incremental data tables. For incremental data tables, the ETL component only needs to expo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 刘永华杨佳浩王孝庆
Owner LETV HLDG BEIJING CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap