Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

High-efficient and low-cost loading method for heterogeneous mass data

A massive data, heterogeneous technology, applied in data exchange networks, digital transmission systems, electrical components, etc., can solve the problem of high procurement costs, reduce development costs, and solve the effect of heterogeneous mass data loading

Active Publication Date: 2012-10-10
上海全成通信技术有限公司
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] To implement ETL, one is to use third-party tools, such as Data Integrator, Data stage, Infomatica and other tools, which not only require high procurement costs, but also require specialized server hardware and software configuration, and professional technical developers and system maintenance personnel. Unacceptable for most SMEs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • High-efficient and low-cost loading method for heterogeneous mass data
  • High-efficient and low-cost loading method for heterogeneous mass data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] Such as figure 1 As shown, the present invention relates to a high-efficiency and low-cost loading method for heterogeneous mass data, the method comprises that other manufacturers transmit the data to the FTP server (the same local area network as the database server) through CLIENT-FTP; After the file, check the verification file and the original data file. If they are consistent, they will be stored in the warehouse, and the storage will be marked in the database cache table and the file status will be marked as normal. Otherwise, the abnormality and the cause of the abnormality will be marked; the last step is to clean the buffer data and Inserted into the formal interface table.

[0023] A method for loading and processing heterogeneous massive data, which integrates heterogeneous data sources;

[0024] (1) Adopt high-parallel direct path data loading; increase the storage buffer interface table, and partition the storage interface table by day;

[0025] (2) Ther...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a high-efficiency and low-cost loading method for heterogeneous mass data, which comprises the following steps that: a heterogeneous data source is converted into a general unified flat file interface; a user transmits the data to an FTP server; an interface buffer table is built for an target database creating and an interface table are partitioned by dates; a warehousing loader program receives a file, and then detects whether a verification file is consistent with an original data file; if so, the file is warehoused and is marked to be the warehoused in the warehousing interface buffer table, and the file state is marked as normal; otherwise, the file is marked as abnormal and abnormal cause; and data in a buffer area is subjected to data cleaning, and is inserted to a formal warehousing interface table. Compared with the prior art, the method effectively solves the problem of loading the heterogeneous mass data, avoids purchasing third-party ETL software and employing professional technical personnel at high investment at the same time, and provides a solution for medium and small-sized companies to reduce development cost.

Description

technical field [0001] The invention relates to a high-efficiency, stable and reliable data loading scheme for massive data, in particular to a high-efficiency and low-cost loading method for heterogeneous massive data. Background technique [0002] The information system on which BI operates is a complex data collection composed of traditional systems, incompatible data sources, databases and applications, and the various parts cannot communicate with each other. From this perspective, the currently running application system is an irreplaceable system built by the enterprise with a lot of energy and financial resources, especially the system data. The purpose of the newly-built BI system is to assist in decision-making through data analysis, but the sources and formats of these data are different, which makes system implementation and data integration difficult. At this time, enterprises very much hope to have a comprehensive solution to solve their own difficulties and s...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08H04L29/06H04L12/56H04L12/26
Inventor 冯谧
Owner 上海全成通信技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com