A method and system for heterogeneous big data integration based on data warehouse

A data warehouse and database technology, which is applied in the Internet field to relieve pressure, improve reuse, and facilitate calculation.

Active Publication Date: 2016-09-07
北京新丝路咨询集团有限公司
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This method is mainly proposed for relational data databases, and does not involve the processing of heterogeneous data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and system for heterogeneous big data integration based on data warehouse
  • A method and system for heterogeneous big data integration based on data warehouse
  • A method and system for heterogeneous big data integration based on data warehouse

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0059] In the present invention, the respective advantages of relational databases, distributed databases, and memory databases are combined to associate structured data, semi-structured data, and unstructured data in Internet applications, and use Map / Reduce distributed processing and Data mining processing, the processing results and related data are written into the memory in the form of a database structure to form a simple memory database, which is convenient for high-speed calculation and rapid response.

[0060] Refer to figure 1 As shown, the specific steps of the data processing flow of the embodiment of the present invention are:

[0061] Step 100: Obtain data from the data source. Part of the structured data is collected through various business systems and stored in relational databases, including registration data, product data, sales data, inquiry data, business data, keyword data, etc. In addition, unstructured data such as social information, detailed product descr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a heterogeneous large data integration method and system based on data warehouses. The incidence relation between structurized data, semi-structurized data and non-structurized data are is established, all kinds of data are integrated by combining the advantages of a relational database, a distributed database and a memory database, deep data analysis is carried out on the basis of the data warehouses, data mining is deepened continuously, and thus high-efficiency and high-quality heterogeneous large data analysis is achieved. The structurized data, the semi-structurized data and the non-structurized data in Internet applications are associated, through Map / Reduce distributed processing and data mining, the processing result and relevant data are written into a memory in a database structure mode, thus, a simple memory database is formed, and high speed calculation and fast response can be carried out conveniently.

Description

Technical field [0001] The present invention mainly relates to the Internet field, in particular to a heterogeneous big data integration method and system based on a data warehouse. Background technique [0002] Business Intelligence (BI, Business Intelligence) comprehensively utilizes data warehouse, ETL technology, OLAP analysis and data mining technology to effectively integrate and store data, analyze the data, and extract the knowledge contained therein, thereby helping enterprises to make decision analysis , Has been more and more widely used in enterprises. [0003] With the rapid development of the Internet, the applications of the Internet have become more and more abundant. These applications have allowed the Internet to retain a large amount of data information, including user browsing records, transaction records, log files, web page information, and hyperlinks. How to obtain useful knowledge from the massive and dynamic Internet information data is the value of busine...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 徐晓冬邹铁鹏何昌桃黄建鹏
Owner 北京新丝路咨询集团有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap