Method and device for on-line analysis and processing of large data

An online analytical processing and big data technology, applied in the field of big data processing, can solve problems such as waste of resources and multiple computing resources, and achieve the effect of quickly responding to query requirements

Pending Publication Date: 2017-05-10
FLYING FOX INFORMATION TECH TIANJIN CO LTD
View PDF8 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

HBASE-based OLAP requires combined storage of all dimensions. When the dimension grows, the growth of the results is exponential. It also requires a lot of computing resources to store these r...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for on-line analysis and processing of large data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0031] Explanation of terms:

[0032] OLAP: Online Analytical Processing (Online Analytical Processing), which enables analysts, managers or executives to transform from raw data from multiple perspectives, can be truly understood by users, and truly reflects the dimensional characteristics of the enterprise. A class of software technologies that allow rapid, consistent, and interactive access to information to gain greater insight into the data.

[0033] Dimension: dimension, a dimension is a set of attributes that represent areas related to measures in a cube and are used to analyze measures in a cube.

[0034] HADOOP: Apache open source top project, distributed computing framewor...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and device for on-line analysis and processing of large data. The method comprises a storing step and a searching step. The data processed by ETL is assigned to the computing nodes for aggregation calculation. According to the method and device, the method combining the pre-calculation and compression is used to solve the storage pressure caused by large data to OLAP. Storage of historical data over a long period of time is achieved through cold and hot data separation. Moreover, distributed computing is used to separate computing pressure and quickly respond to searching requirements. The method and device for on-line analysis and processing of large data solves the storage pressure caused by large data to the OLAP by adopting the method combining pre-calculation and compression, achieves the storage of historical data over a long period of time through cold and hot data separation, and uses distributed computing to separate computing pressure and quickly respond to searching requirements.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a big data online analysis and processing method and device. Background technique [0002] The Internet industry has always been a producer and user of big data. Especially in recent years, the concept of Internet + has been proposed, which has greatly accelerated the development of the Internet industry. Opportunities and challenges often coexist. The rapid development of the Internet has brought us Valuable data, if these data are analyzed to obtain important knowledge and help decision makers make decisions is the main concern of major Internet companies. OLAP is undoubtedly the best way to solve such problems. Traditional Internet companies mostly rely on MYSQL and HBASE for OLAP. [0003] Based on MYSQL, MYSQL is one of the best open source relational databases. OLAP under this architecture can achieve most of the query and analysis needs only by writing SQL and...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/24552G06F16/2471G06F16/254
Inventor 史立校亢永杰王金明
Owner FLYING FOX INFORMATION TECH TIANJIN CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap