Intelligent double-engine analysis system and intelligent double-engine analysis method based on column storage

An analysis system and dual-engine technology, which is applied in the field of intelligent dual-engine analysis system based on column storage, can solve the problems that the performance cannot meet the user's needs for large-scale data query and statistical analysis, MapReduce cannot distinguish business scenarios, and MapReduce is cumbersome. Achieve efficient statistics and analysis, save time and resources, and consume less resources

Active Publication Date: 2014-11-05
WUHAN POST & TELECOMM RES INST CO LTD
View PDF4 Cites 22 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] However, MapReduce cannot distinguish business scenarios. No matter how simple the query is, MapReduce will convert it into a parallel processing task of scanning the entire table. Therefore, in some scenarios such as location search and retrieval, MapReduce is too cumbersome, time-consu

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Intelligent double-engine analysis system and intelligent double-engine analysis method based on column storage
  • Intelligent double-engine analysis system and intelligent double-engine analysis method based on column storage

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0030] see figure 1 As shown, the embodiment of the present invention provides an intelligent dual-engine analysis system based on column storage, including an HBase storage unit, a client and an intelligent SQL routing unit, and the system implements an intelligent SQL routing unit on the user side; the system adopts the HBase storage unit As the storage unit of the underlying data, a real-time query engine and a batch analysis engine are established on the HBase storage unit. The data targeted by the system is structured relational data.

[0031]The HBase storage unit is used to: realize the underlying storage of data. When storing data, the primary key of the data is automatically mapped to the primary key of HBase. If the user's data has other columns with indexes, it is generated through a separate HBase table. Secondary index t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an intelligent double-engine analysis system and an intelligent double-engine analysis method based on column storage, and relates to the field of smart city and big data processing. The system comprises an HBase storage unit, a client and an intelligent SQL (Structured Query Language) router unit. The system realizes the intelligent SQL router unit at a user side; the system adopts the HBase storage unit as a bottom layer data storage unit, and a real-time query engine and a batch analysis engine are built on the HBase storage unit; and the system aims at the data of structurized relational data. The system and the method have the advantages that the real-time simple query can be met; the performance for analyzing large-scale data can be effectively improved; and the statistics and analysis requirements of users on the large-scale data can be met.

Description

technical field [0001] The invention relates to the fields of smart cities and big data processing, in particular to an intelligent dual-engine analysis system and method based on column storage. Background technique [0002] With the popularity of mobile Internet, smart terminals, Internet of Things, cloud computing and smart cities, people have gradually entered the era of "big data". Big data is a very large and complex data set. When the amount of data reaches the level of PB, EB or ZB, traditional database management tools face many problems in processing, such as acquisition, storage, retrieval and analysis. In traditional relational databases, indexes and full table scans are generally used to process queries and analysis. Indexes can be positioned quickly and are efficient and fast for simple queries. However, for some complex statistical analysis or data mining applications, Generally, a full table scan is used. Big data has caused some problems, such as high conc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30H04L29/08
CPCG06F16/221G06F16/951H04L69/12
Inventor 郝俊瑞向智宇许德玮高汉松郭嘉
Owner WUHAN POST & TELECOMM RES INST CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap