Intelligent double-engine analysis system and intelligent double-engine analysis method based on column storage

Abstract

The invention discloses an intelligent double-engine analysis system and an intelligent double-engine analysis method based on column storage, and relates to the field of smart city and big data processing. The system comprises an HBase storage unit, a client and an intelligent SQL (Structured Query Language) router unit. The system realizes the intelligent SQL router unit at a user side; the system adopts the HBase storage unit as a bottom layer data storage unit, and a real-time query engine and a batch analysis engine are built on the HBase storage unit; and the system aims at the data of structurized relational data. The system and the method have the advantages that the real-time simple query can be met; the performance for analyzing large-scale data can be effectively improved; and the statistics and analysis requirements of users on the large-scale data can be met.

Description

technical field [0001] The invention relates to the fields of smart cities and big data processing, in particular to an intelligent dual-engine analysis system and method based on column storage. Background technique [0002] With the popularity of mobile Internet, smart terminals, Internet of Things, cloud computing and smart cities, people have gradually entered the era of "big data". Big data is a very large and complex data set. When the amount of data reaches the level of PB, EB or ZB, traditional database management tools face many problems in processing, such as acquisition, storage, retrieval and analysis. In traditional relational databases, indexes and full table scans are generally used to process queries and analysis. Indexes can be positioned quickly and are efficient and fast for simple queries. However, for some complex statistical analysis or data mining applications, Generally, a full table scan is used. Big data has caused some problems, such as high conc...

AI Technical Summary

Problems solved by technology

[0007] However, MapReduce cannot distinguish business scenarios. No matter how simple the query is, MapReduce will convert it into a parallel processing task of scanning the entire table. Therefore, in some scenarios such as location search and retrieval, MapReduce is too cumbersome, time-consuming and consume resources

Since the files are not organized in a structured manner, and MapReduce itself is very time-consuming, in the Hadoop environment, the traditional Hive still cannot meet the user's query and statistical analysis requirements for large-scale data in terms of performance.

Images