MPP engine-based cross-data center rapid query method and system

Abstract

The invention relates to an MPP engine-based cross-data center rapid query method and system, belongs to the field of big data retrieval and analysis, and can be applied to real-time systems or offline backup systems. The method comprises the following steps of: uniformly marking data in different MPP engines and storing the data into metadata; receiving a query request by a global center point and carrying out grammar analysis on the query request, and sending the query request to corresponding data branch center nodes; carrying out query by each data branch center node through corresponding MPP engines of the metadata, and transmitting the data to the global center node; and rapidly querying the data returned by the data branch center points by the global center point by utilizing the MPP engines, and outputting a query result. According to the method and system, the union query of data of different data centers can be realized, a plurality of MPP engines are compatible, characteristics of different storage modes are sufficiently utilized to optimize the query, and the export of query result in a plurality of manners is supported, and convenience is brought to realize the different requirements, for result data re-analysis, of different upper applications.

Description

technical field [0001] The present invention relates to a cross-data center rapid query technology based on an MPP (Massively Parallel Processor) engine, in particular to metadata unified identification, reliable and fast data transmission, optimized query analysis engine and support for multiple result export The key technology of the method belongs to the field of big data retrieval. Background technique [0002] With the continuous popularization of network and information technology, the amount of data generated by human beings is increasing exponentially. It doubles approximately every two years, and according to monitoring, this rate will continue until 2020. This means that the amount of data generated by humans in the last two years is equivalent to the entire amount of data generated before. It can be predicted that the global data will reach 40ZB by 2020. The emergence of a large number of new data sources has led to the explosive growth of unstructured and semi...

AI Technical Summary

Problems solved by technology

[0007] Hive and the relational database store files differently. Hive uses Hadoop's HDFS (Hadoop's distributed file system), and the relational database is the server's local file system; the computing model used by Hive is mapreduce, while the relational database is its own Designed computing model; relational database is suitable for real-time query business, while Hive is suitable for massive data mining; inherited from Hadoop, Hive is easy to expand storage scale and computing power, while relational database is not easy to expand

[0008] Because Hive uses SQL, its problem domain is narrower than that of Map-Reduce, because many problems cannot be expressed through SQL, such as some data mining algorithms, recommendation algorithms, image recognition algorithms, etc., which can only be completed by writing Map-Reduce

Images