Standard interface accessing HDFS (Hadoop Distribute File System) distribution type storage system

Abstract

The invention discloses a standard interface accessing an HDFS (Hadoop Distribute File System) distribution type storage system. By storing system table information of a database in a local file at amaster node of a distribution type database and storing data in all other tables in an HDFS, a user receives client connection through the master node and processes a command of a client finally, andthe master node parses and optimizes query, distributes the query to a segment, coordinates execution of the query and stores metadata of all the systems without storing any user data. The standard interface has the beneficial effects that by storing the metadata in the local file system on a master host computer and storing the data of all other tables in the HDFS, the segment does not store anystate and data information and is only charge of calculation, so that separation of read-write, storage and calculation is realized; arbitrary multiple virtual segments can be dynamically started to execute query, so that the execution efficiency is improved; and a cluster is more liable to extend as the segment in a non-storing a state.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to a standard interface for accessing a HDFS distributed storage system. Background technique [0002] Big data is a very hot topic at present, and the core of big data is how to store, analyze and mine massive data to solve practical problems. How to store, query and analyze TB / PB-level data is an unavoidable problem in the era of big data. The HDFS file system is a distributed file system running on ordinary hardware. It provides a highly fault-tolerant and high-throughput mass data storage solution. It is very suitable for applications on large-scale data sets and is widely used in big data. as a storage system. Therefore, the research on the access of the HDFS distributed storage system has become a hot spot in the study of big data. [0003] HDFS is one of the core components of Hadoop. Although the MapReduce component in the Hadoop system can extract data from it and perfo...

AI Technical Summary

Problems solved by technology

[0007] Regardless of the various solutions adopted at present, either there is insufficient compatibility with SQL, or the execution efficiency is not high, or the cost is high, and the needs of users cannot be well met.

Images