A metadata reading method and device based on a Hive data warehouse

Abstract

The invention discloses a metadata reading method and device based on a Hive data warehouse. The method comprises the steps of receiving a reading request which is sent by a security management and control system and carries a key field; determining whether metadata corresponding to the key field is stored in a local cache or not; if it is determined that the metadata corresponding to the key field is stored in the local cache, sending a reading response carrying the metadata corresponding to the key field to the security management and control system; and if it is determined that the metadatacorresponding to the key field is not stored in the local cache, reading the metadata corresponding to the key field from the Hive data warehouse in a currently effective first reading mode, and sending a reading response carrying the metadata corresponding to the key field to the security management and control system. According to the scheme, an efficient and convenient metadata reading mode can be realized.

Description

technical field [0001] The invention relates to the technical field of data reading, in particular to a metadata reading method and device based on a Hive data warehouse. Background technique [0002] With the improvement of big data technology, more and more enterprises start to use Hadoop platform to manage their own data resources. Hive is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and provides a simple structured query language (Structured Query Language, SQL) query function, which can convert SQL statements into MapReduce tasks to run. Its advantage is that the learning cost is low, and simple MapReduce statistics can be quickly realized through SQL statements, without the need to develop special MapReduce applications, which is very suitable for statistical analysis of data warehouses. At the same time, as a member of the Hadoop ecosystem, Hive has the ability to manage massive data, and can perform statistical a...

AI Technical Summary

Problems solved by technology

[0005] The emergence of Apache Sentry has basically solved the security management and control problem of the Hive data warehouse, but it has a disadvantage: for business systems that need to control the operation security of the Hive data warehouse, a data table from roles to permissions is maintained, and each time the user needs to operate the Hive data Warehouse, Sentry will check the legitimacy of permissions before operation to achieve the purpose of control

[0009] As for the solution to read the metadata database, because the metadata database configuration of the business system is different in different environments, and in many cases, the user password of the metadata database will not be disclosed for security, so the use is still inflexible

Images