HIVE task scheduling method, device and equipment based on big data platform and storage medium

Abstract

The invention discloses an HIVE task scheduling method and device based on a big data platform, equipment and a storage medium. The method comprises the steps of obtaining an original HIVE task sent by a client, triggering a log program based on a starting file in the original HIVE task, and obtaining a task log table; obtaining a target HIVE task from the to-be-processed HIVE task; reading a configuration file in the target HIVE task by adopting a configuration file reading tool; if the reading is successful, obtaining a front task identifier and a self task identifier contained in a configuration file in the target HIVE task; if the front task log carries a task completion label corresponding to the front task identifier, successfully completing the front HIVE task corresponding to the front task identifier, and executing a business file in the target HIVE task; and if the business file is successfully executed, generating a task completion tag, and storing the task completion tag and the task identifier of the task completion tag in a target task log corresponding to the target HIVE task in an associated manner. According to the method, the HIVE task execution efficiency can beimproved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a HIVE task scheduling method, device, equipment and storage medium based on a big data platform. Background technique [0002] HIVE is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and provides a simple sql query function, which can convert sql statements into MapReduce tasks for operation. Its advantage is that the learning cost is low, and simple MapReduce statistics can be quickly realized through SQL-like statements, without the need to develop special MapReduce applications, which is very suitable for statistical analysis of data warehouses. HIVE is a data warehouse infrastructure built on Hadoop. It provides a set of tools that can be used to extract, transform and load (ETL), a mechanism that can store, query and analyze large-scale data stored in Hadoop. HIVE defines a simple SQL-like query language ca...

AI Technical Summary

Problems solved by technology

[0003] Embodiments of the present invention provide a HIVE task scheduling method, device, device, and storage medium based on a big data platform to solve the problem that the current HIVE task is not associated with its predecessor HIVE task, resulting in low task execution efficiency

Images