Unlock instant, AI-driven research and patent intelligence for your innovation.

Hive job management method and system based on web application and readable storage medium

A job management and network application technology, applied in special data processing applications, electronic digital data processing, structured data retrieval, etc., can solve problems such as high application difficulty and difficult hive, and achieve the effect of reducing application difficulty

Pending Publication Date: 2022-04-12
CHONGQING CHANGAN AUTOMOBILE CO LTD
View PDF1 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the existing solution focuses on the execution process control of the hive task group, and its application is difficult. It is only suitable for users with a certain big data foundation. For users who generally do not have a big data foundation, it is difficult to effectively operate and execute hive job-related functions

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hive job management method and system based on web application and readable storage medium
  • Hive job management method and system based on web application and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0031] This embodiment discloses a web application-based hive job management method.

[0032] Such as figure 1 As shown, the hive job management method based on the web application includes the following steps:

[0033] S1: Upload sql (structured query language) instructions for executing hive (data warehouse) tasks through web applications (network applications);

[0034] S2: Connect to hive based on sql commands and execute corresponding hive tasks;

[0035] S3: During the execution of the hive task, update the execution log of the hive task periodically (for example, every 3 seconds) through an asynchronous method;

[0036] S4: After the execution of the hive task is completed, the corresponding execution result and execution log are stored for downloading and calling.

[0037] In this embodiment, the execution result of the hive task is stored in the csv (comma-separated value) format: the resultSet of the execution result is written to a csv file in the project directo...

Embodiment 2

[0049] This embodiment discloses a web application-based hive job management system.

[0050] combine figure 2 As shown, the hive job management system based on web application is implemented based on the hive job management method of the present invention, specifically including:

[0051] Mysql database (relational database), used to store the execution log and execution status of hive tasks. The Mysql database creates a table to store the primary key id, application_id, execution status of the hive task, and the execution log of the hive task. The execution status is updated at each stage of hive task execution, and the execution status is mainly used to judge the end of the asynchronous method of the web application.

[0052] The web application (network application) is used to upload the sql command to execute the hive task and connect to hive, and periodically update the execution log in the Mysql database through an asynchronous method; the web application connects to...

Embodiment 3

[0063] This embodiment discloses a readable storage medium.

[0064] A readable storage medium, on which a computer management program is stored, and when the computer management program is executed by a processor, the steps of the web application-based hive job management method of the present invention are implemented. The readable storage medium may be a device with a readable storage function such as a U disk or a computer.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the technical field of big data application, in particular to a hive job management method and system based on web application and a readable storage medium. The method comprises the following steps: uploading a structured query language instruction for executing a data warehouse task through a network application; a data warehouse is connected based on the structured query language instruction, and a corresponding data warehouse task is executed; in the execution process of the data warehouse tasks, periodically updating execution logs of the data warehouse tasks through an asynchronous method; and after the data warehouse task is executed, storing the corresponding execution result and the execution log for downloading and calling. The invention further discloses a hive operation management system and a readable storage medium. According to the hive operation management method, the application difficulty of hive operation management can be effectively reduced, so that the big data processing effect can be improved.

Description

technical field [0001] The invention relates to the technical field of big data applications, in particular to a web application-based hive job management method, system and readable storage medium. Background technique [0002] Hadoop implements a distributed file system (Hadoop Distributed File System), referred to as HDFS. HDFS has the characteristics of high fault tolerance and is designed to be deployed on low-cost (low-cost) hardware; and it provides high throughput (high throughput) to access application data, suitable for applications with very large data sets . Hive is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and provides a simple SQL query function, which can convert SQL statements into MapReduce tasks to run. Its advantage is that the learning cost is low, and simple MapReduce statistics can be quickly realized through SQL-like statements, without the need to develop special MapReduce applications, which i...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/23G06F16/242G06F16/182G06F16/27G06F16/28
Inventor 李臻阳
Owner CHONGQING CHANGAN AUTOMOBILE CO LTD