Hadoop task monitoring method and system

A task monitoring and task technology, applied in the field of big data processing, can solve problems such as wasted labor and poor user experience, and achieve the effect of protecting security

Inactive Publication Date: 2018-03-06
BEIJING GEO POLYMERIZATION TECH
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this way, the user experience is very poor and a lot of labor is wasted

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hadoop task monitoring method and system
  • Hadoop task monitoring method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0024] Such as figure 1 Shown, this Hadoop task monitoring method, this method comprises the following steps:

[0025] (1) start, perform the initialization operation;

[0026] (2) Obtain user whitelist white users: Obtain a list of authorized account names by querying the database, which is convenient for checking the validity of user names in subsequent tasks;

[0027] (3) Obtaining task whitelist white tasks: Obtain the task whitelist list by querying the database, which is convenient for subsequent task legitimacy checks, and tasks in the whitelist are skipped;

[0028] (4) Obtain the running task task set: obtain the task set through the hadoop API;

[0029] (5) Determine whether the task set is empty, if yes, execute step (6), otherwise execute step (8);

[0030] (6) Remove a task task from the task set: remove a task from the task set;

[0031] (7) Check the legitimacy of a single task: check the indicators of the task, determine whether the task is illegal, and jum...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a hadoop task monitoring method. The method comprises the steps of (1) starting a process, and executing initialization operation; (2) obtaining white users: obtaining an account name list with a permission by querying a database, thereby facilitating subsequent task user name legality check; (3) obtaining white tasks: obtaining a white tasks list by querying the database,thereby facilitating subsequent task legality check, and skipping the tasks in the white list; (4) obtaining a running task set: obtaining the task set through api of hadoop; (5) judging whether the task set is null or not, and if yes, executing the step (6), or otherwise, executing the step (8); (6) shifting out a task from the task set: extracting the task from the task set; (7) checking the legality of single task: checking various indexes of the task, judging whether the task is legal or not, and skipping to the step (5); and (8) ending the process. The invention furthermore discloses a hadoop task monitoring system.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a hadoop task monitoring method and a hadoop task monitoring system. Background technique [0002] Regarding the task viewing function implemented by the web page, currently users can only view the web page. If the task seriously drags down the cluster, it can only be found and killed manually. In this way, the user experience is very poor and a lot of labor is wasted. Contents of the invention [0003] In order to overcome the defects of the prior art, the technical problem to be solved in the present invention is to provide a hadoop task monitoring method, which automatically and in real time monitors the resource occupancy of each task of the hadoop cluster, and will automatically close when it is found that the tasks occupy too many resources Task. [0004] Technical scheme of the present invention is: this Hadoop task monitoring method, this method comprises ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F21/55
CPCG06F9/485G06F21/554
Inventor 崔晶晶吴淑佳
Owner BEIJING GEO POLYMERIZATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap