Method for preventing excessive occupation of big data task resources

A technology of resource occupation and big data, applied in the field of big data, can solve problems affecting the overall operation of big data services, downtime of big data clusters, high resource occupation of big data clusters, etc.

Inactive Publication Date: 2021-11-26
SHANDONG ARTAPLAY INTELLIGENT TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, when a task is abnormal or the running time is too long, it will cause the resource usage of the big data cluster to be too high, thereby affecting the normal execution of other tasks, resulting in a backlog of tasks, and causing the downtime of the big data cluster
Will seriously affect the overall operation of big data services

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for preventing excessive occupation of big data task resources

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0035] The method of establishing the big data Yarn task health status reporting service in step a) is:

[0036] a-1.1) Create a tool class YarnAppListener, and inherit the SparkListener class in this class;

[0037] a-1.2) Establish onApplicationStart and onApplicationEnd in the YarnAppListener class to realize sending task details to the control server when the task starts and ends, respectively;

[0038] a-1.3) When creating a spark object, add config, specify spark.extraListeners as the YarnAppListener class, and automatically execute YarnAppListener when the task is running.

Embodiment 2

[0040] The big data Yarn task control client established in step a) executes and controls the instructions sent by the server. The big data Yarn task control client has the authority to execute the Kill Yarn task. The big data Yarn task control client is deployed on the nodes of the big data cluster .

Embodiment 3

[0042] The functions provided by the big data Yarn task management and control server established in step a) are:

[0043] a-2.1) The big data Yarn task control server provides a timeout threshold for configuring each task;

[0044] a-2.2) The big data Yarn task management and control server provides and receives the Yarn task information reported by the big data Yarn task health status reporting service;

[0045]a-2.3) When the big data Yarn task management and control server judges that the current task running time exceeds the configured time threshold, it inputs the task running information into the data model, and judges the task based on the task's running status, resources occupied by running, and running progress Whether it affects other tasks and cluster operation, and if so, send an instruction to stop the task to the big data Yarn task control server;

[0046] a-2.4) The big data Yarn task management and control server counts the data according to the running time ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a method for preventing excessive occupation of big data task resources, in particular to a method and device for managing and controlling the running state, running time, resource occupation and the like of a Yarn task and stopping the Yarn task after the task is accurately predicted when the Yarn task is overtime. The running state of the Yarn task can be effectively managed and controlled, when the task exceeds a time threshold value, task running information is input into a data model, whether the task influences other tasks or not and influences normal running of a cluster or not is predicted, if a model output result is abnormal, the Yarn task is stopped in time, resource occupation caused by task timeout is prevented, and then the normal operation of other Yarn tasks and the normal operation of the whole cluster are influenced.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to a method for preventing the resource occupation of big data tasks from being too high. Background technique [0002] In recent years, the application of big data technology has become more and more extensive, and more and more tasks need to be run. However, when a task is abnormal or the running time is too long, it will cause the resource usage of the big data cluster to be too high, which will affect the normal execution of other tasks, cause a backlog of tasks, and cause the big data cluster to go down. It will seriously affect the overall operation of big data services. [0003] In order to prevent problems such as high resource usage and task backlog of big data tasks, it is necessary to control each big data task, and control whether the task is started normally, whether it runs normally, and whether it runs overtime. Once the task running time exceeds the time threshold...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
Inventor 李帅徐杰刘宇祥谢恩鹏
Owner SHANDONG ARTAPLAY INTELLIGENT TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap