Cluster resource scheduling method and system

A scheduling method and resource scheduling technology, applied in resource allocation, program control design, instruments, etc., can solve problems such as resource contention scheduling and exceptions, reduce scheduling exceptions and resource contention, and solve resource contention and scheduling exceptions , Improve the effect of cluster computing capability and reliability

Pending Publication Date: 2019-12-20
南京星云数字技术有限公司
View PDF2 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to provide a scheduling method and system for cluster resources, which are used to improve cl...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Cluster resource scheduling method and system
  • Cluster resource scheduling method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 2

[0067] see figure 2 , the embodiment of the present invention also provides a cluster resource scheduling system, which adopts the cluster resource scheduling method in the previous embodiment, and the system includes a YARN cluster scheduling module and a resource monitoring module.

[0068] Among them, the YARN cluster scheduling module includes a resource manager and a node manager. The resource manager performs task resource scheduling on the node manager in real time based on the task priority; heartbeat communication is maintained between the resource manager and the node manager; resource scheduling and heartbeat The communication adopts an asynchronous and parallel execution mode; the resource monitoring module is used to obtain the total resource information and task execution information of the cluster in real time, and traverse the real-time resource occupation data of each task being executed; judge whether the resource occupation data is abnormal in real time acco...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a cluster resource scheduling method and system. The cluster resource scheduling method comprises the steps: carrying out the resource scheduling of a task in real time based on the priority of the task; acquiring total resource information and task execution information of the cluster in real time, and traversing real-time resource occupation data of each task being executed; and judging whether the resource occupation data is abnormal in real time according to the abnormal standard, and optimizing the resource scheduling of the task in real time according to the judgment result, so that the conditions of abnormal scheduling, resource contention and the like of the YARN under a high-load condition are reduced, and the cluster operation capability and reliability are further improved. The cluster resource scheduling system disclosed adopts the scheduling method, thus effectively improving the cluster operation capability and reliability, and solving the problemsof resource contention, scheduling abnormality and the like in a service peak period.

Description

technical field [0001] The invention relates to the field of resource scheduling in big data distributed computing, in particular to a cluster resource scheduling method and system. Background technique [0002] With the growth of data and enterprises processing more and more data every day, the original BI (Business Intelligence) business intelligence model can no longer satisfy the data storage and analysis functions of enterprises. The high performance, high reliability and high reliability represented by Hadoop A scalable big data platform emerges as the times require. Among them, YARN (Yet AnotherResource Negotiator, referred to as YARN), as the native resource scheduling system of Hadoop 2.x series, can provide unified resource management and task scheduling for various computing frameworks including programming models MapReduce, Storm, and Flink. [0003] Although the emergence of YARN has great benefits for the cluster in terms of utilization, unified resource manag...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50
CPCG06F9/5038G06F9/5083
Inventor 陈远光宇世飞方军
Owner 南京星云数字技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap