YARN resource allocation and energy-saving scheduling method and system based on service level agreement

A service level agreement and resource allocation technology, applied in resource allocation, energy-saving computing, data processing power supply, etc., can solve problems such as suboptimal scheduling and resource allocation decisions, and YARN cannot be applied

Active Publication Date: 2015-10-21
SHANDONG UNIV
View PDF2 Cites 32 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, due to differences in system structure and resource management mechanisms, some of these existing scheduling s...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • YARN resource allocation and energy-saving scheduling method and system based on service level agreement
  • YARN resource allocation and energy-saving scheduling method and system based on service level agreement
  • YARN resource allocation and energy-saving scheduling method and system based on service level agreement

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0093] The present invention will be further described below with reference to the drawings and embodiments.

[0094] YARN is a resource management system in Hadoop 2.0, which is mainly composed of several components such as resource manager RM, node manager NM, application manager AM, and resource container Container. Resource scheduler is one of the core components of YARN, such as figure 2 As shown, YARN uses a two-tier resource scheduling model:

[0095] In the first layer, the resource scheduler in the global resource manager RM allocates resources to each application manager AM;

[0096] In the second layer, the application manager AM further allocates resources to its internal tasks.

[0097] The present invention proposes a YARN resource allocation and energy-saving scheduling strategy based on a service level agreement. Based on the YARN architecture, the resource scheduler is modified, and program analyzer, parallelism estimator, performance monitor and frequency estimator ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention discloses a YARN resource allocation and energy-saving scheduling method and system based on a service level agreement. The YARN resource allocation and energy-saving scheduling method comprises the following steps of: before submitting MapReduce programs, carrying out pre-analysis on the MapReduce programs and analyzing required performance indexes from previous running logs of the programs; after submitting the MapReduce programs, calculating the minimum task parallelism degrees based on a completion time upper limit according to the performance indexes of the MapReduce programs; according to different parallelism degrees of each MapReduce program allocating quantitative resources to the MapReduce program by an SLA resource scheduler; monitoring the task completion condition of each MapReduce program so as to obtain ideal execution time and frequencies of residual tasks; and according to expected execution frequencies of the residual tasks, dynamically regulating a voltage and a frequency of a CPU by utilizing a CPUfreq subsystem so as to fulfill the aim of saving energy. According to the present invention, on the premise of ensuring the service level agreement of the MapReduce program, the quantitative resources are allocated to the MapReduce program; and a dynamic voltage frequency regulating technology is combined to reduce energy consumption in a cloud calculation platform to the greatest extent.

Description

Technical field [0001] The invention belongs to the technical field of cloud computing, and in particular relates to a method and system for YARN (Yet Another Resource Negotiator) resource allocation and energy-saving scheduling based on a service level agreement. Background technique [0002] With the rise of cloud computing, more and more companies have begun to use MapReduce and Apache Hadoop to provide cloud computing services. MapReduce is a programming model proposed by Google, which is usually used for parallel operations on large-scale data sets. This model can automatically parallelize tasks on large clusters, so it is particularly suitable for big data analysis and processing. Hadoop is an open source distributed parallel programming framework that implements the distributed file system (HDFS) and MapReduce models. This framework can be deployed on common commercial hardware and has the characteristics of low cost, high efficiency, and high fault tolerance. [0003] Alt...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50G06F1/32
CPCY02D10/00
Inventor 鞠雷贾智平李萍
Owner SHANDONG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap