A method for adaptive allocation of Hadoop cluster resources

A hadoop cluster and allocation method technology, applied in resource allocation, multi-program device, program control design, etc., can solve problems such as insufficient resource division

Active Publication Date: 2020-12-29
UNIV OF SCI & TECH BEIJING
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to provide a Hadoop cluster resource adaptive allocation method to solve the problem of insufficient fineness of resource division in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method for adaptive allocation of Hadoop cluster resources
  • A method for adaptive allocation of Hadoop cluster resources
  • A method for adaptive allocation of Hadoop cluster resources

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0053] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will describe in detail with reference to the drawings and specific embodiments.

[0054] Aiming at the problem that the existing resource division is not refined enough, the present invention provides a Hadoop cluster resource self-adaptive allocation method.

[0055] In order to better understand the Hadoop cluster resource adaptive allocation method described in this embodiment, first briefly explain the slave node, master node, Map task, and Reduce task:

[0056] 1. The slave node is equivalent to a computing point, has a network connection, and can independently process the tasks issued by the master node and the resource manager. A server can deploy one slave node, or multiple slave nodes;

[0057] 2. The master node is responsible for job classification, resource and task scheduling, and the execution of tasks is carried out on...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a Hadoop cluster resource self-adaptive allocation method, which can make the cluster operation more efficient. The method includes: determining the type of the job submitted by the user according to the preset job type classification rules, wherein each job can be split into N tasks to realize distributed parallel computing; if the type of the job submitted by the user is CPU type job, I / O type job or important job, according to the type of job submitted by the user, determine the weight ratio column parameter of the slave node, wherein, the weight ratio column parameter of slave node i is equal to the weight value of slave node i and the cluster The ratio of the sum of the weights of all slave nodes in , the weight of slave node i is used to measure the performance of slave node i; according to the weight ratio of each slave node, assign a corresponding proportion of task requests to each slave node. The invention relates to the fields of big data and cloud computing.

Description

technical field [0001] The invention relates to the fields of big data and cloud computing, in particular to a Hadoop cluster resource adaptive allocation method. Background technique [0002] With the popularity of large-scale parallel distributed processing systems, especially the widespread application of cluster systems, what effective scheduling strategy to adopt to balance the load of each node and improve the utilization of the entire system resources has become the focus of research and hotspot. [0003] In recent years, novel, excellent and effective load balancing algorithms have become one of the research hotspots of research institutions at home and abroad. Among them, distributed heterogeneous clusters generally have load balancing problems, and the Hadoop platform itself does not have the ability to detect node performance. Although the resource management system YARN of Hadoop clusters has a scheduling strategy for load imbalance, it is too simple and not sui...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50H04L29/08
CPCG06F9/5083H04L67/1001
Inventor 李林林张勇军
Owner UNIV OF SCI & TECH BEIJING
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap