Hadoop Yarn (Yet another Resource Negotiator) network bandwidth resource scheduling and isolating method and device

A network bandwidth and resource scheduling technology, applied in the field of big data computing, can solve the problems of not supporting network bandwidth management, sharing resources without sharing occupation, affecting cluster robustness, etc.

Inactive Publication Date: 2017-12-15
NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
View PDF5 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In the field of big data, Yarn needs to become a general resource management system, establish a comprehensive resource management and control network, and manage various resources in the cluster, including memory, CPU, network bandwidth, disk, etc., but currently, Yarn only supports The management of memory and CPU does not support the management of network bandwidth; moreover, since there are still many resources in the Yarn cluster that are n

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hadoop Yarn (Yet another Resource Negotiator) network bandwidth resource scheduling and isolating method and device
  • Hadoop Yarn (Yet another Resource Negotiator) network bandwidth resource scheduling and isolating method and device
  • Hadoop Yarn (Yet another Resource Negotiator) network bandwidth resource scheduling and isolating method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0029] figure 1 is a schematic diagram of a system structure of a Yarn cluster according to an embodiment of the present invention.

[0030] The Yarn cluster includes: resource scheduler RM, node manager NM, application manager AM and client client.

[0031] The NM, the AM, the Client, and the RM implement a communication connection through RPC (Remote Procedure Call Protocol, remote procedure call protocol). Each node in the system includes: one NM and at least one AM. This node has computing, storage and other functions.

[0032] Client is used to submit jobs to AM. Include multiple tasks in a job.

[0033] AM is used to request resources required by the job from RM. The resources req...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Hadoop Yarn (Yet another Resource Negotiator) network bandwidth resource scheduling and isolating method and device. The scheduling method is performed at an RM (Resource Manager) side and comprises the steps of receiving work resource requests sent by an AM (Application Master), wherein the work resource requests comprise network bandwidth resource demand quantity of each task in work; and correspondingly allocating a container to each task through adoption of a preset fair scheduling algorithm according to the network bandwidth resource demand quantity of each task, wherein the network bandwidth resource demand quantity of the corresponding task is packaged in each container. The isolating method is performed at an NM (Node Manager) side and comprises the steps of obtaining a plurality of containers from the Am, wherein the network bandwidth resource demand quantity of the corresponding task is packaged in each container; and isolating a network bandwidth resource occupied by each task according to the network bandwidth resource demand quantity of each task. According to the methods and the devices, the scheduling for the network bandwidth resources is increased in a Yarn cluster, so the mutual interference among the tasks resulting from competition for the network bandwidth resources can be avoided.

Description

technical field [0001] The invention relates to the technical field of big data computing, in particular to a Hadoop Yarn network bandwidth resource scheduling and isolation method and device. Background technique [0002] Yarn (Yet Another Resource Negotiator, another resource coordinator) is a general resource management system responsible for resource allocation and task scheduling of Yarn clusters. Yarn mainly includes three components: RM (ResourceManager, resource manager), NM (Node Manager , Node Manager) and AM (Application Master, Application Manager). [0003] In Yarn, the unit of representation of resources is a container (Container), which is a part decomposed from MRv1. Container is an abstraction of resources in Yarn, which encapsulates a certain amount of computing resources and storage resources on a node, such as CPU and memory. Container is the division unit of dynamic resources. AM applies for resources from RM, and the scheduler in RM allocates Containe...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L12/917H04L47/76
CPCH04L47/76
Inventor 徐晓燕樊鑫鑫臧天宁李高超周渊
Owner NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap