System and method for cross-data center scheduling of mapreduce tasks based on master-slave architecture

A task scheduling and data center technology, applied in the field of cloud computing, can solve the problem that data center processing tasks are difficult to carry out, and achieve the effect of fast and effective cross-data center scheduling, efficient and stable reduction

Active Publication Date: 2015-09-30
SERVYOU SOFTWARE GRP
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] At present, Hadoop MapReduce is widely used in data analysis of data centers by many enterprises, universities, research institutes, etc., but these analysis tasks are mainly performed in a single data center, and it is difficult to carry out processing tasks between data centers

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • System and method for cross-data center scheduling of mapreduce tasks based on master-slave architecture
  • System and method for cross-data center scheduling of mapreduce tasks based on master-slave architecture
  • System and method for cross-data center scheduling of mapreduce tasks based on master-slave architecture

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] In order to illustrate the technical solution of the present invention more clearly, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0022] like figure 2 As shown, a cross-data center scheduling system for MapReduce tasks based on a master-slave architecture includes a global task scheduling center (Global Scheduling Server, GSS), n data centers numbered from 01 to N, and a client client. Each data center has a resource manager (ResourceManager) and multiple node managers (NodeManager). The global task scheduling center and the resource managers of each data center are linked by a wide area network, the client client and the global task scheduling center are linked by a wide area network, and the resource managers and node managers of each data center are linked by a local area network. The resource manager of each data center contains a GSS plugin (global task scheduling center communic...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a Map Reduce task data-center-across scheduling system and method based on a master-slave framework. The system and method are characterized in that a Map Reduce task overall task scheduling center is built and is responsible for managing resource managers of data centers and receiving Map Reduce task requests submitted by a client side Client, and the data centers meeting the requirement are selected according to a preset scheduling algorithm; the resource managers of the data centers synchronize the states of the centers and task execution information to the overall task scheduling center at regular time. According to the system and method, data-center-across scheduling of a Map Reduce task is achieved, a uniform entrance is provided for the Map Reduce task across the data centers, and the data and computing resource sharing of the data centers is effectively achieved.

Description

technical field [0001] The present application relates to cloud computing technology, in particular to a MapReduce (a programming model for parallel computing of large-scale data sets) task scheduling system and method. Background technique [0002] Cloud computing (Cloud Computing) is produced with the development of processor technology, virtualization technology, distributed storage technology, Internet technology and automatic management technology, and is built by distributed large-scale clusters and server virtualization software. At present, technologies represented by the Hadoop Project Distributed File System (Hadoop Distributed File System, HDFS for short) of the open source community Apache and the parallel programming framework Hadoop MapReduce have gradually become the mainstream technologies for massive data storage and analysis processing. Among them, Hadoop MapReduce is currently the most widely used massive data analysis technology. [0003] MapReduce syste...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48
Inventor 张未展阮建飞郑庆华董博张汉宁贺欢
Owner SERVYOU SOFTWARE GRP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap