Dynamic MapReduce dispatching method and system based on task type

A scheduling method and task type technology, applied in the direction of multi-programming devices, resource allocation, etc., can solve the problem that I/O resources are not fully utilized, and achieve the effect of improving throughput

Inactive Publication Date: 2013-10-23
HARBIN INST OF TECH SHENZHEN GRADUATE SCHOOL
View PDF5 Cites 25 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Imagine the worst case: scheduling all CPU-intensive tasks to the same worker node or scheduling all I / O-intensive tasks to the same node, so that the I / O resources of the worker nodes running CPU-intensive tasks are reduced. underutilized

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Dynamic MapReduce dispatching method and system based on task type
  • Dynamic MapReduce dispatching method and system based on task type
  • Dynamic MapReduce dispatching method and system based on task type

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] Such as figure 1 As shown, the present invention discloses a dynamic MapReduce scheduling method based on task types, including the following steps:

[0029] In step S1, enter the waiting queue, and all work tasks submitted to the work node cluster enter the waiting queue first, and the work tasks are partially scheduled to the work node cluster by the waiting queue;

[0030] In step S2, work tasks are classified, and work tasks are classified into CPU-intensive and I / O-intensive according to the prediction mechanism;

[0031] In step S3, the work task is migrated, and the work task is migrated to the CPU-intensive queue or the I / O-intensive queue according to the prediction result obtained in step B;

[0032] In step S4, work task scheduling, CPU-intensive queue and I / O-intensive queue are independently scheduled, and the work task is scheduled to the work node cluster to execute the task.

[0033] In step S1, the waiting queue does not directly schedule the task to the work nod...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a dynamic MapReduce dispatching method and system based on a task type. The dynamic MapReduce dispatching method based on task type comprises the steps as follows: A. entering the waiting queue, B. classifying the work task, C. moving the work task, and D. dispatching the work task, independently dispatching a CPU (Central Processing Unit) intensive queue and an I/O intensive queue respectively, dispatching the work task to a work node cluster, and executing the task. The method and the system have the benefits that the dynamic MapReduce dispatching method based on the task type arranges queue to the task with various types respectively and independently dispatching the queues through the prediction to the work task type. The dynamic MapReduce dispatching method based on the task type improves the throughput of the cluster under task environment with various types.

Description

Technical field [0001] The invention relates to a MapReduce scheduling method, in particular to a dynamic MapReduce scheduling method and system based on task types. Background technique [0002] With the continuous growth of Internet data scale, the services provided by the Internet need to be able to store and process massive amounts of data. MapReduce is a parallel programming model that is used for large-scale data parallel operations and can be used to build data centers. It is the current leading parallel computing solution. In the current Internet environment, Internet services are emerging in an endless stream, and the same data center is likely to provide multiple services at the same time, which leads to the data center may be running different types of tasks at the same time. For example, the data center is running a CPU-intensive task such as video transcoding, and at the same time, it is also running an I / O-intensive task such as audio and video streaming. [0003] T...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
Inventor 何震宇薛鸿杰盛义涛葛百根史梦龙胡文毅
Owner HARBIN INST OF TECH SHENZHEN GRADUATE SCHOOL
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap