Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Task scheduling algorithm based on MapReduce

A technology of task scheduling and scheduling algorithm, applied in the direction of program startup/switching, multiprogramming device, etc.

Active Publication Date: 2014-03-12
LANGCHAO ELECTRONIC INFORMATION IND CO LTD
View PDF5 Cites 43 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

It can overcome many problems existing in the existing scheduling algorithm, effectively solve the problems of local computing and small job processing, and take into account the data skew on the nodes, thus balancing the task allocation on the nodes and improving the scheduling performance of the cluster platform

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Task scheduling algorithm based on MapReduce
  • Task scheduling algorithm based on MapReduce
  • Task scheduling algorithm based on MapReduce

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0069] A task scheduling algorithm based on MapReduce. In a heterogeneous cluster environment, the multi-task scheduling algorithm based on the ant colony algorithm measures the processing performance of the computing nodes, according to the new task target transfer function and the new node update rules, according to The local computing principle distributes tasks to individual computing nodes.

Embodiment 2

[0071] On the basis of Embodiment 1, the measurement of the processing performance of computing nodes described in this embodiment mainly measures the initial processing capability of the node and the target transition probability of tasks assigned to the node, wherein the initial processing capability of the node is based on the processing speed, Memory capacity, number of CPUs, and network transmission bandwidth are comprehensively measured, and thresholds are set for these four metrics. If the thresholds are exceeded, the thresholds are used for calculation; in task scheduling, a scheduler is set up to be responsible for Computes the initial transition probabilities of task assignments to requesting nodes.

Embodiment 3

[0073] On the basis of Embodiment 2, the initial processing capability of the node depends on the initial pheromone of the node, which is calculated and determined by formula 1.1.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to task scheduling algorithms in a very important programming computing framework MapReduce in the current field of large data, and discloses a task scheduling algorithm based on MapReduce. According to the task scheduling algorithm, under the heterogeneous cluster environment, tasks are assigned to all computing nodes on the basis of a multitask scheduling algorithm of an ant colony algorithm through computing node processing capacity measuring according to a new task object transition function, a new node updating rule and a local computing rule. According to the task scheduling algorithm, on the basis of the classical ant colony algorithm, large-scale optimization is performed, the multitask scheduling algorithm under the heterogeneous cluster environment is provided, testing and performance analysis of scenes such as small operation, load and local are performed on an open source Hadoop platform, and results show that the execution efficiency and the task balance are greatly improved.

Description

technical field [0001] The present invention relates to a task scheduling algorithm in MapReduce, a very important programming computing framework in the current big data field, and in particular relates to a dynamic copy management method based on HDFS. technical background [0002] As a technology for processing large-scale data sets, MapReduce was first proposed by Google in 2007, and has received extensive attention from academia and industry. At present, the parallel programming model of MapReduce has become one of the key technologies integrated into cloud products by major IT vendors, and open source products are constantly being put into this industry, such as open source cloud systems Hadoop, Sector&Sphere, etc. In recent years, MapReduce has become the mainstream technology in the field of cloud computing, and has also become a research hotspot in scientific research institutions, open source organizations and Internet companies, and was included in the top ten eme...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/48
Inventor 孟祥飞吴楠邓鹏飞宗栋瑞邓强
Owner LANGCHAO ELECTRONIC INFORMATION IND CO LTD
Features
  • Generate Ideas
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com