An optimal localization task scheduling method based on mapreduce

A task scheduling and task technology, applied in the computer field, can solve the problems of inability to obtain the degree of data localization and the overall execution time of the job, the scope of application is not wide, the applicability is not wide, etc., to shorten the overall execution time, improve the degree of parallelism, The effect of reducing network bandwidth usage
CN104461748BInactive Publication Date: 2017-06-09UNIV OF ELECTRONICS SCI & TECH OF CHINA
3 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN ยท China
Patent Type
Patents(China)
Current Assignee / Owner
UNIV OF ELECTRONICS SCI & TECH OF CHINA
Publication Date
2017-06-09
Estimated Expiration
Not applicable ยท inactive patent

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention proposes a MapReduce task scheduling algorithm that can work simultaneously in homogeneous and heterogeneous cluster environments, belonging to the field of computer technology. The scheduling algorithm can comprehensively consider the processing performance of each computing node in the cluster, abstract the computing nodes and computing tasks into a bipartite graph, and form the final global task scheduling scheme by appropriately expanding the bipartite graph and combining the KM weighted optimal matching algorithm. Experimental data show that the scheduling algorithm can improve the data localization degree in the Map stage to nearly 100%, and the overall execution time of the MapReduce job can be reduced by 67.1%.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the technical field of computers, and in particular relates to an optimal localization task scheduling method based on MapReduce. Background technique

[0002] MapReduce task scheduling directly affects the execution time of MapReduce computing jobs, and an efficient scheduling algorithm can effectively improve job execution efficiency.

[0003] The degree of data localization directly affects the execution efficiency of MapReduce jobs. The MapReduce job is mainly composed of the Map stage and the Reduce stage. The intermediate output data generated by the computing nodes in the Map stage needs to be transmitted through the network to the computing nodes in the Reduce stage as their input data. This intermediate stage is called Shuffle. The resource consumption of network bandwidth brought about by the data transmission in the Shuffle stage and the persistent storage of data in the Reduce stage is inevitable. Under the conditi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More