Speculative execution method and device for mapreduce tasks based on resource list to be released
A resource list, speculative execution technology, applied in the field of cloud computing, to achieve the effect of overall performance improvement
Active Publication Date: 2019-05-14
HUNAN UNIV
View PDF5 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
But the disadvantage of this method is that the resource currently submitting the task request is not necessarily the resource that can make the slow task complete the fastest
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0038] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings and examples.
[0039] figure 1 It is a schematic flow chart of an embodiment of the present invention, the method includes:
[0040] S101. An idle resource appears during the execution of the job, and the idle resource submits a task request to the master node;
[0041] S102. Determine whether there are tasks that have not yet started to run in the job. If there are tasks that have not yet started to run in the current job, select a task from the tasks that have not started to run to execute on idle resources, otherwise, continue to execute;
[0042] S103. Obtain required job runtime information;
[0043] S104. Find and filter out slow tasks, obtain the remaining completion time EstimatedTaskTimeLeft of all slow tasks, sort the slow task list according to the remaining completion time from large to small, and generate a slow task list CandidateTasks[N]...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The present invention discloses a to-be-released resource list based MapReduce task speculation execution method and apparatus in a heterogeneous environment, which describe the condition that free resources submit a task request to a master node in a job execution process. The method comprises: if tasks that do not start to run exist in a current job, selecting one task from the tasks, that do not start to run, to be executed on the free resources, and ending an algorithm; otherwise, generating a slow task list of the job by means of recorded job execution information; if the size of the slow task list is 0, indicating that a slow task does not exist and speculation execution does not need to be performed, and ending an algorithm; otherwise, continuously generating a to-be-released resource list; and if the size of the slow task list is N and the size of the to-be-released resource list is M, according to a backup task selection algorithm of the present invention, selecting a task M and allocating the task M to a resource that applies for a task, and ending the algorithm. Compared with a conventional speculation execution algorithm, by using the to-be-released resource list based MapReduce task speculation execution method provided by the present invention, resources that enable slow tasks to be completed more quickly can be found, so that execution efficiency is improved.
Description
technical field [0001] The invention relates to the technical field of cloud computing, in particular to a method and device for speculatively executing MapReduce tasks based on a resource list to be released in a heterogeneous environment. Background technique [0002] In recent years, the application of big data platform Hadoop technology has become more and more mature. Hadoop is an open source distributed computing platform born in the context of cloud computing and big data. It draws on Google's GFS and MapReduce technologies to enable developers to easily develop and run applications that process massive amounts of data. [0003] Task backup is the most common fault-tolerant scheduling technique in various computing systems. In Hadoop, the problem related to task backup is the task speculative execution mechanism. In a distributed cluster environment, due to program bugs, unbalanced load, or uneven resource distribution, the running speeds of multiple tasks of the sa...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50
Inventor 李智勇陈京袁廷坤陈少淼杨波李仁发
Owner HUNAN UNIV