Task allocation method and device based on MapReduce programming framework
A task allocation and task technology, applied in the field of network communication, can solve problems such as no consideration, and achieve the effects of preventing jitter, avoiding repeated transfers, and avoiding overload.
Active Publication Date: 2012-09-05
HUAWEI TECH CO LTD
View PDF0 Cites 1 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] Since the technical solution provided by the prior art is to be allocated to each node (computer) for calculation and processing at one time, it only considers the task allocation, and does not consider the dynamic changes of node (computer) resources during the task execution process. load imbalance problem
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0024] The embodiment of the present invention provides a task allocation method based on the MapReduce programming architecture, the method includes, when it is judged that it is idle, sending an idle message, and receiving a response message of the idle message, the response message includes the amount of tasks to be transferred and The node address corresponding to the task amount; when it is judged that the task amount to be transferred is less than the remaining capacity, the node address corresponding to the task amount to be transferred is stored in the overload node list, and sent to a node in the overload node list The address sends a task transfer request message, receives the transferred task and performs calculation processing; or, when it is judged that the task is overloaded and no task transfer is performed, reply to the received response message of the idle message, the response message includes the amount of tasks to be transferred and the node address; after r...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The implementation mode of the invention provides a task allocation method and a device based on a MapReduce programming framework. The method and the device belong to the field of network communication. The method comprises the following steps: data partitioned logic is split into a plurality of logic pieces in advance, and the logic pieces are taken as elementary units. In an idle state, an idle message is sent, a response message of the idle message is received, and the response message comprises task load which needs to be transferred and node address corresponding to the task load; the node addresses corresponding to the task load which needs to be transferred are stored into an overload node list, a task transferring request message is sent, a transferring task is received, and calculation is carried out. When a plurality of Reduce tasks are processed, processing results are merged according to the quantity of files which are specified by a user and need to be output. The methodand the device provided by the concrete implementation mode have the advantages of having balanced load, avoiding repeated task transfer and avoiding shake of the system.
Description
technical field [0001] The present invention relates to the field of network communication, in particular to a task allocation method and device based on the MapReduce programming architecture. Background technique [0002] With the rapid development of high-performance applications and computing requirements, a single high-performance computer can no longer solve some ultra-large-scale application problems, which requires the combination of multiple computer resources to form a computer cluster to jointly solve large-scale application problems. Parallel programming technology can effectively develop the computing power of parallel computers, especially cluster computers. It is a bridge between hardware and software, and an interface between low-level implementation and high-level abstraction of parallel computing. [0003] A task allocation method based on the MapReduce programming architecture provided by the prior art specifically includes dividing the calculation task of...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08H04L29/06
Inventor 李丽娟陈浩华严哲峰
Owner HUAWEI TECH CO LTD