Task scheduling method and device based on resource estimation in big data system
A task scheduling and data source technology, applied in the field of task scheduling based on resource estimation, can solve problems such as decreased efficiency, difficult implementation, complex task dependencies, etc., to achieve the effect of simple implementation and high reliability
Inactive Publication Date: 2017-01-25
MIGU MUSIC CO LTD
View PDF5 Cites 4 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0003] The complete task scheduling process must include at least two parts: task resource estimation and task execution planning. However, due to the huge amount of data in the big data system and the complexity of tasks, there is currently no effective way to achieve resource estimation. In actual work, the task is generally given up. Resource estimation; in terms of task execution planning, it is usually implemented based on the principle of control flow. This method has certain operability when the task scale is small, but as the number of tasks increases and task dependencies become complicated, Not only the efficiency drops but also the implementation is very difficult
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0031] The storage strategy of the big data system is to distribute as randomly and evenly as possible on each node of the cluster. Usually, task planning and scheduling are mainly based on two aspects: the time cost and storage cost required to run the task; In this case, the time cost and storage cost mainly depend on the amount of data processed by the task, the calculation logic and algorithm time complexity, and for a given task, the processing logic and algorithm complexity are determined, therefore, the time cost of the task is related to The storage cost is directly proportional to the amount of data processed by the task.
[0032] In the embodiment of the present invention, resource estimation is performed on the received tasks, and the tasks are added to the task list; current system idle resources are estimated, and the total amount of resources required by the tasks in the task list and the current system The size relationship of idle resources schedules the tasks ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a task scheduling method based on resource estimation in a big data system. The method includes the steps that a received task is subjected to resource estimation and added into a task list; current system idle resources are estimated, and the task in the task list is scheduled according to the total quantity of resources required by the task in the task list and the size relationship of the current system idle resources. The invention further discloses a task scheduling device based on resource estimation in the big data system.
Description
technical field [0001] The present invention relates to the technical field of task planning and scheduling, in particular to a task scheduling method and device based on resource estimation in a big data system. Background technique [0002] Big data systems have the characteristics of large amount of stored data, complex structure, numerous running tasks, large amount of data processed by tasks, and complex dependencies between tasks. Big data systems are very powerful in terms of computing and storage, but as far as a specific big data system is concerned, within a certain period of time, both its time resources and storage resources have been determined. Only when the tasks are scheduled and executed in a coordinated manner can the limited resources of the system be fully utilized to realize the real value of the big data system. [0003] The complete task scheduling process must include at least two parts: task resource estimation and task execution planning. However, ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F9/48G06F9/50
Inventor 朱泓钟咏曾东张聪夏峻峰李小东
Owner MIGU MUSIC CO LTD