Method and device for dispatching tasks

A task scheduling and task technology, which is applied in the directions of multi-program device, program startup/switching, resource allocation, etc., can solve problems such as long running time of jobs and inability to guarantee parameter settings, so as to reduce subjective deviation, improve operating efficiency, shorten The effect of completion time

Active Publication Date: 2014-04-16
INSPUR BEIJING ELECTRONICS INFORMATION IND
View PDF4 Cites 18 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the current parameter setting mainly depends on user experience. For users who lack rich experience, reasonable parameter setting cannot be guaranteed, resulting in job running time significantly greater than the theoretical optimal value.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for dispatching tasks
  • Method and device for dispatching tasks
  • Method and device for dispatching tasks

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] In order to facilitate the understanding of those skilled in the art, the present invention will be further described below in conjunction with the accompanying drawings, which cannot be used to limit the protection scope of the present invention.

[0049] see figure 1, the present invention proposes a task scheduling method, comprising:

[0050] Step 100, setting corresponding CPU resources and memory resources for any task of the user, and predicting the completion time of the task.

[0051] In this step, the user uploads the job to the YARN platform, and the YARN platform splits the job into N identical tasks, where N is a natural number greater than or equal to 1. YARN's splitting rules for jobs are prior art and are not within the protection scope of the present invention.

[0052] see figure 2 , in this step, the method for predicting the completion time of the task includes:

[0053] Step 200, generating an instance database, which stores the characteristic ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a method for dispatching tasks. The method comprises the steps that a corresponding CPU resource demand and a corresponding internal storage resource demand are set for any task of a user, and the time when the task is finished is predicted; the number of tasks operated by the user at the same time is predicted so that the dominated share ratios of all users in a YARN platform are identical; the total time spent on a job is calculated according to the number of the total tasks of the user, the predicted time when the tasks are finished and the number of the tasks capable of being operated by the user at the same time; the set CPU resource demand and the internal storage resource demand are changed within the preset CPU resource demand range and the internal storage resource demand range, and the steps are executed repeatedly until the worked out total time spent on the operation is finished is minimum. The subjective deviation of setting the CPU resource demand and the internal storage resource demand of the user when the YARN platform submits the job can be reduced, the job finishing time can be shortened, and the overall operating efficiency of the YARN platform is improved.

Description

technical field [0001] The invention relates to the job optimization technology of the YARN platform in the second generation of Hadoop, in particular to a task scheduling method and device. Background technique [0002] Hadoop is currently the most popular big data processing tool. It implements a distributed storage and computing system, especially suitable for terabyte (TB, Terabyte) and PB (Petabyte) level data processing, and with the help of the MapReduce framework, users can easily write distributed programs, replacing traditional The business is migrated to the distributed platform. [0003] Most of the current commercial products are based on the first-generation Hadoop technology, and there are problems such as single-point bottlenecks, inflexible resource allocation, and single programming frameworks in use. The second-generation Hadoop under development, in order to overcome the above shortcomings, has built a new underlying platform YARN, which is responsible ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/50
Inventor 刘璧怡
Owner INSPUR BEIJING ELECTRONICS INFORMATION IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap