Task scheduling method and device based on hadoop cluster

A hadoop cluster and task scheduling technology, applied in the computer field, can solve problems such as low timeliness, limited expansion space, and failure to meet user requirements, so as to avoid resource grabbing and meet the effect of large data volumes

Active Publication Date: 2019-03-01
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF7 Cites 14 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, the RDBMS network computing application system cannot meet user requirements when the amount of data increases massively.
And with the increase of data, the expansion space of RDBMS hardware is limited. After the data increases to a large enough order of magnitude, because of the bottleneck of hard disk input / output, the timeliness of processing a large amount of data is very low, resulting in RDBMS network computing. The Development Requirements of Parallel Computing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Task scheduling method and device based on hadoop cluster
  • Task scheduling method and device based on hadoop cluster
  • Task scheduling method and device based on hadoop cluster

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] The application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain related inventions, rather than to limit the invention. It should also be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

[0034] It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0035] figure 1 It shows an exemplary system architecture 100 to which the embodiment of the hadoop cluster-based task scheduling method or the hadoop cluster-based task scheduling device of the present application can be applied.

[0036] Such as figure 1 As shown, the syst...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

An embodiment of the present application discloses a task scheduling method and device based on a hadoop cluster. A specific embodiment of the method includes: deploying a plurality of virtual nodes according to a service type of each task to be scheduled, wherein each service type corresponds to at least one virtual node; receiving a task to be scheduled sent by a user, and determining a virtualnode for allocating the task to be scheduled according to the service type of the task to be scheduled; using the determined virtual node, the task to be scheduled is assigned to a proxy node corresponding to the virtual node so that the proxy node submits the task to be scheduled to the hadoop cluster, wherein each virtual node corresponds to at least one proxy node. The embodiment determines a virtual node and a corresponding proxy node for submitting a task to be scheduled to a hadoop cluster according to the service type of the task to be scheduled, and can realize the parallel computing requirements of a large number of tasks to be scheduled of different service types.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to the field of Internet technology, and in particular to a method and device for scheduling tasks based on hadoop clusters. Background technique [0002] With the rapid development of the national economy, the amount of data generated and stored in various industries has increased rapidly. "Big data" has penetrated into every industry and field and has become an important factor of production. In the prior art, a large number of enterprises use the network computing base of RDBMS (relational database management system) to store and compute massive data. [0003] However, the RDBMS network computing application system cannot meet user requirements when the amount of data increases massively. And with the increase of data, the expansion space of RDBMS hardware is limited. After the data increases to a large enough order of magnitude, because of the bottleneck of hard dis...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/50
CPCG06F9/4806G06F9/5011
Inventor 杨泽森
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap