A control method, device and hadoop system

A control method and technology of computing nodes, applied in the field of data processing, can solve problems such as affecting the processing efficiency of computing tasks, and achieve the effect of satisfying data locality and ensuring processing efficiency

Active Publication Date: 2019-02-05
LENOVO (BEIJING) LTD
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] However, in the virtualized Hadoop system, the host computer of the virtual machine runs the computing task, and the virtual machine of the same rack does not necessarily correspond to the same rack, which will affect the processing efficiency of the computing task.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A control method, device and hadoop system
  • A control method, device and hadoop system
  • A control method, device and hadoop system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0062] The technical scheme of the present application is mainly used in the virtualized Hadoop system. Since the Hadoop system is a distributed system, it usually adopts a distributed deployment mode of multiple computers. The multiple computers that deploy the Hadoop system are called Hadoop clusters. Computers that compute tasks are called computing nodes; computers that store data are called storage nodes. A virtualized Hadoop system means that computing nodes and storage nodes ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a control method and device and a Hadoop system, and is applied to a virtual Hadoop system. Calculation nodes and storage nodes in the Hadoop system are deployed in different virtual machines, and the virtual machines virtualized by each host machine in the Hadoop system at least include the virtual machine for deploying the calculation nodes, and the virtual machine for deploying the storage nodes. The method comprises the following steps: obtaining a calculation task; looking up a first storage node which stores data required by the calculation task; judging whether the resources of the first calculation node in the first host machine corresponding to the first storage node meet the operation requirements of the calculation task or not; when the resources of the first calculation node do not meet the operation requirements, regulating the resources, which are occupied by the first calculation node, of the first host machine; and distributing the calculation task to the first calculation node which succeeds in resource regulation to operate. The embodiment of the invention guarantees the processing efficiency of the calculation task.

Description

technical field [0001] The present application relates to the technical field of data processing, and more specifically relates to a control method, device and Hadoop system. Background technique [0002] The Hadoop system is a distributed system. Generally, the computers that run computing tasks in the Hadoop system are called computing nodes; the computers that store data are called storage nodes. [0003] Traditionally, the Hadoop system is directly deployed on a physical machine, and computing nodes and storage nodes are deployed on one physical machine. When the Hadoop system allocates computing tasks, it first assigns the computing tasks to the physical machine that stores the data required by the computing tasks, which is called data locality. Data locality can save data processing time; Tasks are preferentially assigned to run on the physical machine in the same rack as the physical machine storing the data required by the computing task to ensure processing efficie...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50G06F9/455
Inventor 孙瑞琦杨杰高瞻贺志强
Owner LENOVO (BEIJING) LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap