Resource allocation method, related equipment and system

A technology of distributed systems and resources, applied in the fields of related equipment and systems, and resource allocation methods, can solve the problems of waste of computing resources, different computing resources, etc.

Active Publication Date: 2018-04-06
HUAWEI TECH CO LTD
View PDF8 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0011] The defect of the existing technology is that the execution of some Reduce tasks is divided into multiple calculation phases, and the complexity of the algorithm used in each calculation phase is different. Correspondingly, the calculation required for

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Resource allocation method, related equipment and system
  • Resource allocation method, related equipment and system
  • Resource allocation method, related equipment and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0075] The technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings.

[0076] See image 3 , image 3 It is a schematic structural diagram of a MapReduce-based distributed system 30 provided by an embodiment of the present invention. The system 30 includes a management node 301 and a computing node 302. In practical applications, there may be multiple management nodes 301 and multiple computing nodes 302. The embodiment of the present invention uses a management node 301 as an example to describe how to execute Map tasks and Reduce tasks in MapReduce based on the system 30 . It should be noted that the management node 301 and the computing node 302 in the embodiment of the present invention can be two types of nodes in the Spark architecture. Currently, the Spark architecture has the following modes: Standalone mode, Spark On YARN mode and Spark On Mesos mode , no matter which mode there is ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a resource allocation method, related equipment and a distributed system based on mapping/reduction MapReduce. The system includes a management node and a target calculation node, the management node is used for obtaining M calculation tasks, and a resource assessment model is built according to the data size of the M calculation tasks; information of onepart of calculation tasks in the M calculation tasks and the resource assessment model are sent to a target calculation node; the target calculation node is used for receiving the information, sent by the management node, of the part of calculation tasks and the resource assessment model; before each calculation state of the target task is executed, input data of the calculation stage is substituted into the resource assessment model so as to calculate the resource size of the calculation resource required in the calculation stage; through the calculation resource with the resource size in apreset resource pool, the input date is calculated. According to the resource allocation method, the related equipment and the distributed system, the utilization ratio of the calculation resource canbe improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a resource allocation method, related equipment and a system. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation, which can perform distributed processing on large amounts of data. Its core design is a distributed file system (English: Hadoop Distributed File System, referred to as: HDFS) and a mapping / reduction MapReduce programming model. HDFS provides storage for massive data, while MapReduce provides calculation for massive data. Normally, Hadoop's MapReduce processes a set of input key-value pairs (key / value pairs), and finally outputs a set of key-value pairs after being processed by a user-specified MapReduce function. MapReduce actually defines a Map function interface and a Reduce function interface. The Map function is used to convert input records to obtain intermediate results, and the Reduce func...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50
CPCG06F9/5083G06F9/5027G06F9/505G06F9/5055G06F9/5038
Inventor 黄国位邓利群魏建生
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap