Configuration method and apparatus for related parameters of MapReduce application

A technology related to parameters and configuration methods, applied in the direction of program control devices, special data processing applications, etc., can solve problems such as the inability to achieve optimal utilization of system resources, achieve the effects of reducing configuration burden, realizing localized processing, and reducing network overhead

Inactive Publication Date: 2016-02-03
IBM CORP
View PDF4 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

For the existing distributed file system, the relevant parameters of the MapReduce job are configured by the system administrator based on experience. However, as the job, data, and cluster characteristics change, a set of general manual configuration cannot maximize the utilization of system resources. excellent

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Configuration method and apparatus for related parameters of MapReduce application
  • Configuration method and apparatus for related parameters of MapReduce application
  • Configuration method and apparatus for related parameters of MapReduce application

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0012] Preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0013] figure 1 A block diagram of an exemplary computer system / server 12 suitable for use in implementing embodiments of the invention is shown. figure 1 The computer system / server 12 shown is only an example and should not impose any limitation on the functions and scope of use of the embodiments of the present invention.

[0014] Such as figure 1 As shown, computer system / server 12 takes the form of a general purpose computing device. ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a distributed file system and provides a configuration method and apparatus for related parameters of a MapReduce application. The method comprises: receiving a processing request of a first MapReduce operation; obtaining operation feature attributes of a historical MapReduce operation; finding operation feature attributes of a second MapReduce operation related with the first MapReduce operation from the operation feature attributes of the historical MapReduce operation; and configuring related parameters of the first MapReduce operation according to the operation feature attributes of the second MapReduce operation. According to the parameter configuration method, the network overhead of file transmission can be effectively reduced, the parameter configuration method for the MapReduce operation subjected to localization processing is realized as far as possible, and the utilization rate of system resources is effectively increased while the configuration burden of administrators is reduced.

Description

technical field [0001] The present invention relates to a distributed file system, and more particularly, to a method and device for configuring relevant parameters of a MapReduce application based on a distributed file system. Background technique [0002] Distributed File System (Distributed File System) means that the physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network. The design of the distributed file system is based on the client / server model. A typical network may include multiple servers accessed by multiple users. MapReduce is a software architecture proposed by Google for large-scale parallel programming. Since the MapReduce architecture can realize parallel computing of large-scale data sets (greater than 1TB), and achieve scalability by distributing large-scale operations on data sets to multiple nodes on the network for parallel computing, it i...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/44
CPCG06F16/182
Inventor 邹嘉史巨伟郑勇王晨刘杰
Owner IBM CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap