Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and system for calculating Hadoop configuration parameters

A technology for configuring parameters and parameters, applied in the computer field, can solve the problems of low precision, insufficient granularity, large amount of calculation, etc., and achieve the effect of accurate detection data

Inactive Publication Date: 2016-06-08
SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
View PDF3 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] A method for calculating configuration parameters of Hadoop is provided, and the method for calculating configuration parameters of Hadoop solves the problems of large amount of calculation, insufficient granularity and low precision in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for calculating Hadoop configuration parameters
  • Method and system for calculating Hadoop configuration parameters
  • Method and system for calculating Hadoop configuration parameters

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0033] refer to figure 1 , figure 1 The calculation method of the configuration parameter of a kind of Hadoop that the first preferred embodiment of the present invention provides, this method is finished by computer equipment or cloud platform, and this method is as figure 1 shown, including the following steps:

[0034] Step S101, sampling the actual production data in the industrial environment to obtain a small data set of the industrial environment.

...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method and a system for calculating Hadoop configuration parameters; the method comprises the following steps: sampling actual production data under an industrial environment to obtain a small data set of the industrial environment; randomly generating the Hadoop configuration parameters, operating the small data set of the industrial environment in a Hadoop cluster, outputting a run time, taking the time as a class identification and a combination of the Hadoop configuration parameters as an input, adopting an information gain scheme and outputting important configuration parameters of the Hadoop; and adopting a genetic algorithm to iterate the obtained important parameters to search an optimal configuration combination. The technical scheme provided by the invention has the advantage of small calculated amount.

Description

technical field [0001] The invention relates to the field of computers, in particular to a calculation method and system for configuration parameters of Hadoop. Background technique [0002] Hadoop is an open source distributed computing framework, which draws on the idea of ​​MapReduce programming, simplifies data distribution, processing, calculation, and task scheduling, and has the characteristics of fault tolerance, high reliability, and scalability. Programmers only need to write Map and Reduce functions, and Hadoop will automatically distribute tasks to each node of the cluster and execute the tasks. Therefore, the framework reduces the difficulty of parallel programming, and programmers can make full use of hardware resources. At present, Hadoop has been widely used in industry and academia. [0003] However, the performance of MapReduce tasks is composed of many factors, such as the hardware environment of the physical cluster, the configuration of operating syste...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/46
CPCG06F9/465
Inventor 刘勇喻之斌须成忠
Owner SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com