Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Hbase configuration parameter automatic tuning method, Hbase configuration parameter automatic tuning device and user equipment

A configuration parameter and automatic technology, applied in the database field, can solve problems such as time-consuming, low efficiency and poor manual tuning, and achieve the effect of improving operation efficiency and automatic tuning of configuration parameters

Inactive Publication Date: 2018-06-29
SHENZHEN INST OF ADVANCED TECH
View PDF3 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At present, the effect of parameter tuning depends on the specific implementer. Engineers with rich experience and a deep understanding of the HBase architecture can achieve better results, but for engineers with general technical backgrounds, the effect of parameter tuning is often not good.
Moreover, manual tuning is inefficient and time-consuming
The method of manual tuning is usually to try a group of different parameter values. In the actual implementation process, it is very unrealistic to try a group of parameters

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hbase configuration parameter automatic tuning method, Hbase configuration parameter automatic tuning device and user equipment
  • Hbase configuration parameter automatic tuning method, Hbase configuration parameter automatic tuning device and user equipment
  • Hbase configuration parameter automatic tuning method, Hbase configuration parameter automatic tuning device and user equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0056] see figure 1 , figure 1 It is a schematic flowchart of a method for automatically tuning HBase configuration parameters disclosed in an embodiment of the present invention. Such as figure 1 As shown, the HBase configuration parameter automatic tuning method may include the following steps:

[0057] 101. Obtain HBase database configuration parameters, and randomly generate multiple sets of HBase configuration files according to the HBase configuration parameters;

[0058] First obtain the HBase database configuration parameters, and randomly generate multiple sets of HBase configuration files according to the HBase configuration parameters, specifically: select several HBase configuration parameters according to the HBase reference manual, and determine the upper limit, lower limit and step size of each parameter. Each parameter is randomly selected to form multiple groups, for example, 600 groups of randomly generated HBase configuration files are formed. Some confi...

Embodiment 2

[0070] see figure 2 , figure 2 It is a schematic flow chart of another HBase configuration parameter automatic tuning method disclosed in the embodiment of the present invention. Such as figure 2 As shown, the HBase configuration parameter automatic tuning method may include the following steps:

[0071] YCSB (Yahoo! Cloud Serving Benchmark) is a set of benchmarking tools commonly used in academia and industry. Its function is to simulate the real application operation HBase cluster. YCSB defines five basic operations, namely read, write, update, insert and scan. Each load consists of four attributes, including: (1) the type of basic operation; (2) the proportion of each operation; (3) the record size corresponding to the operation, which defaults to 1024KB; (4) the size of each operation sequence etc. At the end of the test, YCSB reports the performance of the program in log form, including throughput and latency.

[0072] In this embodiment, it is first necessary to...

Embodiment 3

[0097] see image 3 , image 3 It is a schematic structural diagram of an automatic tuning device for HBase configuration parameters disclosed in an embodiment of the present invention. Such as image 3 As shown, the HBase configuration parameter automatic tuning device may include:

[0098] HBase configuration file generating module 301, used to obtain HBase configuration parameters, and randomly generate multiple groups of HBase configuration files according to the HBase configuration parameters; specifically: select several HBase configuration parameters according to the HBase reference manual, and determine the upper limit of each parameter value, lower limit value and step size. Each parameter is randomly selected to form multiple groups, for example, 600 groups of randomly generated HBase configuration files are formed. According to the upper limit value, lower limit value and step size, the random function is used to randomly select values ​​to generate the HBase co...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a HBase configuration parameter automatic tuning method, a HBase configuration parameter automatic tuning device and user equipment, the method comprises the following steps: after acquiring a HBase database configuration parameter, firstly generating multiple groups of HBase configuration files randomly according to the HBase configuration parameter, then, simulating load,completing the same number of parameter configuration according to the generated multiple groups of HBase configuration files, generating a parameter throughput matrix, constructing a performance model based on a random forest according to the parameter throughput matrix, finally, inputting the performance model into a genetic algorithm, and acquiring an optimal HBase configuration parameter through iterative computation of the genetic algorithm. Obviously, by application of the embodiment of the invention, automatic tuning of HBase configuration parameters can be realized without dependenceon personal capability and experience of an implementer, and working performance is improved greatly.

Description

technical field [0001] The present invention relates to the field of databases, in particular to a method and device for automatic tuning of HBase configuration parameters, and user equipment. Background technique [0002] HBase (Hadoop Database, HBase) database is a high-reliability, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap PC Server. HBase is widely used in large Internet companies at home and abroad, and is also widely adopted by many small and medium-sized IT companies. [0003] HBase parameter tuning is an important activity in the process of system deployment, operation and maintenance. The HBase architecture is complex in design, involving a series of modular components. These components include remote procedure calls responsible for network communication, caches and buffers, etc. The behavior of these components and the interactions between t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L12/24H04L29/08
CPCH04L41/0823H04L67/1097
Inventor 熊文喻之斌须成忠
Owner SHENZHEN INST OF ADVANCED TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com