HBase hotspot eliminating method and system

A technology to eliminate systems and hot spots, applied in special data processing applications, instruments, electrical digital data processing, etc., can solve problems such as difficulty in eliminating HBase hot spots

Active Publication Date: 2015-09-30
SHANGHAI CTRIP COMMERCE CO LTD
View PDF3 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved in the present invention is how to overcome the defect that it is difficult to eliminate hot spots in HBase in the prior art, and provide a HBase hot spot elimination method and system that can automatically eliminate hot spots

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • HBase hotspot eliminating method and system
  • HBase hotspot eliminating method and system
  • HBase hotspot eliminating method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0072] A HBase hotspot elimination method, such as figure 1 shown, including:

[0073] Step 101, setting the initial value of the number of exchanges of each Region in the HBase table of the cluster to 0.

[0074] Step 102, detect the cluster, use the statistical method to statistically analyze the HBase table of the cluster, obtain the data distribution and variance, determine a first request volume threshold according to the average load of the cluster, then first find the region exceeding the threshold, and then according to these The regionServer where the region is located is classified and added to the list of hot machines. Moreover, the number of regions to be operated can also be limited here. For example, if 20 hotspot regions are detected, the first 10 hotspot regions can be processed first;

[0075] Step 103, judging whether the number of Region hotspots included in the hotspot machine list is 0, if yes, then end the process, if not, then execute step 104. That i...

Embodiment 2

[0099] The HBase hotspot elimination method of the present embodiment is basically the same as the HBase hotspot elimination method of embodiment 1, and the difference is that the HBase hotspot elimination method of the present implementation also includes the following steps:

[0100] like image 3 As shown, replace step 101 with the following steps:

[0101] Step 1011, setting the initial value of the number of exchanged times of each Region in the HBase table of the cluster to 0, and setting the initial value of the number of times of detection of the cluster to 0.

[0102] Step 1012. Determine whether the number of detected clusters reaches a detection threshold. If so, end the process. If not, execute step 102'. The detection threshold can be freely set by the technician according to actual needs. If the technician only wants to detect a cluster once, the detection threshold can be set to 1. If the technician wants to detect the cluster multiple times, the detection thre...

Embodiment 3

[0114] The HBase hotspot elimination method of the present embodiment is basically the same as the HBase hotspot elimination method of embodiment 1, and the difference is that, as Figure 5 As shown, the HBase hotspot elimination method of this implementation also includes performing the following steps after step 101:

[0115] Step 001. Statistically analyze the HBase table of the cluster using a statistical method to obtain data distribution and variance, determine the second request volume threshold according to the average load of the cluster, and obtain the RegionServer hotspot of the HBase table of the cluster, where the RegionServer hotspot is the request volume RegionServers that exceed the second request threshold.

[0116] Step 002. Calculate the variance of the N Regions with the highest traffic hotspots in the RegionServer, where N is a positive integer. The value of N can be freely set by technicians according to actual conditions.

[0117] Step 003. Determine w...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an HBase hotspot eliminating method and system. The HBase hotspot eliminating method comprises steps as follows: a cluster is detected, Region hotspots in the cluster are added to a hotspot machine list; whether the number of the Region hotspots included in the hotspot machine list is zero or not is judged, if yes, a process is finished, and otherwise, one to-be-eliminated Region hotspot is selected; the to-be-eliminated Region hotspot is divided into two Regions, or one Region Server without the Region hotspots is selected from the cluster to be exchanged with the to-be-eliminated Region hotspot; the to-be-eliminated Region hotspot is removed from the hotspot machine list. According to the HBase hotspot eliminating method and system, the Region hotspots and Region Server hotspots can be detected and eliminated automatically, the data request time is shortened, the normal operation of the whole system is guaranteed, and the method and the system have the advantages of rapidness and efficiency.

Description

technical field [0001] The invention relates to the field of Internet big data, in particular to a HBase (a distributed, column-oriented open source database) hotspot elimination method and system. Background technique [0002] The HBase table includes multiple Regions, and certain data is stored in each Region. A RegionServer in a cluster may be assigned to one or more Regions. Due to the difference in the amount of data requests in each Region, one or more Region hotspots may appear. The so-called Region hotspots refer to Regions whose request amount exceeds a first request amount threshold. The emergence of these Region hotspots will lengthen the data request time, greatly affect the stability and availability of the entire cluster, and may even cause system paralysis in severe cases. [0003] At present, HBase does not have a tool for Region hotspot discovery and automatic elimination. It can only use the built-in split (a statement in Java) method to split the specifi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/217
Inventor 李亚锋刘钰帆
Owner SHANGHAI CTRIP COMMERCE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap