HBase data cleaning method and device

A technology of data cleaning and data location, which is applied in the field of HBase data cleaning, can solve the problems of high pressure on the master node and affect the performance of HBase, and achieve the effect of reducing operating pressure, improving overall performance and operating stability, and improving efficiency.

Pending Publication Date: 2021-10-19
INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] At present, the data cleaning method for HBase is mainly to use the cleaning operation truncate to clean the table of HBase. However, since the master node Master has only two instances, one master and one backup, it cannot be expanded. Therefore, when using the cleaning operation truncate to clean HBase Frequent table clearing operations will easily cause greater pressure on the master node, which will affect the overall performance of HBase

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • HBase data cleaning method and device
  • HBase data cleaning method and device
  • HBase data cleaning method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

[0047] It should be noted that the HBase data cleaning method and device disclosed in the application can be used in the field of big data technology, and can also be used in any field other than the field of big data technology. The application field of the HBase data cleaning method and device disclosed in the application is not Do limited.

[0048]In view ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides an HBase data cleaning method and device, which can be used in the technical field of big data, and the method comprises the following steps: if an HBase cluster is currently in a file merging period, searching a target data table of which the current data storage duration exceeds an overtime threshold in the HBase cluster, and obtaining a current target batch identifier of the target data table; and clearing the data with the target batch identification from the target data table, and adding the target batch identification into the data operated by the user in the target data table in advance. According to the method and the device, the operation pressure of the main node in the HBase cluster can be effectively reduced, the reliability and the efficiency of the HBase data cleaning process can be improved, and the overall performance and the operation stability of the HBase cluster are improved.

Description

technical field [0001] This application relates to the technical field of data processing, in particular to the technical field of big data, and in particular to a method and device for cleaning HBase data. Background technique [0002] The architecture of the distributed columnar storage database HBase mainly includes two parts: the master node Master and the slave node RegionServer. When operations involving table addition, deletion, modification, and query are involved, the master node Master is required to manage and transmit external requests. [0003] At present, the data cleaning method for HBase is mainly to use the cleaning operation truncate to clean the table of HBase. However, since the master node Master has only two instances, one master and one backup, it cannot be expanded. Therefore, when using the cleaning operation truncate to clean HBase Frequent table clearing operations will easily cause greater pressure on the master node, which will affect the overall...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/27G06F16/22
CPCG06F16/215G06F16/27G06F16/221
Inventor 梁晔华张世瑛赵吉昆杨嘉欣
Owner INDUSTRIAL AND COMMERCIAL BANK OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap