Data management method and system of HBase database

A data management system and data management technology, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problem that large field data cannot be stored and read well, and achieve high reliability, guarantee easy Usability, high throughput effect

Active Publication Date: 2014-06-04
GUANGDONG ESHORE TECH
View PDF4 Cites 28 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0019] Based on this, it is necessary to provide a data management method and system for an HBase database in view of the technical problem that large-field data cannot be well stored and read in the HBase database in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data management method and system of HBase database
  • Data management method and system of HBase database
  • Data management method and system of HBase database

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

[0038] Such as figure 1 Shown is the work flowchart of the data management method of a kind of HBase database of the present invention, comprises:

[0039] Step S101, responding to the data writing request, generating an HBase record to be stored for saving in the HBase database;

[0040] Step S102, taking the field whose size of the content contained in the HBase record to be stored exceeds the preset field threshold as a large field, and extracting the content of the large field to generate a large field data block;

[0041] Step S103, writing the large field data block to the container file stored in the HDFS file system;

[0042] Step S104, using the position of the large field data block in the HDFS file system as an index, and replacing the content of the large field in the HBase record to be stored with the index;

[0043] S...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data management method and a system of an HBase database. The management method comprises the steps of writing response data into a request, generating HBase records to be stored in the HBase database; using fields with the content sizes larger preset field threshold values and included in the HBase records to be stored as large fields and extracting contents of the large fields to generate large field data blocks; writing the large field data blocks into container files stored in an HDFS file system; using positions of the large field data blocks in the HDFS file system as references and using the references to take the place of the contents of the large fields in the HBase records to be stored; storing the HBase records to be stored in the HBase. By adopting the data management method and the system of the HBase database, the advantages of an HBase and an HDFS can be well played, the large-field processing defect of the HBase is overcome, the shortcoming of overlarge resource consumption of the HDFS during small file storage is overcome, the business demand is met well, and good effect is obtained in the actual application.

Description

technical field [0001] The invention relates to the technical field related to databases, in particular to a data management method and system for an HBase database. Background technique [0002] With the surge in business volume, the amount of data and visits that the IT operation system needs to process has also soared. The traditional database solutions represented by Oracle are far from satisfying users due to their low efficiency and poor scalability. storage and query requirements, and gradually give way to a new generation of distributed databases represented by HBase. [0003] HBase–Hadoop Database is a high-reliability, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap PC Server. HBase can well support the record at the kilobyte level, but limited by its realization principle, common records and data files (such as tables, documents, picture scans, etc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2219G06F16/2228
Inventor 钟坚何广柏邹国栋张伟王腾石巍赖志坚
Owner GUANGDONG ESHORE TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap