real-time dynamic data management system based on HBase

A data management system, real-time dynamic technology, applied in the fields of data search and big data storage, can solve problems such as the inability to make full use of multi-core processor resources, relieve concurrent writing pressure, speed up writing and writing, and meet high dynamic Effect

Pending Publication Date: 2019-04-05
CHINA UNIV OF PETROLEUM (EAST CHINA)
View PDF8 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But they are all based on single-threaded query engines, which cannot make full use of multi-core processor resources
How to use multi-thread

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • real-time dynamic data management system based on HBase
  • real-time dynamic data management system based on HBase
  • real-time dynamic data management system based on HBase

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0019] The logic framework of the real-time dynamic data management system based on HBase in the present invention is as follows: figure 1 Shown, including data storage, data writing, index design, and parallel query design.

[0020] Massive data storage optimization starts with a logical structural division, which can be divided into two parts according to the storage method: storage in the form of files and storage in the form of HBase tables. Storage in th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a real-time dynamic data management system based on HBase. The real-time dynamic data management system comprises data reading optimization, a composite index mechanism and parallel query optimization. According to the invention, a persistent data storage mechanism based on HDFS and HBase is designed, so that the problem of mass storage of multi-structure data is solved; a composite index storage model is designed based on an HBase table structure, and realizing mapping management of index cache addresses and index table design of cache data storage; queue division is performed on the data based on a multi-source buffer area structure to realize parallel storage, and the concurrent writing pressure of mass data is relieved by combining strategies such as a consistency hash algorithm, a multi-thread technology, a line key optimization design and the like; And a multi-query parallel mechanism is designed based on an HBase query real execution module and a multi-line technology, so that the query speed is increased. According to the real-time dynamic data management system based on the HBase, the data writing and reading performance is effectively improved, theproblem of mass storage of multi-structured data is solved, and the high-dynamic and real-time environmental requirements are met.

Description

technical field [0001] The invention relates to the fields of big data storage and data search, in particular to an HBase-based real-time dynamic data management system. Background technique [0002] As a database based on big data, HBase is very important for the efficiency of data management services. [0003] As the underlying database, HBase must first ensure the storage efficiency of various types of data to meet the different storage requirements of different environments. How to use the multi-source buffer structure to divide different types of stream data into queues, and which strategies can be combined to store the data in parallel in the HBase cluster server, so as to alleviate the pressure of massive data and write concurrency. This is real-time dynamic data based on HBase The management system deals with the challenges faced. Secondly, the indexing mechanism of HBase is not perfect, so it cannot meet the efficient query requests in the real-time environment. ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/13G06F16/182G06F16/2453G06F16/2455
Inventor 张卫山任鹏程房凯
Owner CHINA UNIV OF PETROLEUM (EAST CHINA)
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap