Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A method and system for managing message data based on distributed database

A technology of message data and management methods, applied in the field of data management, can solve problems such as inability to support large concurrent fast read and write

Active Publication Date: 2020-01-14
广东湛数大数据有限公司
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The main purpose of the present invention is to propose a method and system based on distributed database message data management, to overcome the deficiency that the existing RDBMS management message data technology cannot support large concurrent fast read and write

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and system for managing message data based on distributed database
  • A method and system for managing message data based on distributed database
  • A method and system for managing message data based on distributed database

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0078] At present, when the process industry is facing the dilemma of time-series storage and management of massive data, Internet companies are also facing the problem of massive data management. Big data technologies represented by Hadoop, HBase, etc. provide a solution for massive data management, and are currently adopted by many Internet companies such as Taobao, Baidu, and Sohu. Hadoop is a reliable, scalable distributed file system, HBase (Hadoop Database, distributed database) is a distributed, scalable NoSQL (Not Only Structured Query Language, non-relational database) database, which is built on Hadoop Based on the infrastructure, relying on the rapid development of Hadoop, HBase is more and more widely used in the field of big data. The embodiment of the present invention is based on the description of the distributed database of HBase. The primary key in the HBase distributed database is called the RowKey primary key, and the Schema information in the database is c...

Embodiment 2

[0114] The second embodiment provides a HBase-based large-scale message data management system, including a client and a server, and the server includes:

[0115] The structured message data storage unit is used to store the structured message data to be stored in the HBase database through the Schema information design; Create a database table, design the Schema information of the data table, convert the structured message data to be stored into a domain model, and then store it in a separate column cluster in the database table.

[0116] The unstructured message data storage unit is used to store the unstructured message data to be stored in the HBase database in the form of data blocks; the specific implementation method is: add a new column cluster in the HBase database, and store Storing unstructured message data is stored in this column cluster. In the HBase database, other column clusters are used to store related information of unstructured message data to be stored. W...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed database based message data management method and system. The method comprises the following steps: storing structured message data through database object set design based on the distributed database; storing unstructured message data through a data block manner; querying the structured message data through a major key manner; and querying the unstructured message data through a filter manner. According to the distributed database based message data management method and system, Hadoop and HBase are used as the bottom layer storage supporting technology; the shortcoming that the existing RDBMS management message data technology cannot support high concurrence rapid read-write is overcome by utilizing the high expansibility and the capability of supporting large-scale message data high-performance storage management of the Hadoop and HBase; meanwhile, the storable management data type comprises the structured data and unstructured data; and the data management requirements of the message system having the characteristics of big data, high time sequence, simple structure and the like are satisfied.

Description

technical field [0001] The invention relates to the technical field of data management, in particular to a message data management method and system based on a distributed database. Background technique [0002] With the development of today's Internet, various message systems are faced with the test of a sharp increase in data volume every year. At the same time, with the diversification of message input methods, there will be a large amount of semi-structured and unstructured data in the system. This makes it difficult for database designers to determine the data relationship model that meets business requirements in advance, and the horizontal expansion capability of traditional RDBMS is usually relatively poor, and the change of database table structure will incur high costs, which is undoubtedly a disaster for large-scale message systems . [0003] Moreover, the message system has high requirements for real-time performance, so the performance of the database for high...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/27G06F16/21
CPCG06F16/21G06F16/27
Inventor 林浩
Owner 广东湛数大数据有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com