HBase-based mobile internet mass data processing system and method

A mass data processing and mobile Internet technology, applied in the field of big data, can solve problems such as inability to effectively store mass data, and achieve the effect of improving query efficiency

Inactive Publication Date: 2018-04-20
WUHAN HONGXU INFORMATION TECH
View PDF3 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, to provide a HBase-based mobile Internet massive data processing system and its method, in order to solve the problem that the current technology cannot effectively store massive data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • HBase-based mobile internet mass data processing system and method
  • HBase-based mobile internet mass data processing system and method
  • HBase-based mobile internet mass data processing system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] The system is described in detail below in conjunction with the accompanying drawings and embodiments:

[0050] 1. Overall

[0051] Such as figure 1 , the system includes a data polling module 10, a data analysis module 20, a data cleaning module 30, an ETL module 40, a data table record encapsulation module 50, an index table record encapsulation module 60, an HBase writing module 70, an HBase cluster module 80 and a Region control module 90;

[0052] Its interaction relationship is:

[0053] Data polling module 10, data parsing module 20, data cleaning module 30, ETL module 40, data table record encapsulation module 50, index table record encapsulation module 60, HBase writing module 70, HBase cluster module 80 interact sequentially, and the Region control module 90 controls the allocation of Regions in the HBase cluster module 80 .

[0054] 2. Functional modules

[0055] 1) data polling module 10

[0056] The data polling module 10 is a method to regularly scan...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an HBase-based mobile internet mass data processing system and method, and belongs to the field of big data. The system comprises a data polling module (10), a data analysis module (20), a data cleaning module (30), an ETL module (40), a data table record packaging module (50), an index table record packaging module (60), an HBase writing module (70), an HBase cluster module (80) and a Region control module (90), wherein the Region control module (90) is used for controlling distribution, in the HBase cluster module (80), of Regions. According to the system and method,an HBase distributed database is used and an HDFS is used as a file system, so that the disk extension problem is solved; secondary index is realized, so that the retrieval speed is high; data table Regions are in one-to-one correspondence with index table Regions, on a same Region Server, RPC calling is decreased for once during the query, so that the query efficiency is improved; and effective technical support can be provided for mobile internet mass data processing.

Description

technical field [0001] The invention belongs to the field of big data, and in particular relates to an HBase-based mobile Internet massive data processing system and a method thereof. Background technique [0002] In recent years, mobile terminals such as mobile phones and PCs have become more and more popular, and the mobile Internet has developed rapidly, resulting in a sharp increase in the amount of information. How to store and retrieve massive data has increasingly measured the success of an enterprise, and data storage capabilities will become the core competitiveness of enterprises. The importance of data is unquestionable, but as data is generated faster and faster, and the amount of data is getting bigger and bigger, the challenges of data processing technology are also getting bigger and bigger. [0003] HBase is a distributed, column-oriented, scalable open source database mainly used for unstructured data storage purposes. HBase is an open source implementatio...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/316G06F16/3331
Inventor 程幸福戴长江
Owner WUHAN HONGXU INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap