Distributed data processing platform

A distributed data and processing platform technology, applied in the computer field, can solve the problem of low analysis efficiency of Hadoop platform

Active Publication Date: 2015-05-06
BEIHANG UNIV
View PDF7 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The invention provides a distributed data processing platform, which is used t

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data processing platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0031] figure 1 A schematic structural diagram of an embodiment of a distributed data processing platform provided by the present invention, such as figure 1 shown, including:

[0032] Storage layer 11, computing layer 12, query interface and algorithm library 13, and application layer 14;

[0033] Storage layer 11 includes: distributed fi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a distributed data processing platform. The distributed data processing platform comprises a storage layer, a calculation layer, a query interface, an algorithm library and an application layer, wherein the storage layer comprises a hadoop distributed file system HDFS, an HBase database system and a distributed index system ES; the HBase is built on the HDFS and is used for storing corresponding relationships between microblog identifiers and microblog data; corresponding relationships between microblog key fields and the microblog identifiers are built in the ES; the application layer is used for receiving a processing instruction sent by a user terminal and sending a corresponding query request to the query interface and the algorithm library according to the processing instruction; the query interface and the algorithm library are used for querying the microblog data from the storage layer according to the query request; the calculation layer is used for processing the queried microblog data according to the processing instruction and returning a processing result to the application layer, so that the query speed and analysis efficiency of the distributed data processing platform are improved through cooperation of the HBase and the ES; the requirements of big data of microblogs can be met.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a distributed data processing platform. Background technique [0002] Weibo is a typical type of big data. It has developed rapidly from its birth to the present. For example, the daily volume of Sina Weibo has exceeded 100 million. Especially in emergencies and hot events, the influence of Weibo The scale and speed of dissemination surpass ordinary blogs and traditional news media. At present, enterprise marketing and public opinion monitoring for Weibo are hot spots of concern, for example, real-time query of Weibo, statistical analysis, Weibo classification, hot spot detection, etc. [0003] In the prior art, the Hadoop platform is used to realize real-time query, statistical analysis, microblog classification, hotspot detection, etc. of microblogs. In the Hadoop platform, Weibo data is stored in the relational database Hbase. When the Hadoop platform analyzes Weibo, it n...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 沃天宇孙承根吴博于伟仁李建欣
Owner BEIHANG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap