Distributed data processing platform

A distributed data and processing platform technology, applied in the computer field, can solve the problem of low analysis efficiency of Hadoop platform

Active Publication Date: 2017-11-14
BEIHANG UNIV
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The invention provides a distributed data processing platform, which is used to solve the problem of low analysis efficiency of the Hadoop platform in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data processing platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0031] figure 1 It is a schematic structural diagram of an embodiment of a distributed data processing platform provided by the present invention, such as figure 1 Shown, including:

[0032] Storage layer 11, computing layer 12, query interface and algorithm library 13, and application layer 14;

[0033] Storage layer 11 includes: dist...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a distributed data processing platform. The distributed data processing platform comprises a storage layer, a calculation layer, a query interface, an algorithm library and an application layer, wherein the storage layer comprises a hadoop distributed file system HDFS, an HBase database system and a distributed index system ES; the HBase is built on the HDFS and is used for storing corresponding relationships between microblog identifiers and microblog data; corresponding relationships between microblog key fields and the microblog identifiers are built in the ES; the application layer is used for receiving a processing instruction sent by a user terminal and sending a corresponding query request to the query interface and the algorithm library according to the processing instruction; the query interface and the algorithm library are used for querying the microblog data from the storage layer according to the query request; the calculation layer is used for processing the queried microblog data according to the processing instruction and returning a processing result to the application layer, so that the query speed and analysis efficiency of the distributed data processing platform are improved through cooperation of the HBase and the ES; the requirements of big data of microblogs can be met.

Description

Technical field [0001] The present invention relates to the field of computer technology, in particular to a distributed data processing platform. Background technique [0002] Weibo is a typical type of big data. From its birth to the present, it has developed rapidly. For example, Sina Weibo’s daily number of posts has exceeded 100 million, especially in emergencies and hot events. The scale and speed of dissemination have surpassed ordinary blogs and traditional news media. At present, corporate marketing and public opinion monitoring for Weibo are hot topics, such as Weibo real-time query, statistical analysis, Weibo classification, and hot spot detection. [0003] In the prior art, the Hadoop platform is used to realize real-time query, statistical analysis, microblog classification, and hot spot detection of microblogs. In the Hadoop platform, Weibo data is stored in the relational database Hbase. When the Hadoop platform analyzes Weibo, it needs to retrieve Weibo data fro...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
Inventor 沃天宇孙承根吴博于伟仁李建欣
Owner BEIHANG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap