Stream preprocessing system and method based on mass information of mobile internet

A technology of mobile Internet and massive information, applied in the field of streaming preprocessing system based on massive information of mobile Internet, it can solve the problem of unable to process streaming data in real time and reliably

Inactive Publication Date: 2016-04-06
WUHAN HONGXU INFORMATION TECH
View PDF5 Cites 36 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and provide a streaming preprocessing system and method based on massive information on the mobile Internet, so as to solve the problem that the current technology cannot process streaming data in real time and reliably

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Stream preprocessing system and method based on mass information of mobile internet
  • Stream preprocessing system and method based on mass information of mobile internet
  • Stream preprocessing system and method based on mass information of mobile internet

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0065] Below in conjunction with accompanying drawing and embodiment describe in detail:

[0066] 1. System

[0067] 1. Overall

[0068] Such as figure 1 , the system includes a mobile Internet 000, which is provided with an interface machine 100, a Kafka cluster 200, a Spark cluster 300 and a Hadoop cluster 400;

[0069] The mobile Internet 000, the interface machine 100, the Kafka cluster 200, the Spark cluster 300 and the Hadoop cluster 400 are connected in sequence.

[0070] 2. Functional components

[0071] 0) Mobile Internet 000

[0072] Mobile Internet 000 refers to the general term for activities that combine and practice Internet technologies, platforms, business models and applications with mobile communication technologies.

[0073] 1) Interface machine 100

[0074] The interface machine 100 includes a first interface machine 110, a second interface machine 120...the Mth interface machine 1M0;

[0075] Each interface machine includes a file polling module 101 ,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a stream preprocessing system and method based on mass information of a mobile internet, and belongs to the field of big data. The system comprises the mobile internet (000), an interface machine (100), a Kafka cluster (200), a Spark cluster (300) and a Hadoop cluster (400) which are communicated in sequence. The method mainly comprises the steps that a control module is connected with the Kafka cluster through a Zookeeper and reads data cached in the Kafka cluster; and a preprocessing module processes the data according to a specified processing process and finally writes the data into an HBase. According to the preprocessing system and method, the deficiency that MapReduce does not support stream data processing is made up for; the processing speed is high and the fault tolerance is achieved; by integrating three technologies of a distributed message system, a distributed stream processing framework and a hadoop distributed file system in the field of big data, a reference is provided for modular integration of other big data technologies; and all the mass information of the mobile internet can be processed in real time and technical support can be provided for real-time services of operators.

Description

technical field [0001] The invention belongs to the field of big data, and in particular relates to a streaming preprocessing system and method based on massive information of the mobile Internet. Background technique [0002] In recent years, the mobile operator network has become a natural carrier of big data storage and flow. The mobile Internet has a variety of data sources, including online behavior data, location data, signaling data, Weibo data, instant messaging data, web pages, audio data, video files and pictures, etc., and these data will continue to be generated. In order to extract useful information from these streaming data in real time and filter out useless information, it is often necessary to process it quickly, and then store it in a standardized form in the data center to provide efficient query services. [0003] Compared with general data services, this type of big data service has its own characteristics: first, the business data is big data, and the...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/182
Inventor 严雄伟
Owner WUHAN HONGXU INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap