Stream preprocessing system and method based on mass information of mobile internet

Abstract

The invention discloses a stream preprocessing system and method based on mass information of a mobile internet, and belongs to the field of big data. The system comprises the mobile internet (000), an interface machine (100), a Kafka cluster (200), a Spark cluster (300) and a Hadoop cluster (400) which are communicated in sequence. The method mainly comprises the steps that a control module is connected with the Kafka cluster through a Zookeeper and reads data cached in the Kafka cluster; and a preprocessing module processes the data according to a specified processing process and finally writes the data into an HBase. According to the preprocessing system and method, the deficiency that MapReduce does not support stream data processing is made up for; the processing speed is high and the fault tolerance is achieved; by integrating three technologies of a distributed message system, a distributed stream processing framework and a hadoop distributed file system in the field of big data, a reference is provided for modular integration of other big data technologies; and all the mass information of the mobile internet can be processed in real time and technical support can be provided for real-time services of operators.

Description

technical field [0001] The invention belongs to the field of big data, and in particular relates to a streaming preprocessing system and method based on massive information of the mobile Internet. Background technique [0002] In recent years, the mobile operator network has become a natural carrier of big data storage and flow. The mobile Internet has a variety of data sources, including online behavior data, location data, signaling data, Weibo data, instant messaging data, web pages, audio data, video files and pictures, etc., and these data will continue to be generated. In order to extract useful information from these streaming data in real time and filter out useless information, it is often necessary to process it quickly, and then store it in a standardized form in the data center to provide efficient query services. [0003] Compared with general data services, this type of big data service has its own characteristics: first, the business data is big data, and the...

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and provide a streaming preprocessing system and method based on massive information on the mobile Internet, so as to solve the problem that the current technology cannot process streaming data in real time and reliably

Images