Set top box watching method and system based on big data analysis

Abstract

The invention discloses a set top box watching statistical method and system based on big data analysis. The set top box watching statistical method is characterized by comprising the following steps:receiving massive log files continuously uploaded by a client in real time and collecting the massive log files by a Flume program; storing the acquired data into a kafka distributed subscription system; carrying out eTL data cleaning operation on the data in the Kafka, and storing the structured data in the hbase after being established; associating data in the Hbase by using Hive, and mapping the structured data file into a database table; adopting hive for statistical summary, and storing statistical results in Mysql; and supporting real-time query of the structured data in the hbase and display of a statistical result report of the Mysql by the web page. The system comprises a data collection unit, a data preprocessing unit, a data cleaning unit, a data analysis processing unit, a data storage unit, a data statistical summarization and presentation unit and a visualization unit. According to the invention, the processing speed and capability of the set top box for collecting statistical data are greatly improved, and diversified information collection can be supported.

Description

technical field [0001] The invention relates to the technical field of streaming media collection equipment, in particular to a method for forcedly updating a system by using a U disk. Background technique [0002] The speed of data generated in people's daily life is getting faster and faster. To store these data, a large amount of disk capacity is required. After storage, data analysis also requires a lot of operating performance. Traditional technology, for massive data, only relying on mysql to process and analyze, has been unable to meet and realize our needs. In the traditional collection method, facing customers There are hundreds of millions of data generated by the end every day, and mysql has bottlenecks in receiving and processing. The storage and statistics of a large amount of data cannot be supported by mysql, and it may even be too late to process, resulting in congestion or downtime, making it impossible to continue statistical work. At the same time, becau...

AI Technical Summary

Problems solved by technology

After storage, data analysis also requires a lot of operating performance. Traditional technology, for massive data, only relying on mysql to process and analyze, has been unable to meet and realize our needs. In the traditional collection method, facing customers There are hundreds of millions of data generated by the end every day, and mysql has bottlenecks in receiving and processing

The storage and statistics of a large amount of data cannot be supported by mysql, and even it is too late to process, resulting in congestion or downtime, making it impossible to continue statistical work

At the same time, because of too much data, mysql query takes a long time, and the effect of second-level real-time response cannot be achieved. More server support is needed, especially in the viewing of set-top boxes. The traditional method will greatly increase the cost of the server and increase the difficulty of operation and maintenance

Images