Deep packet inspection based network real-time large data system and large data analysis method

Abstract

The present invention discloses a deep packet inspection based network real-time large data system and a large data analysis method, and relates to the field of deep packet inspection and large data analysis. The system comprises a deep packet inspection unit, a deep packet inspection control unit, a database unit, a data mining unit and an online analytical processing unit, wherein the system collects a large data in the network in real-time according to an upper layer application demand, and extracts valuable information in the large data to submit to an upper-layer application entity for use. The method comprises the following steps: acquiring information demand of an upper layer application; converting the information demand into a strategy of network data collection; according to the strategy, collecting a corresponding data from the network to establish a database; based on the database, analyzing and mining to acquire demand information for the upper layer application; and submitting the acquired information to the upper layer application entity for use. According to the deep packet inspection based network real-time large data system and the large data analysis method disclosed by the present invention, a network technology is combined with a large data analysis technology, so that the relevant application of network demands can be met, and investment of hardware will not be significantly increased.

Description

Technical field [0001] The invention relates to the field of deep packet inspection and big data analysis, and in particular to a network real-time big data system and big data analysis method based on deep packet inspection. Background technique [0002] The rapid development of the network is one of the most obvious characteristics of modern times. The benefits of the network to the society are innumerable, but with the development of the network and the advent of the cloud era, the data generated in the network has exploded, indicating the era of big data. Has arrived. The industry generally believes that “big data” is a massive, high-growth, and diversified information asset that requires new processing models to have stronger decision-making power, insight and discovery, and process optimization capabilities; in other words, big data refers to unaffordable A collection of data captured, managed and processed with conventional software tools within the time frame of In othe...

AI Technical Summary

Problems solved by technology

[0006] (1) Poor real-time performance. Since the scale of big data in the network has reached an astonishing magnitude, and the data capacity generated by the network also shows randomness in time distribution, the peak data scale far exceeds the average value, and it takes a long time to process these data. Time, most existing methods use non-real-time methods, resulting in poor real-time output of big data systems and reducing the value of output

[0007] (2) High requirements for data processing hardware

Just because the scale of big data is too large, ordinary hardware configuration is not capable of processing big data. Therefore, more powerful and higher-configuration hardware support is required.

[0008] (3) There are many detours in the analysis process and serious waste of resources

Due to the large amount of sand (valueless data) and the lack of gold (valuable information) in big data, refining gold requires a comprehensive inspection of various sands, or even checking the same type of sand multiple times, which makes the analysis process tortuous and lengthy, and resource utilization is low. Poor, low analysis processing efficiency

[0009] Therefore, in the face of the huge amount of big data generated by the network, network-related applications have high requirements for real-time performance, which makes big data-related technologies face huge difficulties and challenges when facing real-time network analysis.

Images