A method and system for multi-level network traffic storage based on big data

Abstract

The invention discloses a large data-based multi-level network flow storage method and system, and relates to the technical field of data storage. The present invention adopts a multi-level storage mode, flexibly configures a storage scheme according to business requirements, and greatly reduces the hardware resources consumed by only full-package storage or only big data for file archiving and storage. Analyze the collected traffic, obtain multi-layer network protocol header information, and store the information to be stored as a structured log file; then analyze and calculate the structured log file, and store the analysis results as a relational database table to meet the needs of upper-layer applications real-time business needs. The present invention utilizes big data technology to archive and store traffic packet files and tables of relational databases according to business needs, and can utilize the cluster computing capability of big data to meet the needs of large-scale high-speed network traffic environments.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a large data-based multi-level network flow storage method and system. Background technique [0002] Network traffic data plays an important role in network management, network optimization operation and maintenance, network security, and network user behavior analysis and mining. With the rapid development of the Internet and the advancement of network technology, the types and quantities of network traffic data are increasing rapidly, which brings great challenges to the storage and analysis of network traffic data. As the basic work of network traffic analysis, network traffic storage technology has important practical significance for the development of network management and optimization, network security and other technologies. [0003] At present, the main storage methods are: [0004] 1. All-inclusive storage, which directly stores the traffic packets collected by ...

AI Technical Summary

Problems solved by technology

[0009] 1. All-inclusive storage: It requires a large number of hardware storage resources to support, while retrieval and analysis consumes system resources, and the speed is very slow when the amount of data is large, and it is difficult to support the business needs of upper-level applications

[0010] 2. Big data storage: similar to the all-inclusive storage solution, but uses big data technology to realize distributed storage, increase the reliability of file storage, and improve efficiency, but still requires the support of a large number of storage and computing resources, although To a certain extent, the efficiency of retrieval and analysis is improved, but it still cannot satisfy the real-time business requirements of the upper layer.

[0011] 3. Big data indexing: Store traffic-related information in a structured manner in the HBASE database to assist full-package storage index retrieval, but it still belongs to the big data full-package storage strategy, which requires a lot of storage resources. At the same time, after the data package is retrieved, still It requires a lot of analysis and calculation work, consumes a lot of computing resources and time, and is difficult to meet the real-time business of the upper layer

[0012] 4. Three-level data flow storage: This storage scheme has a short storage time for ordinary flow databases, only between one week and a few weeks, and the storage time is short. At the same time, both ordinary flow databases and basic index databases use relational databases. As the flow continues to grow, the data table will continue to increase, and the query and analysis performance will continue to decline, making it difficult to meet the needs of large-scale high-speed network traffic environments

Images