A multi-level network flow storage method and system based on big data

Abstract

The invention discloses a multi-level network flow storage method and system based on big data, and relates to the technical field of data storage. The multi-level storage mode is adopted, the storagescheme is flexibly configured according to the service requirements, and hardware resources consumed for file archiving and storage only through full package storage or big data are greatly reduced.Analyzing the collected flow, obtaining multi-level network protocol packet header information, and storing the information needing to be stored as a structured log file; And analyzing and calculatingthe structured log file, and storing an analysis result as a relational database table to meet the real-time business requirement of the upper application. According to the invention, the flow packetfile and the table of the relational database are archived and stored according to the service requirements by using the big data technology, the cluster computing capability of big data can be exerted, and the requirements of a large high-speed network flow environment are met.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a large data-based multi-level network flow storage method and system. Background technique [0002] Network traffic data plays an important role in network management, network optimization operation and maintenance, network security, and network user behavior analysis and mining. With the rapid development of the Internet and the advancement of network technology, the types and quantities of network traffic data are increasing rapidly, which brings great challenges to the storage and analysis of network traffic data. As the basic work of network traffic analysis, network traffic storage technology has important practical significance for the development of network management and optimization, network security and other technologies. [0003] Currently the main storage methods are: [0004] 1. All-inclusive storage, which directly stores the traffic packets collected by th...

AI Technical Summary

Problems solved by technology

[0009] 1. All-inclusive storage: It requires a large number of hardware storage resources to support, while retrieval and analysis consumes system resources, and the speed is very slow when the amount of data is large, and it is difficult to support the business needs of upper-level applications

[0010] 2. Big data storage: similar to the all-inclusive storage solution, but uses big data technology to realize distributed storage, increase the reliability of file storage, and improve efficiency, but still requires the support of a large number of storage and computing resources, although To a certain extent, the efficiency of retrieval and analysis is improved, but it still cannot satisfy the real-time business requirements of the upper layer.

[0011] 3. Big data indexing: Store traffic-related information in a structured manner in the HBASE database to assist full-package storage index retrieval, but it still belongs to the big data full-package storage strategy, which requires a lot of storage resources. At the same time, after the data package is retrieved, still It requires a lot of analysis and calculation work, consumes a lot of computing resources and time, and is difficult to meet the real-time business of the upper layer

[0012] 4. Three-level data flow storage: This storage scheme has a short storage time for ordinary flow databases, only between one week and a few weeks, and the storage time is short. At the same time, both ordinary flow databases and basic index databases use relational databases. As the flow continues to grow, the data table will continue to increase, and the query and analysis performance will continue to decline, making it difficult to meet the needs of large-scale high-speed network traffic environments

Images