A method and device for storing massive network flow data

Abstract

The invention relates to a storage method and device of mass network flow data. The storage method includes the steps of collecting data query requests submitted by the user in an optional period and achieving query conditions according to the data query requests; analyzing time attributes and characteristic attributes of the query conditions and selecting a characteristic attribute of a threshold value exceeding preset occurrence frequency as a cluster attribute; determining number and end point of segments of to-be-queried data, determining size of a cache region according to memory space of the to-be-queried data in the segments and generating configuration files according to the cluster attribute, the number and the end point of the segments of the to-be-queried data and size of the to-be-written cache region; subjecting a collector to receiving the network flow data and transmitting the same to a file server, and subjecting the file server to storing the received network flow data according to the configuration files. According to the arrangement, the query conditions are directly reflected to corresponding space divisions to directly achieve writing or querying of the data, costs on uploading and storing are maximally decreased, and query performance is guaranteed.

Description

technical field [0001] The invention relates to the field of massive flow data storage and query, in particular to a method and device for storing massive network flow data. Background technique [0002] NetFlow is a network protocol released by Cisco in 1996 to collect and monitor network flow data. Because it can provide some key services for applications, including network data collection, network traffic statistics, denial of service monitoring, intrusion detection, etc., it has high application value and practical significance. [0003] Relational databases have been widely used as a traditional solution for NetFlow flow data management. Thanks to the mature index and query mechanism, the database has obvious advantages in data query processing. However, with the continuous expansion of data scale, database solutions encounter serious challenges in terms of scalability and data storage. First of all, NetFlow flow data arrives at a fast speed, and the loading speed of...

AI Technical Summary

Problems solved by technology

First of all, there is the problem of data expansion. Since Hbase uses column storage, when the original data is loaded into Hbase, a lot of information about columns and column clusters will be added, which will cause serious data expansion and directly cause the storage cost to increase exponentially.

Secondly, the single-point loading capability of Hbase is generally at the millisecond level for a single record. Since the NetFlow flow data arrives very fast, the existing loading speed of Hbase cannot meet the requirements of the actual environment at all.

Images