Network monitoring data compression storing and combied detecting method based on similar data set

Abstract

The invention discloses a network monitoring data compression storage and joint detection method based on similar data sets. The technical problem to be solved is to propose a table-based relational database compression storage and joint detection method for multiple data sets with many identical data items. Combined detection method. The technical solution is to use the relational model to model the data, use the relational database to store and manage the data, and use the compression method based on the state tag sequence to compress and store multiple similar data sets under the same relational framework; use the query statement to restore the original The data set; by querying the compressed data table, various types of joint detection of multiple original data sets are realized. The present invention uses the similarity between a large number of data sets to incrementally compress multiple data sets into the same table of the database, which can be applied to the compressed storage of various types of network monitoring data, and can also be used to store other types of Web historical data or similar datasets.

Description

technical field [0001] The invention relates to a data compression storage and joint detection method, in particular to a compressed storage and joint detection method for massive network monitoring data with similar data sets. Background technique [0002] In recent years, with the development of information technology, especially the development of Internet technology, the amount of information in all countries in the world has shown an explosive growth trend, higher than 10 12 Massive databases of bytes have become commonplace. For example, the artificial satellites launched by NASA in the United States return 10 times to the ground every year. 15 bytes of observation data. The high-energy physics experiment data of the Lawrence National Laboratory in the United States is as high as 3×10 per year 14 byte. [0003] Network security monitoring also needs to store and process a large amount of data. The data saved in the network security monitoring can be information su...

AI Technical Summary

Problems solved by technology

The hardware overhead of the three-level storage method is relatively large, mainly through the expansion of hardware devices to obtain larger storage space, while increasing the storage capacity, it also greatly increases the query processing time and reduces the efficiency of the database.

When there are many data sets, the efficiency of joint detection is relatively low.

Images