System and method for saving and analyzing log data

Abstract

The invention provides a system for saving and analyzing log data, and the system comprises a log data compression and conversion part which is used for compressing high-frequency data appearing in the log data or using a conversion function to compress the high-frequency data to obtain a new log data structure and storing the new log data structure; a log analysis part used for rearranging, iteratively accessing and sequencing the converted log data by using a specified algorithm; and a log data decompression and restoration part that restores the original log data using inverse transformation of the algorithm transformation used by the log data compression and conversion section. Logs are stored by utilizing the hash table through an optimized storage structure, so that the processing time of the business logs can be greatly shortened, and the optimization problem of query and analysis of big data is solved.

Description

technical field [0001] The present disclosure relates to a system and method for storing and analyzing log data, in particular to a system and method for storing and analyzing log data with an optimized log storage structure. Background technique [0002] As an important part of security products, business logs can use its information to help us effectively manage the network and improve network security. [0003] The existing logs are basically stored in the database or log files. The database adds the corresponding log word field in the table creation and writes the file through the database statement, and the log file is written into the file through the specified log format. When analyzing logs, the log files are continuously parsed and counted according to requirements. [0004] Compared with the file system, the database is much more complicated, and the overhead is naturally higher. In the case of big data, the database storage method is not suitable. In the file sy...

AI Technical Summary

Problems solved by technology

[0004] Compared with the file system, the database is much more complicated, and the natural overhead is also higher. In the case of large data, the database storage method is not suitable

In the file system, accessing files is very simple, compared with the database overhead, but in the old log format, it is inefficient to analyze the log information of big data processing

[0005] Whether it is parsing logs or query analysis, a lot of work needs to be repeated each time, and query statistics must be re-parsed for each different analysis requirement, which wastes performance and costs a lot

Images