Log information acquiring and monitoring method

Abstract

The invention relates to the technical field of data monitoring and processing, and particularly discloses a log information acquisition and monitoring method, which comprises the following steps: establishing a connection channel with a log source, and acquiring log data in real time; performing log compression, log cutting, format arrangement, data filtering and log classification on the log data to obtain cold data and hot data; the cold data is stored in a MinIO distributed file storage system, and the hot data is stored in an ElasticSearch distributed search engine; the log data in the ElasticSearch distributed search engine is cooled, and the cooled log data is moved to the MinIO distributed file storage system. According to the method, multiple log sources can be efficiently collected, data are not lost, and the log retrieval efficiency is improved; massive data can be stored through distributed storage, so that the memory pressure is reduced; cold and hot data can be processed in time, data of log data on a retrieval system is reduced, and log retrieval analysis efficiency is better improved.

Description

technical field [0001] The invention relates to the technical field of data monitoring and processing, in particular to a log information acquisition and monitoring method. Background technique [0002] With the rise and development of cloud computing, mobile Internet, and the Internet of Things, the era of big data has arrived. The amount of log data generated by most Internet companies every day can reach the level of hundreds of millions. How to efficiently collect, manage and analyze the massive data generated by daily business has become an urgent problem to be solved. [0003] The existing massive data processing methods are mainly distributed log systems. Most of these log systems use the ELK architecture. The so-called ELK architecture is a system based on ElasticSearch, Logstash, and Kibana. Among them, ElasticSearch is an open source distributed search engine that provides three functions of collecting, analyzing and storing data. Logstash is a tool for log collec...

AI Technical Summary

Problems solved by technology

[0004] However, the ELK architecture does not have a message queue cache, and there is a hidden danger of data loss. With the construction of the system, the amount of logs will increase. The storage of a large amount of log data has brought great challenges to disk resources. The existing distributed log They are all stored in the ElasticSearch memory. Although this method is convenient for log retrieval and analysis, it requires a lot of memory.

Images