Distributed log system

Abstract

The invention provides a distributed log system. The distributed log system of the invention adopts GO language to compile a collection module, when working, the collecting module collects the original log data generated by the log source and carries out compression processing to generate compressed log data for data transmission to the processing and storage module, the processing and storing module stores the compressed log data transmitted by the collecting module and generates a log index, A log retrieval module receive query criteria and transmits that query criteria to a proces and storage module, the processing and storage module reads the compressed log data corresponding to the query condition according to the query condition and the log index to generate the corresponding query result and transmits the query result to the log retrieval module, The log retrieval module visually displays the query results, which can collect the log data in real time and efficiently, reduce theamount of log data, and improve the efficiency of log retrieval.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a distributed log system. Background technique [0002] With the rise and development of cloud computing, mobile Internet, and Internet of Things, the era of big data has come. The log collection and analysis of the big data business system is the basis for evaluating the health of the node host, analyzing the problems in the system, and displaying various reports. [0003] In the cloud computing environment, the distributed log system aims to provide collection, storage, and query services for call chain logs and system logs under distributed clusters, and mainly solves difficult problems such as distributed system log dispersion, log tracking, and problem location. [0004] Existing distributed log systems mostly use the ELK architecture. The so-called ELK architecture refers to a system based on ElasticSearch, Logstash, and Kibana. Among them, ElasticSearch is an open sourc...

AI Technical Summary

Problems solved by technology

Logstash consumes a lot of resources, and occupies a large amount of central processing unit (CPU) and memory when the system is running. In addition, the traditional ELK architecture does not have a message queue cache, and there is a hidden danger of data loss. Usually, the daily log data of a distributed log system Reaching the TB level requires a distributed log system to efficiently collect logs and avoid the impact of log collection on business procedures. At the same time, log collection needs to be stable and continuous to avoid missing log data.

[0005] Furthermore, since the log data of the distributed log system reaches TB level every day, the amount of logs will increase with the construction of the system. The storage of a large amount of log data brings great challenges to disk resources, and the excessive log The conference affects the efficiency of log retrieval by the distributed log system. At the same time, the distributed log system needs to mine and analyze the log data in the face of collected TB-level log data to realize the maximum utilization of the log data. The existing distributed The log system cannot be implemented

Images