Method and system for losslessly acquiring logs in real time in distributed environment

Abstract

The invention relates to a method and system for losslessly acquiring logs in real time in a distributed environment. The method includes the steps that log messages in log files are acquired; the log messages are cached in a Disk-Assisted buffer mode; state messages are updated to offset state files in real time; the corresponding relation between the log files and the offset state files is recorded; whether the log files have the missing acquisition condition or not is judged; if the log files have the missing acquisition condition, a configuration module interface is called, and acquisition configuration files of the log files in missing acquisition are generated. The log messages are persisted in a magnetic disk, it is guaranteed that the messages are not lost, updating of the offset state files can guarantee that the messages are not lost or data is not repeatedly transmitted, the log files are subjected to missing acquisition judgment, the missing acquisition condition is avoided, lossless real-time log acquisition is achieved, configuration and operation are convenient, the performance is high, the real-time performance and reliability of distributed system log acquisition are guaranteed, and high abnormal re-acquisition capacity is achieved.

Description

technical field [0001] The invention relates to a log collection method, and more specifically refers to a non-destructive collection method and system for real-time logs in a distributed environment. Background technique [0002] With the development of platform technology and distributed computing technology, the logs used to record user operations and system operation status may be distributed on hundreds of servers. How to collect these data together for subsequent analysis is a big problem. technical challenge. The emergence of the distributed collection system is to meet the collection needs of massive logs in the distributed system environment. The distributed collection system uses the collectors distributed on multiple computer nodes connected to the network to realize unified management of metadata and distributed storage of collectors. System log collection in a distributed environment. [0003] At present, Rsyslog and Logstash are usually used to collect logs. ...

AI Technical Summary

Problems solved by technology

[0005] 1. The single-point problem of Rsyslog will cause some scrolling output log files to fail to be collected during abnormal exit

[0006] 2. Rsyslog does not support the collection of dynamic log directories such as daily updated date directories, nor does it support unified configuration and collection of multiple directories with the same structure

[0007] 3. Rsyslog does not support the archiving and deletion of log files after collection, and the number of files in the log output directory is increasing, requiring manual archiving and deletion

[0009] 1. The single point problem of Logstash will lead to high probability of resending and loss of log messages during abnormal exit

[0010] 2. Logstash does not support collection of dynamic log directories such as daily updated date directories, nor does it support unified configuration and collection of multiple directories with the same structure

[0011] 3. Logstash does not support the archiving and deletion of log files after collection, and the number of files in the log output directory is increasing

[0012] 4. In a distributed system environment, metadata such as Logstash collection configuration does not have centralized configuration and release functions, and cannot effectively manage the collection metadata of each host

[0013] 5. In a distributed system environment, Logstash does not provide a monitoring output interface, and cannot effectively manage the status of Logstash

[0014] 4. Logstash is developed with ruby, and the collection performance is low

Images