Log collection method and system for distributed system

Abstract

The invention provides a log collection method and system for a distributed system. The log collection method specifically comprises the steps of generating a collection request according to a collection demand, sending the collection request to any control node in a login state in the distributed system, and executing log collection of each node in parallel according to a numerical value of a storage space of the control node under the condition that a preset collection condition is met, and under the condition that the log file collection of each node is completed, generating and storing a first compressed file, sending a plurality of first compressed files to the control node, generating a second compressed file, and storing the second compressed file in the control node. Through the technical scheme provided by the invention, the support of a third-party database is not depended on, the risk that the log file cannot be collected and cannot be analyzed and queried due to database exception is greatly reduced, and meanwhile, the stability and the reliability of log file storage are improved.

Description

technical field [0001] The invention relates to the field of log collection, in particular, to a log collection method and system for a distributed system. Background technique [0002] In order to adapt to the normal operation of a larger system, we choose to split the system into multiple independent processes, that is, a distributed system. The operation in a distributed system needs to be supported by a large number of hardware and software services. The data files of actions in the running process, that is, the collection of distributed logs. All nodes in the distributed system are connected to each other, and any node in the system can easily share data with other nodes. Distributed log collection Record the running status and abnormal information of each node, so that when the system is in an abnormal state, the abnormal time and cause can be located according to the analysis of the corresponding log information. At this time, the distributed system needs to spend mor...

AI Technical Summary

Problems solved by technology

[0002] In order to adapt to the normal operation of a larger system, we choose to split the system into multiple independent processes, that is, a distributed system. The operation of a distributed system needs to be supported by a large number of hardware and software services. The data files of the actions during the running process, that is, the collection of distributed logs. All nodes in the distributed system are connected to each other. Any node in the system can easily share data with other nodes. Distributed log collection Record the running status and abnormal information of each node, so that when the system is in an abnormal state, the abnormal time and cause can be located by analyzing the corresponding log information. At this time, the distributed system needs to consume more resources to manage the log

[0003] In the existing technology, the log collection tool of the distributed system relies on the support of the database or third-party software, regularly stores the log files in the database, and then analyzes the log files through the third-party plug-in, and obtains the logs through the database storage information. Information, during the operation of the distributed system, it is necessary to collect the logs of each server node in the system, obtain the log information of all nodes, and the log files of any node need to rely on database resources to store and read, when the log file storage capacity is too large In the case of , it takes longer to query the database, and the efficiency of log query decreases with the increase of log files

[0004] At the same time, when the database is abnormal or when the third-party plug-in relied on in the log collection tool is abnormal, the system log information cannot be obtained, and the stability and reliability of log collection are greatly reduced.

Images