Distributed log processing method, server, system and computing device

Abstract

The invention discloses a distributed log processing method, which is suitable for being executed in a configuration server. The method comprises the following steps of: receiving registration software parameters generated by a user in a browser at a publishing end, the registration software parameters at least comprising a software unique identifier, a software type, a deployment type, a log server IP address and a log storage path; determining a log collector type corresponding to the registration software parameter, calculating a registration hash value corresponding to the registration software parameter, determining a message queue cluster and a message theme corresponding to the registration hash value by adopting a consistent hash algorithm, and then generating a configuration file;and sending the configuration file to the log server IP address, so that the log collector of the determined type is installed in the corresponding log server, and the log collector collects the logmessage of the registration software and then sends the log message to the determined message queue cluster and the message theme. The invention further discloses a corresponding configuration serverand a distributed log processing system.

Description

technical field [0001] The invention relates to the field of log processing, in particular to a distributed log processing method, server, system and computing equipment. Background technique [0002] With the rapid growth of internal and external businesses of IT companies, subscription software real-time rolling logs are the most intuitive and urgently needed cloud tools for system operation and maintenance engineers and software development engineers to check software running status, troubleshoot, and solve problems in real time. At this time, engineers usually need to enter the virtual machine or container through the bastion machine first, and then use Linux commands to view software logs, which poses a huge security risk. Moreover, the system operation and maintenance and business development within large Internet companies belong to different teams, and the software deployment structure is complex. The two teams are doomed to be unclear about the storage location of s...

AI Technical Summary

Problems solved by technology

In the face of massive logs, this method cannot realize the instant subscription of software logs with a second-level delay

Moreover, although the Kafka cluster can handle a large amount of logs for users to consume and consult, when the number of subscribers increases sharply, it will lead to fragmentation of topics and full network bandwidth, which will block the internal network connection of the enterprise.

Finally, this solution will occupy a large amount of storage resources, resulting in a waste of storage resources, which does not meet the company's current development goals of energy saving, emission reduction and cost saving.

Images