Big data-based cloud platform log management system

Abstract

The invention belongs to the technical field of big data, and discloses a big data-based cloud platform log management system, which comprises a log collection module used for analyzing and mining logpractices through a distributed multi-task method; a log preprocessing module is used for filtering log events and merging similar events through an auditing strategy, and sending the log events andthe similar events to the real-time detection module and the database system respectively; the database system is used for storing the log events sent by the log preprocessing module; the real-time detection module is used for auditing the processed log event and responding to an auditing result according to a response strategy; and the log analysis module is used for analyzing historical data inthe database system and displaying an analysis result through a chart according to user settings. According to the invention, the working efficiency of operation and maintenance personnel in the operation and maintenance management process and the perception ability of the safety situation can be improved.

Description

Technical field[0001]The invention belongs to the technical field of big data, and specifically relates to a cloud platform log management system based on big data.Background technique[0002]With the continuous development of information technology, users' demands for new services and requirements for service quality are also constantly improving. This brings about a new challenge, that is, the new environment generates a large number of logs that have never been seen before, and brings related problems, such as: log files are stored scattered, large in number, and log files that can be directly accessed have a short storage period , It is extremely inconvenient for operation and maintenance; the log format is inconsistent and the readability is too low; the query is time-consuming and labor-intensive, and the efficiency is low; the correlation of related logs is low and cannot be clearly located; a large number of logs cannot be counted, and the business cannot be accurately analyze...

AI Technical Summary

Problems solved by technology

However, after the centralized management of logs, the statistics and retrieval of logs has become a more troublesome thing. In the prior art, Linux commands such as grep, awk and wc can be used to achieve retrieval and statistics, but for more demanding queries and sorting And statistics and other requirements and a huge number of machines still use this method is inevitably a bit powerless

[0004] Traditional relational databases cannot meet the requirements of the new situation. It is necessary to establish a cloud platform log management system based on big data to improve log management efficiency.

Images