Method and system for showing and querying logs in real time

Abstract

The invention relates to a method and a system for showing and querying logs in real time. The method comprises steps as follows: log messages are collected by a LOG collecting server Logstash and sent to a message system server Kafka in a message mode; a Tornado server reads log message records from the message system Kafka in a multi-thread concurrent reading mode and provides filter service; the Tornado server adopts a high-performance Websocket long link to send log messages to a foreground terminal Livetail quickly; the foreground terminal Livetail displays the log messages in real time in a specific manner and provides filtering and searching functions. The collected log messages are directly written into the storage management system Kafka, then read by the Tornado concurrently and sent to the front-end server Livetail with data filtering and querying functions by using efficient WebSocket, so that information time delay is reduced, the timeliness of the log messages is greatly improved, a filtering mechanism is provided at the server end, the messages are effectively pushed, the safety of the log messages is guaranteed, defects of existing systems are greatly overcome, and the user experience is improved.

Description

technical field [0001] The present invention relates to a method and system for real-time display and query of logs, in particular to a method and system for real-time display and query of logs in the field of big data processing. Background technique [0002] In the context of the big data era, how to quickly classify and efficiently and accurately search real-time log data has gradually become an important factor affecting the productivity of log management. If you can manage massive real-time log data and at the same time refine and classify the data according to customer needs and then push them accurately, you can not only reduce user waiting time, improve production efficiency, but also effectively ensure the security of user data. [0003] Real-time log ELK includes log collection and real-time display and query of logs. Traditional ELK uses Logstash to complete log collection and write it into Elasticsearch, and then obtains data from Elasticsearch and displays it t...

AI Technical Summary

Problems solved by technology

Traditional ELK uses Logstash to complete log collection and write it into Elasticsearch, and then obtains data from Elasticsearch and displays it through Kibana, while Kibana focuses on log analysis and statistics, and has always been weak in functions such as real-time log display and filter query. Can not meet the timeliness requirements of real-time logs

Images