A highly reliable real-time log collection and visual retrieval method

Abstract

A high-reliable real-time log collection and visual retrieval method comprises the following steps: 1) setting up a real-time log collection structure, which includes real-time log collection, log data encryption transmission, log data integrity check, missing alarm, and visual inspection result system structure; Deploying Flume client and custom Flume plug-in, Kafka cluster, HDFS distributed filesystem on the target log server; configuring the Flume configuration file, reading the multithreaded target log, and transferring the read data to the Kafka cluster. Kafka clusters cache log data bycreating partitions and Topic classifications of Kafka clusters; 2) log data integrity chec flow and algorithm steps: when log data is transmit, a self-defined Flume plug-in sends checking messages tothat collected log file, so that the Storm cluster checks the integrity of log data stored in the HDFS when landing log data; 3) slice log file deletion and missing data retransmission flow and algorithm step.

Description

technical field [0001] The invention relates to the field of real-time log collection and retrieval, in particular to a highly reliable real-time log collection and visual retrieval method. Background technique [0002] Generally, a large-scale system is a distributed deployment architecture. Different service modules are deployed on different servers, and the logs generated by the services (application logs, access logs, security logs, etc.) are therefore scattered on each server. The retrieval and statistics of log information can be realized directly on the log server through grep, awk, and wc commands. However, in large-scale scenarios (dozens or hundreds of services, one server with multiple log types, and one type with multiple log files), this method is inefficient, and at the same time, it is necessary to face log files to cover them circularly How to archive the log data is too large, the text search is too slow, how to query and analyze statistics in multiple dime...

AI Technical Summary

Problems solved by technology

However, in a large-scale scenario (dozens or hundreds of services, one server with multiple log types, and one type with multiple log files), this method is inefficient, and at the same time, it needs to face log files to cover them cyclically output mode, how to archive the log data is too large, the text search is too slow, how to query and analyze statistics in multiple dimensions, etc.

Images