A highly reliable real-time log collection and visual retrieval method

A log and reliable technology, applied in file metadata retrieval, digital data information retrieval, error detection/correction, etc., can solve problems such as low efficiency, slow text search of log data volume, etc.

Active Publication Date: 2019-03-29
FOCUS TECH
View PDF4 Cites 57 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, in a large-scale scenario (dozens or hundreds of services, one server with multiple log types, and one type with multiple log files), this method is inefficient, and at the same time, it n

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A highly reliable real-time log collection and visual retrieval method
  • A highly reliable real-time log collection and visual retrieval method
  • A highly reliable real-time log collection and visual retrieval method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] The present invention will be described in further detail below in conjunction with the accompanying drawings. Design real-time log collection architecture and real-time log visualization retrieval framework, such as figure 1 shown.

[0034] Design real-time log collection architecture. Design a general real-time log collection architecture to ensure complete log collection:

[0035] It is equipped with real-time log collection, encrypted transmission of log data, integrity check of log data, missing alarm, and visual system architecture of inspection results. Deploy the Flume client and the Flume custom plug-in on the target log server, and configure the Flume configuration file to read the target log with multiple threads, and transmit the read data to the Kafka cluster.

[0036] There are log data compression and encryption algorithms. When Flume sends data, the data encryption and compression algorithms are designed, and then the data is decompressed, decrypted and...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

A high-reliable real-time log collection and visual retrieval method comprises the following steps: 1) setting up a real-time log collection structure, which includes real-time log collection, log data encryption transmission, log data integrity check, missing alarm, and visual inspection result system structure; Deploying Flume client and custom Flume plug-in, Kafka cluster, HDFS distributed filesystem on the target log server; configuring the Flume configuration file, reading the multithreaded target log, and transferring the read data to the Kafka cluster. Kafka clusters cache log data bycreating partitions and Topic classifications of Kafka clusters; 2) log data integrity chec flow and algorithm steps: when log data is transmit, a self-defined Flume plug-in sends checking messages tothat collected log file, so that the Storm cluster checks the integrity of log data stored in the HDFS when landing log data; 3) slice log file deletion and missing data retransmission flow and algorithm step.

Description

technical field [0001] The invention relates to the field of real-time log collection and retrieval, in particular to a highly reliable real-time log collection and visual retrieval method. Background technique [0002] Generally, a large-scale system is a distributed deployment architecture. Different service modules are deployed on different servers, and the logs generated by the services (application logs, access logs, security logs, etc.) are therefore scattered on each server. The retrieval and statistics of log information can be realized directly on the log server through grep, awk, and wc commands. However, in large-scale scenarios (dozens or hundreds of services, one server with multiple log types, and one type with multiple log files), this method is inefficient, and at the same time, it is necessary to face log files to cover them circularly How to archive the log data is too large, the text search is too slow, how to query and analyze statistics in multiple dime...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F11/30G06F16/14G06F16/182G06F16/13G06F16/18
CPCG06F11/3065G06F11/3082G06F11/3093
Inventor 史飞悦房鹏展
Owner FOCUS TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap