Log management method and system

A management system and management method technology, applied in the field of log management methods and systems, to achieve the effects of improving efficiency, preventing parsing errors, and ensuring scalability

Inactive Publication Date: 2017-03-15
IOL WUHAN INFORMATION TECH CO LTD
View PDF7 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is to provide a convenient log management system for development engineers and data engineers, which can search and sta...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log management method and system
  • Log management method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] The technical solutions of the present invention will be further specifically described below in conjunction with the accompanying drawings and specific embodiments.

[0036] In order to solve the above technical problems, the present invention provides a log management method, such as figure 1 as shown,

[0037] figure 1 A schematic flow diagram of a log management method is shown, which is characterized by comprising the following steps:

[0038] (1) Collect logs, collect some logs generated by the demand side, and use the open source Logstash tool to collect several logs generated by several application nodes;

[0039] (2) Centralize the logs collected in the above steps, preferably use the open source Apache Kafka cluster for log concentration, and push them to the log preprocessing module;

[0040] (3) After receiving the above-mentioned concentrated data, perform log preprocessing, the steps of the log preprocessing include formatting, data filtering, and log c...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

Disclosed is a log management method. The log management method comprises the following steps of performing log collection, wherein a plurality of logs generated by a demand side are collected; performing log gathering and pushing; performing log preprocessing, wherein format arrangement, data filtering and log classification are included, and log classification is used for dividing logs into cold data and hot data; performing log storage, wherein the hot data is stored into a distributed file storage system, and the hot data is stored into a distributed search engine; performing log moving, wherein cooled logs in the distributed search engine are moved into the distributed file storage system; performing log retrieval analysis; performing result output. A convenient log management system is provided for development engineers and data engineers, the structured data can be subjected to search and statistical analysis, the consumption for processing file logs through scripts can be reduced, the working efficiency can be greatly improved, and the method is applicable to management of a large number of logs.

Description

technical field [0001] The invention belongs to computer software, in particular to a log management method and system. Background technique [0002] With the advent of the information age, even small and medium-sized companies can generate hundreds of millions of log data every day. Traditional open source relational databases cannot support such a large amount of log data. How to efficiently collect, manage, and analyze daily items Massive data generated by business has become an urgent problem to be solved. [0003] Now there is no complete set of log management system, but simply relocate the logs generated by each application from the application node to a central node on a regular basis. Then development engineers or data engineers use scripts to analyze search logs at the central node, which is very inconvenient. Searching logs is very time-consuming because there is no index. Analysis logs have a low degree of reuse of analysis scripts due to huge changes in dema...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/148G06F16/16G06F16/172G06F16/1727G06F16/1815G06F16/182
Inventor 蔡洁
Owner IOL WUHAN INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap