Log processing system and processing method thereof

A processing system and processing method technology, applied in the direction of file system, file system type, electronic digital data processing, etc., can solve the problem of large consumption of computing resources, achieve less resource occupation, solve the problem of large computing resource consumption, and reduce occupation Effect

Pending Publication Date: 2020-07-17
POSBANK
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The embodiment of the present invention provides a log processing system and its processing method, so as to at least solve the technical problem in the prior art that consumes a lot of computing resources during log processing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log processing system and processing method thereof
  • Log processing system and processing method thereof
  • Log processing system and processing method thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0025] According to an embodiment of the present invention, an embodiment of a log processing system is provided, figure 2 is a schematic diagram of a log processing system according to an embodiment of the present invention, such as figure 2 As shown, the system includes the following steps:

[0026] A log data collector 20, the log data collector is composed of binary files, and is used to collect local log data.

[0027] Specifically, the above-mentioned log data collector can be Filebeat, which is a log data collector for local files, which can monitor log directories or specific log files (tail files), and forward them to Elasticsearch or Logstatsh for indexing. The collection, parsing and visualization of common log formats can be simplified with one specified command. Since Filebeat is just a binary file without any dependencies and takes up very little resources, using Filebeat for log collection can greatly reduce the resource usage during log collection.

[0028...

Embodiment 2

[0042] According to an embodiment of the present invention, a log processing method embodiment of a log processing system is provided. It should be noted that the steps shown in the flow charts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions , and, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

[0043] image 3 is a flowchart of a log processing method of a log processing system according to an embodiment of the present invention, the log processing system includes the log processing system described in Embodiment 1, such as image 3 As shown, the method includes the following steps:

[0044] Step S302, collecting local log data through a log data collector, wherein the log data collector is composed of a binary file.

[0045] Specifically, the above-mentioned log data collector can b...

Embodiment 3

[0060] According to an embodiment of the present invention, a storage medium is provided, and the storage medium includes a stored program, wherein, when the program is running, the device where the storage medium is located is controlled to execute the log processing method described in any one of Embodiment 2 .

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a log processing system and a log processing method. The system comprises a log data collector which is composed of binary files and used for collecting local log data; a distributed flow platform which is used for obtaining the log data collected by the log data collector and storing the log data; and an analyzer which is used for acquiring the log data from the distributed stream platform and analyzing the log data. The log processing method and device solve the technical problem that in the prior art, when logs are processed, computing resource consumption is large.

Description

technical field [0001] The invention relates to the field of computers, in particular to a log processing system and a processing method thereof. Background technique [0002] The distributed log system is implemented based on ELK (Elasticsearch, Logstash, Kibana) open source components. ELK is a collection of Elasticsearch, Logstash, and Kibana open source software. It is an open source solution for a log management system. ELK mainly performs log search, analyzes and obtains data, and displays them in real time. [0003] Among them, the main function of Logstash is a data collector, which collects and analyzes data in various formats and channels through it, formats and outputs them to Elasticsearch, and finally summarizes, analyzes and searches them through the friendly web interface provided by Kibana. But the problem with Logstash is that its resource consumption is too large (the default heap size is 1GB). [0004] figure 1 It is a schematic diagram of the architec...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/18G06F16/182
CPCG06F16/1815G06F16/182
Inventor 徐朝辉范楠张兰英朱礼华易程星彭庚
Owner POSBANK
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap