Log information acquiring and monitoring method

A log and target information technology, applied in digital data information retrieval, instrumentation, unstructured text data retrieval, etc., can solve the problems of ELK architecture without message queue cache, data loss, etc., to reduce data, reduce memory pressure, improve efficiency effect

Pending Publication Date: 2022-08-05
福建财通信息科技有限公司
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, the ELK architecture does not have a message queue cache, and there is a hidden danger of data loss. With the construction of the system, the amount of logs will increase. The storage of a large amount of log data has brought great challenges to disk resources. The existing distributed log They are all stored in the ElasticSearch memory. Although this method is convenient for log retrieval and analysis, it requires a lot of memory.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log information acquiring and monitoring method
  • Log information acquiring and monitoring method
  • Log information acquiring and monitoring method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 2

[0091] like figure 1 As shown, what is different from Embodiment 1 is that in this embodiment of the present invention, there is a method for acquiring and monitoring log information, the method further includes performing data query on hot data in the ElasticSearch distributed search engine, and displaying the data query results; the step of data query includes searching and querying data in Elasticsearch according to preset grammar rules; the preset grammar rules include grammar keywords, query words, query domains, wildcard queries, fuzzy queries, proximity Query and interval query syntax rules.

[0092] The above content is completed by the log analysis module, which is to search and query the data in Elasticsearch by grammatical rules such as grammatical keywords, query words, query domains, wildcard queries, fuzzy queries, proximity queries, and interval queries. You can easily use Kibana to use charts, tables and maps to analyze and present data in a variety of ways. ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the technical field of data monitoring and processing, and particularly discloses a log information acquisition and monitoring method, which comprises the following steps: establishing a connection channel with a log source, and acquiring log data in real time; performing log compression, log cutting, format arrangement, data filtering and log classification on the log data to obtain cold data and hot data; the cold data is stored in a MinIO distributed file storage system, and the hot data is stored in an ElasticSearch distributed search engine; the log data in the ElasticSearch distributed search engine is cooled, and the cooled log data is moved to the MinIO distributed file storage system. According to the method, multiple log sources can be efficiently collected, data are not lost, and the log retrieval efficiency is improved; massive data can be stored through distributed storage, so that the memory pressure is reduced; cold and hot data can be processed in time, data of log data on a retrieval system is reduced, and log retrieval analysis efficiency is better improved.

Description

technical field [0001] The invention relates to the technical field of data monitoring and processing, in particular to a log information acquisition and monitoring method. Background technique [0002] With the rise and development of cloud computing, mobile Internet, and the Internet of Things, the era of big data has arrived. The amount of log data generated by most Internet companies every day can reach the level of hundreds of millions. How to efficiently collect, manage and analyze the massive data generated by daily business has become an urgent problem to be solved. [0003] The existing massive data processing methods are mainly distributed log systems. Most of these log systems use the ELK architecture. The so-called ELK architecture is a system based on ElasticSearch, Logstash, and Kibana. Among them, ElasticSearch is an open source distributed search engine that provides three functions of collecting, analyzing and storing data. Logstash is a tool for log collec...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/35G06F16/31G06F16/332
CPCG06F16/35G06F16/332G06F16/31Y02D10/00
Inventor 张雄明
Owner 福建财通信息科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products