Log analysis method and system

An analysis method and log technology, applied in the field of log analysis methods and systems, can solve problems such as being unable to deal with the real-time processing of massive log data, unable to meet the real-time processing requirements of log data, etc., so as to meet the real-time processing requirements, relieve processing pressure, and improve processing. effect of speed

Pending Publication Date: 2018-04-20
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF7 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] With the explosive growth of business data scale and higher and higher requirements for real-time data processing capabilities, frameworks such as Hadoop, which dominate batch processing systems, cannot m

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log analysis method and system
  • Log analysis method and system

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0018] The core idea of ​​the present invention is: introduce the Stom open source framework to process the logs in real time, and combine the batch processing mode to relieve the system pressure, so that the real-time processing of massive log data can be realized.

[0019] figure 1 It is a schematic flow diagram of the method of the embodiment of the present invention, such as figure 1 As shown, the method embodiment mainly includes:

[0020] Step 101, collect log data generated by each application system in real time, and cache them in a data cache server.

[0021] Preferably, the Flume system can be used to collect the log data in real time. The Flume system is a distributed massive log collection system, which is introduced in the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a log analysis method and system. The method comprises the steps of collecting log data generated by application systems in real time and caching the log data in a data cache server; obtaining the log data in real time according to a caching sequence by utilizing a Storm system through a data processing server, putting the log data into preset batch processing containers, when a preset processing condition is met, performing analysis processing on logs in the batch processing containers, and adding the batch processing containers loaded with analysis processing resultsto an index queue; and when the index queue is non empty, extracting the batch processing containers from the index queue in sequence in real time by the Storm system, storing the log data in the extracted batch processing containers, and creating corresponding data indexes. By adopting the method and the system, large-scale log data can be processed in real time.

Description

technical field [0001] The invention relates to big data processing technology, in particular to a log analysis method and system. Background technique [0002] At present, the IT system of most companies will be composed of multiple independent software systems. Generally speaking, no system is 100% reliable and problem-free. Users will have some problems during use, or some incorrect operations by users will lead to unexpected results. The logs generated by the system can objectively reflect the actual operation of the software system and provide strong support for software performance tuning and troubleshooting. When the above problems occur, it is usually the background R&D personnel to locate the problem and solve it by searching logs and troubleshooting. [0003] In order to be able to analyze massive logs in a timely and effective manner, quickly and accurately locate the cause of problems, and enable system maintenance personnel to find and solve problems before us...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/1815
Inventor 程鹏
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap