Log-based big data analysis method

An analysis method and big data technology, applied in the field of log files, can solve the problems of log statistics and retrieval troubles, inaccurate data, low efficiency, etc., and achieve convenient and intuitive indexing and searching

Inactive Publication Date: 2018-11-13
BEIJING CHINA POWER INFORMATION TECH +3
View PDF3 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0028] As can be seen from the above steps, after the centralized management of logs in the prior art, the statistics and retrieval of logs has become a more troublesome thing. Generally, Linux commands such as grep, awk and wc are used to realize retrieval and statistics, but for the requirements In the case of higher query, sorting and statistics requirements and a huge amount of data, if the traditional method is still used, there will be problems of low efficiency and the final data is not accurate enough

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log-based big data analysis method
  • Log-based big data analysis method
  • Log-based big data analysis method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0052] In order to achieve comprehensive monitoring of the application service program of the application system, implement service deployment at low cost, and provide users with a high-quality and efficient service system, this solution has a presentation layer (View), a control layer (Control), and a service layer from top to bottom. (Service), basic data layer. Wherein, the basic data layer includes: data collection, data analysis, data storage, and data cache.

[0053] Such as image 3 Shown is a schematic diagram of the overall system logic architecture of the embodiment of the present invention.

[0054] The basic data layer is the data source of the entire system, and it interacts with the data collection layer d...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a log-based big data analysis method, the log-based big data analysis method mainly includes the steps of collecting data in each server according to a configuration file of arelated Beats application; and collecting the data collected by the Beats application by LogStash and storing the data in ElasticSearch in an order generated by automatic indexing. The ElasticSearch provides a concise Restful-style query interface to the outside world and develops Lucene through the Java program and uses the Lucene as its core to realize query data calling, i.e. all indexing and searching functions are realized, the functions of accurately locating and searching log information and analyzing related data are achieved, and accurate business data is acquired. Finally, Echart isused for visual display to hide the complexity of the Lucene, which makes indexing and searching easy and intuitive.

Description

technical field [0001] The invention relates to the technical field of log files, in particular to a log-based big data analysis method. Background technique [0002] In recent years, due to the continuous growth of data in the Internet field, companies are faced with the need to process massive data. Data analysis mainly provides support for the company's operation and maintenance decision-making, mainly based on the logs generated by each server. Based on the data information provided by these logs, the user access and data flow are of great significance in terms of time dimension, product line dimension, and domain name dimension. Clear quantitative data, so as to achieve the purpose of providing reference for the company's server operation and maintenance management, traffic distribution, and forecasting. [0003] The main technology of the existing scheme is to adopt the ELK-Stack (elasticsearch, logstash&kibana, analysis and visualization platform) framework, which is...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 邢占伟朱承治戴波张浩周春高磊冷曼
Owner BEIJING CHINA POWER INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap