Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Real-time association analysis method and system based on massive logs

A technology of correlation analysis and logging, applied in file systems, file system types, information technology support systems, etc., and can solve problems such as incremental updates

Active Publication Date: 2020-10-30
STATE GRID FUJIAN ELECTRIC POWER CO LTD +3
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

On the other hand, in many data mining applications including massive log data mining, the database needs to be updated continuously, so there is a problem of incremental update. It is necessary to mine the original database and then update the newly added database. to dig

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time association analysis method and system based on massive logs
  • Real-time association analysis method and system based on massive logs
  • Real-time association analysis method and system based on massive logs

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0057] The technical solution of the present invention will be further described below in combination with specific embodiments and accompanying drawings.

[0058] The invention discloses a real-time association analysis method based on massive logs, including:

[0059] Obtaining all association analysis data of the original log transaction data DB, the association analysis data including infrequent items, frequent items and association rules in the data DB;

[0060] Obtain the log data db collected in real time;

[0061] Real-time association analysis of massive logs based on the improved Storm real-time computing system, the analysis process includes:

[0062] The first-level node startup is used to track the task completion of all nodes in the stream data processing process, while the second-level node startup is used to control the working order of all nodes;

[0063] The second-level node sends the identification field of the log data db to the third-level node;

[006...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a real-time association analysis method and system based on massive logs. Through an improved Storm real-time computing system, real-time association rule mining is performed on massive log data increased in real time. Aimed at massive log data generated in a power information system, through combination of a Storm real-time computing system and an association rule mining related technology, instant association rule mining is carried out on log data newly added in real time, correlation analysis is carried out on multiple index log data of the information system, rapidpositioning can be carried out on system faults, system fault root causes can be found conveniently, and the operation and maintenance efficiency of the information system is improved.

Description

technical field [0001] The invention relates to the technical field of data mining, in particular to a real-time association analysis method and system based on massive logs. Background technique [0002] Association rule mining is a very important method in data mining, and its role is to find the correlation between data. With the vigorous development of information technology in our country, the amount of data in various fields has become more and more, which has pushed us into the era of big data. Therefore, the objects to be mined by using association rules are often huge centralized or Distributed database, which of course also includes power information system log data. On the one hand, in order to meet the requirements of big data mining in terms of storage capacity and mining capacity, a method of processing massive data in parallel is proposed. On the other hand, in many data mining applications including massive log data mining, the database needs to be updated ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/18G06F16/2458
CPCG06F16/1815G06F16/2465Y04S10/50
Inventor 徐海青周刚陈是同周晟吴树霖张江龙陶俊吴小华高扬毛舒乐梁翀浦正国胡心颖郭庆
Owner STATE GRID FUJIAN ELECTRIC POWER CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com