Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Log data processing system and method based on streaming processing

A stream processing and data processing technology, applied in electrical digital data processing, special data processing applications, digital data information retrieval, etc., can solve problems such as time consumption of data developers, low performance consumption, and inability to process well. Achieve the effect of ensuring real-time stable processing and analysis, saving a lot of time, and saving time consumption

Active Publication Date: 2020-07-28
BEIJING UNIV OF POSTS & TELECOMM
View PDF5 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004]However, the existing log parsing schemes and technologies have relatively powerful parsing capabilities and storage templates when parsing logs
However, its consumption performance is not low, and its reusability is not strong, and it cannot handle abnormal data that cannot be parsed well.
Parsing logic can only be entered manually, which requires too much professionalism and consumes a lot of time for data developers
[0005] In addition, due to the lack of testing capabilities for analytic expressions or analytic functions under effective real data, and it cannot provide data developers with a simple and easy-to-operate visual interface , resulting in a large learning cost for data developers, which needs to be solved

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log data processing system and method based on streaming processing
  • Log data processing system and method based on streaming processing
  • Log data processing system and method based on streaming processing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0037]The log data processing system and method based on stream processing according to the embodiments of the present invention will be described below with reference to the accompanying drawings. First, the log data processing system based on stream processing according to the embodiments of the present invention will be described with reference to the accompanying drawings.

[0038] figure 1 It is a schematic structural diagram of a stream processing-based log data processing system according to an embodiment of the present in...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a log data processing system and method based on streaming processing, and the system comprises a rule configuration module which is used for configuring the processing parameters of log flow processing on an interface; a strategy testing module used for carrying out data testing on a log analysis function and a log source; a visualization module used for building a visualization interaction interface on the basis of a Vue framework; a processor adding module used for distributing log data processing tasks to the streaming processing engine cluster for execution; ae data analysis module used for officially processing and analyzing the data and converting the data from unstructured data into structured data in a Grok function analysis mode; and a data conversion module used for matching fields in the structured data and converting the data according to a configuration rule. Real-time data analysis of massive logs, word segmentation generation of an analytical expression under the condition of small and medium data volume, abnormal data processing and real-time measurement of an analytical strategy are realized.

Description

technical field [0001] The invention relates to the technical fields of information technology and data business, and in particular to a stream processing-based log data processing system and method. Background technique [0002] In the process of development and growth of modern Internet companies, log data is an important reference indicator for whether their business is running well, whether the system is healthy, and whether it is under attack. At the same time, when a problem occurs in the system, whether developers or operation and maintenance personnel can solve the problem in a timely manner depends largely on a good log monitoring platform. Among them, the log parsing solution, as a special part of the log monitoring platform, not only needs to be able to analyze basic log data, but also should be able to perform distributed expansion, high-performance computing capabilities, and the parsing process does not affect the business cluster. Ability, analysis function a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2455
CPCG06F16/24568
Inventor 鄂海红陈沅星孙美杰吴超宇曾地
Owner BEIJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products