Log data processing system and method based on streaming processing

Abstract

The invention discloses a log data processing system and method based on streaming processing, and the system comprises a rule configuration module which is used for configuring the processing parameters of log flow processing on an interface; a strategy testing module used for carrying out data testing on a log analysis function and a log source; a visualization module used for building a visualization interaction interface on the basis of a Vue framework; a processor adding module used for distributing log data processing tasks to the streaming processing engine cluster for execution; ae data analysis module used for officially processing and analyzing the data and converting the data from unstructured data into structured data in a Grok function analysis mode; and a data conversion module used for matching fields in the structured data and converting the data according to a configuration rule. Real-time data analysis of massive logs, word segmentation generation of an analytical expression under the condition of small and medium data volume, abnormal data processing and real-time measurement of an analytical strategy are realized.

Description

technical field [0001] The invention relates to the technical fields of information technology and data business, and in particular to a stream processing-based log data processing system and method. Background technique [0002] In the process of development and growth of modern Internet companies, log data is an important reference indicator for whether their business is running well, whether the system is healthy, and whether it is under attack. At the same time, when a problem occurs in the system, whether developers or operation and maintenance personnel can solve the problem in a timely manner depends largely on a good log monitoring platform. Among them, the log parsing solution, as a special part of the log monitoring platform, not only needs to be able to analyze basic log data, but also should be able to perform distributed expansion, high-performance computing capabilities, and the parsing process does not affect the business cluster. Ability, analysis function a...

AI Technical Summary

Problems solved by technology

[0004]However, the existing log parsing schemes and technologies have relatively powerful parsing capabilities and storage templates when parsing logs

However, its consumption performance is not low, and its reusability is not strong, and it cannot handle abnormal data that cannot be parsed well.

Parsing logic can only be entered manually, which requires too much professionalism and consumes a lot of time for data developers

[0005] In addition, due to the lack of testing capabilities for analytic expressions or analytic functions under effective real data, and it cannot provide data developers with a simple and easy-to-operate visual interface , resulting in a large learning cost for data developers, which needs to be solved

Images