Log processing method, equipment, computer device and computer readable storage medium

Abstract

The invention is applicable to the technical field of the Internet, and provides a log processing method, equipment, a computer device and a computer readable storage medium. The method comprises thefollowing steps: acquiring log information through a Kafka system, the log information comprising log data of a plurality of items; storing the log data into topic topics with the same project names as the log data; analyzing log data obtained from topic topics according to a custom data format through a data processing module developed by a JStorm framework to obtain analyzed log data; and storing the analyzed log data into a corresponding data table in an ElasticSearch database through a data storage module developed by a JStorm framework according to a database type, a database name and a target data table specified by a preset analysis rule. According to the log processing method provided by the invention, the log data acquired from the Kafka system can be analyzed according to the self-defined data format, and the analyzed log data is stored into the corresponding table in the ElasticSearch database, so that the log data processing effect is improved.

Description

technical field [0001] The invention belongs to the technical field of the Internet, and in particular relates to a log processing method, device, computer device and computer-readable storage medium. Background technique [0002] Generally, when the log data needs to be processed, the log data is directly written in the file, and then the required data is searched in the file. However, in a big data and distributed environment, there are many log data types, large quantities, and wide distribution. Generally, large-scale systems have a distributed deployment architecture, that is, different data processing modules are deployed on different servers. Logs corresponding to each data processing module are also stored on each server. In the prior art, in the process of collecting and summarizing log data and analyzing them in a unified manner, log collection is slow, log data is numerous and complex, statistics are slow, and analysis is difficult, resulting in poor log data pr...

AI Technical Summary

Problems solved by technology

[0003] Embodiments of the present invention provide a log processing method, device, computer device, and computer-readable storage medium, aiming to solve the problem of slow log collection, large log data and miscellaneous log data in the process of collecting and summarizing log data and unified analysis in the prior art. , Slow statistics, difficult analysis, resulting in poor log data processing effect

Images