Method for classifying and analyzing massive logs by applying Apache Spark

Abstract

The invention discloses a method for classifying and analyzing massive logs by applying Apache Spark. The method comprises following steps: selecting log files and importing the log files in an Apache Spark environment by means of a log data interface; performing entry analysis on file names and description fields of the log files and reading key fields; classifying the key fields and forming classifying information of the log files according to a decision tree classifier; according to a Hive table structure, matching required information from the log files based on Scala to form data bars and importing the data bars into the Hive table; reading data from the Hive table and calculating status information; analyzing the frequency and the amplitude of the status to form a final data status report. The method is easy to apply and has advantages such as larger analysis speed and higher analysis accuracy.

Description

technical field [0001] The invention relates to the field of information technology, and more specifically relates to a method for classifying and analyzing massive logs using Apache Spark. Background technique [0002] With the rapid development of information technology, in order to record the working conditions of applications and systems, a large number of work logs must be generated. These log data not only include application and system status but also error information. By collecting and analyzing log data and digging out useful status and error information, system managers can learn the running status of applications and systems in a timely and accurate manner. [0003] Applications and systems generate log files frequently, so the number of these log files can be quite large. At present, the common parsing method for log files is to use MR to analyze the log file data, but MR has many defects: the abstraction level is low, and it needs to be manually written to co...

AI Technical Summary

Problems solved by technology

At present, the common parsing method for log files is to use MR to analyze the log file data, but MR has many defects: the abstraction level is low, and it needs to be manually written to complete the code, which is difficult to use; it only provides two operations, Map and Reduce, to express Insufficient power; processing logic is hidden in the code details, there is no overall logic; intermediate results are also placed in the HDFS file system; ReduceTask needs to wait for all MapTasks to be completed before starting; the delay is high, only applicable to Batch data processing, for interactive Data processing, insufficient support for real-time data processing; relatively poor performance for iterative data processing; not suitable for describing complex data processing processes

Images