A hadoop-based network security log k-means clustering analysis system and method

Abstract

A network security log k‑means clustering analysis system and method based on Hadoop, including a log data acquisition subsystem, a log data hybrid mechanism storage management subsystem, and a log data analysis subsystem; in the data storage layer, Hadoop and traditional data are used The hybrid storage mechanism of warehouse collaboration stores log data and provides an interface for Hive operations in the data access layer. The data storage layer and computing layer receive instructions from the Hive engine and implement efficient query and analysis of data through HDFS and MapReduce; in the log processing When mining and analyzing data, MapReduce is used to implement k‑means algorithm for cluster mining and analysis; Hadoop and traditional data warehouse collaboration architecture are used to make up for the shortcomings of traditional data warehouses in massive data processing and storage, while also making the original The traditional data warehouse makes full use of its resources; the k-means algorithm based on MapReduce is used for cluster analysis, which can conduct timely security level assessment and early warning of log data.

Description

technical field [0001] The invention belongs to the technical field of computer information processing, and in particular relates to a Hadoop-based network security log k-means cluster analysis system and method. Background technique [0002] With the explosion of data and the sharp increase in the amount of information, the existing traditional data warehouses of enterprises have been unable to cope with the growth rate of data. Traditional data warehouses are usually built with high-performance all-in-one machines, which are costly and poor in scalability, and traditional data warehouses are only good at processing structured data. This feature affects the mining of intrinsic value of traditional data warehouses when faced with massive heterogeneous data. This is the biggest difference between Hadoop and traditional data processing methods. We need to make reasonable use of the existing traditional data warehouse of the enterprise, and at the same time integrate the exist...

AI Technical Summary

Problems solved by technology

[0008] In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a Hadoop-based network security log k-means clustering analysis system and method, on the basis of rationally utilizing the traditional data warehouse that has been built, the big data The platform is integrated to establish a unified data storage and data processing architecture, which overcomes the shortcomings of traditional data warehouses, which are poor in scalability, only good at processing structured data, and unable to mine the intrinsic value of massive heterogeneous data.

Images