Hadoop-based k-means clustering analysis system and method of network security log

Abstract

The invention provides a hadoop-based k-means clustering analysis system and method of a network security log. The hadoop-based k-means clustering analysis system comprises a log data acquisition subsystem, a log data mixing mechanism storage management subsystem and a log data analysis subsystem. The method includes the steps that in a data storage layer, a mixing storage mechanism with Hadoop cooperating with a traditional data warehouse is adopted to store log data, a Hive operation interface is provided in a data access layer, the data storage layer and a computing layer receive instructions from a Hive engine, and efficient query analysis on the data is achieved by being matched with MapReduce through HDFS; when mining analysis is conducted on log data, MapReduce is adopted to conduct clustering mining analysis on the network security log through a k-means algorithm; the framework with the Hadoop cooperating with the traditional data warehouse is adopted, the detects of the traditional data warehouse on the aspects of mass data processing, storage and the like, and meanwhile an original traditional data warehouse is fully used; clustering analysis is conducted through the MapReduce-based k-means algorithm, and safety grade evaluation and early warning can be conducted on log data timely.

Description

technical field [0001] The invention belongs to the technical field of computer information processing, and in particular relates to a Hadoop-based network security log k-means cluster analysis system and method. Background technique [0002] With the explosion of data and the sharp increase in the amount of information, the existing traditional data warehouses of enterprises have been unable to cope with the growth rate of data. Traditional data warehouses are usually built with high-performance all-in-one machines, which are costly and poor in scalability, and traditional data warehouses are only good at processing structured data. This feature affects the mining of intrinsic value of traditional data warehouses when faced with massive heterogeneous data. This is the biggest difference between Hadoop and traditional data processing methods. We need to make reasonable use of the existing traditional data warehouse of the enterprise, and at the same time integrate the exist...

AI Technical Summary

Problems solved by technology

[0008] In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a Hadoop-based network security log k-means clustering analysis system and method, on the basis of rationally utilizing the traditional data warehouse that has been built, the big data The platform is integrated to establish a unified data storage and data processing architecture, which overcomes the shortcomings of traditional data warehouses, which are poor in scalability, only good at processing structured data, and unable to mine the intrinsic value of massive heterogeneous data.

Images