Network abnormal flow analysis method and system based on Spark and clustering

Abstract

The invention provides a network abnormal traffic analysis method and system based on Spark and clustering, and the method comprises the steps: carrying out the clustering analysis of network trafficthrough clustering through a Spark big data processing platform, and carrying out the abnormal traffic analysis of classified network traffic through a detection algorithm. On the basis of the primaryclustering, the abnormal flow cluster and the normal flow cluster are judged by utilizing the Mahalanobis distance, so that the purpose of distinguishing the normal flow from the abnormal flow is achieved. In order to further improve the efficiency of the method, a means of parallelizing the K-means algorithm is adopted in Spark-based clustering flow analysis, the calculation efficiency of the algorithm is improved through parallelization, the requirements of the algorithm for machine memory and kernel processing are reduced, and the practicability of the algorithm is improved.

Description

technical field [0001] The present invention relates to the technical field of communications, in particular to a method and system for analyzing network abnormal traffic based on Spark and clustering. Background technique [0002] After decades of vigorous development, Internet applications have become an indispensable part of our daily life, constantly affecting and changing our lives. Nowadays, network development is becoming more and more virtualized, distributed, and dynamic. New technologies, new foundations, new requirements, and new applications make the network environment increasingly complex, and the traffic in the network is growing geometrically. With the accelerated development of technologies such as the Internet, Internet of Things, cloud computing, and big data, the connotation of the information and communication network industry has also been continuously enriched, extending from traditional telecommunication services and Internet services to new formats ...

AI Technical Summary

Problems solved by technology

However, although many methods for abnormal traffic analysis have been proposed, it is not good to apply them to the analysis of large-scale network abnormal traffic.

The main reasons are: first, the original analysis methods (such as wavelet analysis, statistical analysis, etc.) are not suitable for processing data with high dimensions, and cannot reach the detection range of large-scale networks; second, the algorithm complexity of the original methods is high , usually the detection lags behind and cannot satisfy the abnormal online detection

Images