Parallel Network Traffic Anomaly Detection Method Based on Spark and Isolation Forest

A network traffic and anomaly detection technology, applied in the field of network security, can solve problems such as inability to efficiently process large-scale network traffic data, achieve good scalability, keep the accuracy unchanged, and reduce data processing time.

A network traffic and anomaly detection technology, applied in the field of network security, can solve problems such as inability to efficiently process large-scale network traffic data, achieve good scalability, keep the accuracy unchanged, and reduce data processing time.

CN108900476BActive Publication Date: 2021-05-11GUILIN UNIV OF ELECTRONIC TECH
4 Cites 1 Cited by

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Parallel Network Traffic Anomaly Detection Method Based on Spark and Isolation Forest
  • Parallel Network Traffic Anomaly Detection Method Based on Spark and Isolation Forest
  • Parallel Network Traffic Anomaly Detection Method Based on Spark and Isolation Forest

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in combination with specific examples and with reference to the accompanying drawings.

[0028] A parallel network traffic anomaly detection method based on Spark and isolation forest, its overall structure diagram is as follows figure 1 As shown, collect network traffic sample data to build a training sample set, use the Spark platform to merge isolated tree (iTree for short) collections to build an isolated forest model in parallel, and save the results to the Hadoop distributed file system (HDFS for short), on this basis Perform anomaly evaluation to count and output the results, which specifically includes the following steps:

[0029] Step S1. Build an isolation forest anomaly detection model: Randomly sample the data set to obtain sub-sample data and construct multiple iTrees to realize model construction....

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a parallel network traffic anomaly detection method based on Spark and isolation forest. On the basis of the isolation forest algorithm, with the help of parallel processing technology Spark, computing resources are provided for the tree building process and abnormal evaluation process, and the iForest algorithm modeling process is realized. The parallelization and batch processing of anomaly evaluation can effectively improve the accuracy of anomaly detection and reduce the time complexity. At the same time, it also provides a new idea and theoretical method basis for network traffic anomaly detection.

Description

technical field [0001] The invention relates to the technical field of network security, in particular to a parallel network flow anomaly detection method based on Spark and isolation forest. Background technique [0002] With the rapid development of new technologies such as "Internet +", cloud computing, and big data, network applications and demands are gradually increasing, the network scale is increasing, and the network topology is becoming more and more complex. At the same time, network traffic data also shows an explosive growth trend, which makes network traffic anomaly detection face severe challenges. Network traffic anomaly detection is a scientific and effective method to filter out abnormal information in traffic data. It can effectively identify and locate the security status of the network, and discover potential threats and attacks, thereby ensuring the safe operation of the network environment. [0003] As an anomaly detection algorithm, the isolation dee...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
11 May 2021
Publication
CN108900476B
IPC
H04L29/06
CPC
H04L63/1425; H04L63/1441
Inventors
陶晓玲; 赵培超