Parallel Network Traffic Anomaly Detection Method Based on Spark and Isolation Forest

A network traffic and anomaly detection technology, applied in the field of network security, can solve problems such as inability to efficiently process large-scale network traffic data, achieve good scalability, keep the accuracy unchanged, and reduce data processing time.

Active Publication Date: 2021-05-11
GUILIN UNIV OF ELECTRONIC TECH
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The present invention aims at the problem that the traditional isolated deep forest algorithm is limited by the maximum capacity of the memory in processing data, thereby being unable to efficiently process large-scale network traffic data, and provides a parallel network traffic anomaly detection method based on Spark and isolated forest

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Parallel Network Traffic Anomaly Detection Method Based on Spark and Isolation Forest
  • Parallel Network Traffic Anomaly Detection Method Based on Spark and Isolation Forest
  • Parallel Network Traffic Anomaly Detection Method Based on Spark and Isolation Forest

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in combination with specific examples and with reference to the accompanying drawings.

[0028] A parallel network traffic anomaly detection method based on Spark and isolation forest, its overall structure diagram is as follows figure 1 As shown, collect network traffic sample data to build a training sample set, use the Spark platform to merge isolated tree (iTree for short) collections to build an isolated forest model in parallel, and save the results to the Hadoop distributed file system (HDFS for short), on this basis Perform anomaly evaluation to count and output the results, which specifically includes the following steps:

[0029] Step S1. Build an isolation forest anomaly detection model: Randomly sample the data set to obtain sub-sample data and construct multiple iTrees to realize model construction....

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a parallel network traffic anomaly detection method based on Spark and isolation forest. On the basis of the isolation forest algorithm, with the help of parallel processing technology Spark, computing resources are provided for the tree building process and abnormal evaluation process, and the iForest algorithm modeling process is realized. The parallelization and batch processing of anomaly evaluation can effectively improve the accuracy of anomaly detection and reduce the time complexity. At the same time, it also provides a new idea and theoretical method basis for network traffic anomaly detection.

Description

technical field [0001] The invention relates to the technical field of network security, in particular to a parallel network flow anomaly detection method based on Spark and isolation forest. Background technique [0002] With the rapid development of new technologies such as "Internet +", cloud computing, and big data, network applications and demands are gradually increasing, the network scale is increasing, and the network topology is becoming more and more complex. At the same time, network traffic data also shows an explosive growth trend, which makes network traffic anomaly detection face severe challenges. Network traffic anomaly detection is a scientific and effective method to filter out abnormal information in traffic data. It can effectively identify and locate the security status of the network, and discover potential threats and attacks, thereby ensuring the safe operation of the network environment. [0003] As an anomaly detection algorithm, the isolation dee...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/06
CPCH04L63/1425H04L63/1441
Inventor 陶晓玲赵培超彭洋刘丽燕王勇史科杏强保华
Owner GUILIN UNIV OF ELECTRONIC TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap