Method and system for detecting abnormal network traffic based on Spark
A traffic detection and network anomaly technology, applied in the transmission system, file system, digital transmission system, etc., to achieve the effect of optimizing the network security environment, efficiently obtaining real-time data, and reducing work pressure
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0059] The method for network abnormal traffic detection based on Spark of the present invention, the method is that the log data collected by the Flume NG convergence node passes through the Kafka cluster, and the log data is collected and analyzed based on Spark, and is performed by a trained logical regression model. Log data identification, and finally display and output the processing results of the Spark Streaming real-time calculation program to the user; the details are as follows:
[0060] S1. Define Spark to read the log data access process of the current system from the log file, obtain the parameters of the total number of requests, number of visitors, resource flow size, and log size from the log file, and Spark obtains the current access according to the real-time status code ratio preProcessing The status code method of the access record information is preprocessed and marked;
[0061] S2, Spark loads the processed log data for processing, and obtains the ngram ...
Embodiment 2
[0087] The system of Spark-based network abnormal traffic detection in the present embodiment, the system includes,
[0088] The definition module is used to define the log data access process of the current system that Spark reads from the log file, obtains the parameters of the total number of requests, the number of visitors, the size of resource flow, and the size of the log from the log file, and Spark preProcessing according to the real-time status code ratio The method of obtaining the status code of the current visit, performing data preprocessing on the visit record information and adding tags;
[0089] The loading module is used for Spark to load and process the processed log data to obtain the ngram sequence;
[0090] The extraction module is used to load data into Spark and extract features from the data through TF-IDF. It performs hash word frequency statistics and discrimination degree estimation, and uses a classifier to perform logistic regression operations to...
Embodiment 3
[0103] The embodiment of the present invention also provides an electronic device, including: a memory and a processor;
[0104] Wherein, the memory stores computer-executable instructions;
[0105] A processor executes the computer-executed instructions stored in the memory, so that a processor executes the Spark-based network abnormal traffic detection method described in any one of the present invention.
[0106] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, Discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or the processor may be any conventional processor or the like.
[0107] The memory can be used to store computer programs and / or modules, and the processor implements various funct...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More