Network traffic classification device and method based on Spark performance optimization
A technology of network traffic and classification device, which is applied in data exchange network, instrument, character and pattern recognition, etc., to achieve the effect of improving processing speed, meeting high real-time requirements, and meeting the needs of network traffic classification and processing.
Inactive Publication Date: 2020-11-10
GUIZHOU UNIV
View PDF6 Cites 2 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] The purpose of the present invention is to propose a network traffic classification device and classification method based on Spark performance optimization to solve the problem that the Crail-Spark-IO plug-in cannot handle aggregation operators in a multi-partition environment, so that network service providers Efficiently classifies network traffic quickly and accurately
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment approach
[0069] According to the present invention figure 1 The flow of the network traffic classification device based on Spark performance optimization is shown, including a data preprocessing module, a model training module, a real-time classification module, and a Spark performance optimization module.
[0070] In the data preprocessing module;
[0071] Collect the original pcap packet traffic data of the backbone network through the data collection unit;
[0072] Extract time-related features from the collected pcap package by a feature extraction unit, and convert it into a csv file;
[0073] Use the data cleaning unit to clear empty records and duplicate records in the csv file, and save them to the HDFS file system.
[0074] In the module training model:
[0075] Use the cleaned csv file as input data to train the weight random forest classification model;
[0076] The classification accuracy of each decision tree in the random forest to its out-of-band data is used as the ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a network traffic classification device and method based on Spark performance optimization, and belongs to the technical field of network traffic classification. The device comprises a data preprocessing module used for collecting and extracting time-related features from the original flow data; a model training module used for classifying the network flow; a real-time classification module is used for loading the data processed by the preprocessing module to the classification model trained by the model training module and classifying the data under the Topic; and a Spark performance optimization module used for providing performance optimization support for the model training module and the real-time classification module. A flow chart is constructed through the Spark Shuffle performance optimization architecture diagram and the weight random forest algorithm to realize rapid and accurate classification of the network flow, different service strategies can be provided for network service providers for different application scenes, and powerful support is provided for further improving the network service quality and guaranteeing the network security.
Description
technical field [0001] The invention relates to a network traffic classification device and a classification method based on Spark performance optimization, and belongs to the technical field of network traffic classification. Background technique [0002] The 44th "Statistical Report on Internet Development in China" shows that as of June 2019, the number of Internet users in my country reached 854 million, an increase of 25.98 million from the end of 2018, and the Internet penetration rate reached 61.2%, an increase of 1.6 percentage points from the end of 2018. With the increasing number of netizens in our country, the further improvement of the Internet penetration rate, the continuous emergence of various new applications, and the continuous increase of network bandwidth, huge network traffic is generated all the time. Faced with such a huge network traffic, how can network service providers quickly and accurately classify network traffic efficiently, so as to provide d...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/26H04L12/24G06K9/62
CPCH04L43/04H04L43/12H04L41/145G06F18/241
Inventor 申国伟杨可郭春崔允贺
Owner GUIZHOU UNIV