Network traffic data analysis method and system

Abstract

The invention belongs to the technical field of network flow data analysis and particularly relates to a network flow data exception detection method. The method comprises the following steps of processing original network flow data captured in real time to obtain network flow data; if the network flow data is abnormal data, outputting an exception, inputting the exception data into a pre-trainedfirst exception classifier, judging that the attack type of the exception data is a known attack type, and outputting the attack type of the exception data; if the network flow data is not abnormal data, further detecting whether the network flow data is abnormal or not by adopting an unsupervised anomaly detection method; if the network flow data is abnormal data, inputting the abnormal data intoa pre-trained second abnormal classifier, judging that the type of the abnormal data is an unknown attack type, and marking the abnormal data as the unknown attack type; and if the network flow datais not abnormal data, the output being normal.

Description

technical field [0001] The present invention belongs to the field of abnormality detection technology and network traffic data analysis technology based on machine learning and big data, and in particular relates to a network traffic data analysis method and system, in particular, a network traffic analysis method based on sparse autoencoder and extreme random tree Data analysis method and system. Background technique [0002] In recent decades, with the rapid development of the Internet, people's communication methods, consumption patterns, and even the economic form of the entire country have been reshaped time and time again from the Internet of Consumers, the Internet of Industry, and the Internet of Everything. As a result, network security issues have become more and more difficult, and the endless network attacks have made traditional defense methods powerless in the face of new attack methods. From the basic data link layer to the network layer and transport layer, ...

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is to solve the problems of poor generalization performance of network security anomaly detection technology, low detection rate and high false alarm rate in existing network traffic data analysis methods. The present invention proposes a method based on sparse autoencoder and An extreme random tree network traffic data analysis method. This method can start from network flow data and select corresponding automatic encoding methods for different types of features. It can not only reduce the feature dimension, but also can analyze features such as IP addresses and protocols. Effective distance calculation provides a basis for traditional distance-based or density-based anomaly detection techniques; then, for the processed numerical features, a feature selection method based on an extreme random tree is used, which can not only reduce dimensionality, but also ensure the selected The features still have practical significance, which provides the possibility for subsequent analysis; the extracted feature set can be combined with supervised classification technology or unsupervised outlier anomaly detection technology. Experiments show that the accuracy rate has been effectively improved, and the false positive rate It is greatly reduced, and due to the recoding of data and feature engineering processing, the calculation speed becomes much faster

Images