Network traffic anomaly detection method based on small amount of annotation data

Abstract

The invention discloses a network traffic anomaly detection method based on a small amount of annotation data, and the method comprises the steps: carrying out the dimension reduction of a feature vector through employing double auto-encoders, and then carrying out the supervised training through employing a deep neural network; dividing the network traffic into two types of positive samples and negative samples, and finally screening out a part of important samples in unlabeled data and submitting the samples to experts for labeling, increasing the number of labeled samples, iteratively updating an auto-encoder and a classifier, and then employing the trained classifier for detecting network traffic abnormality. According to the invention, a double-auto-encoder architecture is proposed, pure positive and negative samples are used for respectively training the auto-encoders, and the stability of the classifier is improved. Meanwhile, the loss function of the deep neural network is improved, the sample weight is adjusted in a finer-grained manner, the problem of overfitting caused by imbalance of positive and negative samples and small training sets is solved, a new method for calculating the marking value of the unmarked data is provided, the samples with high marking value are selected to be delivered to experts, and the marking cost is reduced.

Description

technical field [0001] The invention relates to a network flow anomaly detection technology, in particular to a network flow anomaly detection method based on a small amount of labeled data. Background technique [0002] As the scale of cyber attacks increases and the amount of network data grows exponentially, many companies and organizations must develop new methods to protect their networks and data in order to reduce the impact of changing threat factors. As more and more security tools and sensors are deployed in modern enterprise networks, the volume of security event and alert data continues to increase, making accurately identifying anomalies like finding a needle in a haystack. Therefore, it is imperative to rely on new technologies to assist human analysts when dealing with the monitoring, prevention, detection, and response of cybersecurity incidents and potential cyber attacks. However, most of the network traffic anomaly detection based on traditional machine l...

AI Technical Summary

Problems solved by technology

Although the performance of this method is significantly better than that of the traditional Autoencoder, Shrink AE does not apply the data information of abnormal samples, which wastes the cost of labeling, and it is difficult to cover all situations of network anomaly detection

Images