WEB abnormal traffic monitoring method based on integrated learning

Abstract

The invention sets forth a WEB abnormal traffic monitoring method based on integrated learning. The method comprises five processes of data preprocessing, construction of feature engineering, data set reconstruction, establishment and fusion of a model and model test. The data preprocessing refers to valid information extraction on URL (Uniform Resource Locator) data. The construction of feature engineering refers to extraction and construction of URL features by adopting a statistical method of information entropy, mutual information or the like. After the feature engineering is constructed, a data set is adjusted for different access properties, and input to four machine learning algorithms of XGBost, LightGBM and the like for supervised learning. After learning devices are constructed, the learning devices are integrated by adopting a Bagging framework. Based on the original data set, a data set is reselected for classified prediction, labels are decided in a most voting manner, and the accuracy of the model is inspected. In the process of using the model, a URL is input to the model, five sub-models in the model give respective label probability, and the label having the highest probability is given as a final label.

Description

technical field [0001] The invention belongs to the technical field of machine learning, and specifically relates to various statistical algorithms and machine learning algorithms. The algorithm adopts a new feature extraction method, innovatively integrates statistics and machine learning algorithms, and realizes the monitoring of WEB abnormal traffic. Background technique [0002] 1. Network security issues in the information age [0003] Today, with the explosion of information, the scale of computer networks and the number of Internet users have reached an unprecedented scale, and what follows is that the problem of network security has become more prominent. As the most important means of defending against network attacks, the development and upgrading of abnormal traffic monitoring is imminent. After more than 20 years of development, the research on traffic monitoring has evolved into many branches. However, in practical applications, the effect is not satisfactory. ...

AI Technical Summary

Problems solved by technology

[0004] 1) Using fixed rules for real-time monitoring of violation behavior patterns leads to a high false positive rate;

[0005] 2) When using feature matching, the feature library needs to be manually updated, and unknown attack methods cannot be detected;

[0006] 3) The huge number of rules greatly affects the detection performance of the system, and the maintenance of the rule base becomes difficult to maintain;

[0007] 4) When the abnormal traffic detection system with blocking function detects normal communication behavior by mistake, normal communication will be blocked;

[0008] 5) When there is a bottleneck in the data storage capacity of the monitoring system, it is vulnerable to denial of service attacks and communication will be blocked

[0012] However, a single machine learning cannot perfectly solve the problem

The statistical method considers that all events are generated by the statistical model, which ignores the risk that the pre-set distribution model in the parameter method may not match the real data, resulting in a large deviation from the expected results

In addition, most systems composed of statistical models work offline and cannot meet the requirements of real-time monitoring. Therefore, very efficient performance is required to achieve high accuracy; and statistical methods are very difficult to determine the threshold. will lead to an increase in the false negative rate

[0013] Although the machine learning algorithm can seamlessly combine the prior knowledge and the posterior knowledge, and overcome the shortcomings of the unintuitive framework, the simple classification and clustering algorithms will lead to overkill due to noise data interference, wrong sampling methods, and too many modeling variables. Fitting, can not achieve a good monitoring effect

Moreover, the accuracy of the model depends on certain assumptions, which are reflected in the behavior patterns of the target system and network, and violations of the assumptions will result in a significant drop in accuracy

Images