An encrypted traffic network intrusion detection method based on ensemble learning

Abstract

The application discloses an encrypted traffic network intrusion detection method based on integrated learning, which is applied to instant identification and classification of encrypted malicious traffic in a network, and comprises encrypted traffic preprocessing and feature extraction, an encrypted traffic time sequence feature clustering analysis module, an encrypted traffic statistical feature support vector machine classification module, an encrypted source traffic deep learning anomaly detection module and an integrated strategy of each traffic analysis module. The method can cope with unknown malicious traffic categories appearing in the network, timely issues an alarm before the malicious traffic produces an effect, has a certain resistance to packet filling escape attack, and thus protects network infrastructure integrity, availability and user data security.

Description

Technical Field

[0001] The present invention relates to an encrypted malicious traffic identification network intrusion detection technology based on ensemble learning, belonging to the field of information security technology. Background Art

[0002] Network intrusion detection, which identifies malicious traffic and detects intrusions through traffic analysis, is a key research topic in network security. Detection methods based on machine learning and deep learning have gained widespread application due to their superior accuracy and low reliance on expert analysis. However, with advances in encryption technology and growing user awareness of privacy, more and more network traffic is encrypted. Traditional protocol analysis, port analysis, and packet header analysis are becoming increasingly ineffective, leading to the emergence of intrusion detection methods based on encrypted traffic analysis as a new research hotspot.

[0003] When analyzing encrypted traffic, based on the time series features derived from the length of a single packet in the traffic, statistical features such as flow duration and the number of forward and reverse packets are considered in units of flow, which becomes an important basis for classification. Time series features have been proven to be effective in practice. Sometimes, encrypted traffic can be accurately classified using only time series features [4]. In addition, academic circles have proposed efficient time series clustering algorithms such as Symbolic Pattern Forest (SPF) [1]. However, time series features are often affected by packet padding. Attackers may pad the traffic to cover up the length features. In this case, the performance of the classifier built based on time series features will drop sharply, while statistical features that are less affected by packet padding will be more effective. For statistical features, the CS++SVM classifier [2] may be a good processing algorithm. In addition, since source traffic contains all the information in the traffic, if it can be properly used through deep learning methods, such as the anomaly detection algorithm anomaly transformer [3], it can also make outstanding contributions to intrusion detection. How to maximize the potential value of these features has become an urgent problem to be solved.

[0004] Furthermore, even if accurate classifiers are constructed, they can still be difficult to deploy in the real world. Unknown traffic on networks, specifically traffic generated by zero-day applications and zero-day attacks, is constantly emerging, and traditional traffic classification methods are unable to handle this traffic, resulting in serious misjudgments. Therefore, the issue of unknown traffic must also be properly addressed in network intrusion detection scenarios.

[0005] References

[0006] [1]Xiaosheng Li, Jessica Lin and Liang Zhao.Linear Time ComplexityTime Series Clustering with Symbolic Pattern Forest.In IJCAI, pages 2930-2936, 2019.

[0007] [2]Alistair Shilton,SutharshanRajasegaar and MarimuthuPalaniswami.Multiclass Anomaly Detector:the CS++Support VectorMachine.Journal of Machine Learning Research,213:1-39,2020.

[0008] [3] Jiehui Xu, Haixu Wu, Jianmin Wang and Mingsheng Long, “AnomalyTransformer: Time Series Anomaly Detection with Association Discrepancy”. InICLR, 2022.

[0009] [4] M.Shen, Y.Liu, L.Zhu, Summary of the Invention

[0010] The present invention aims to solve the technical problems existing in the prior art and provides an encrypted traffic network intrusion detection method based on ensemble learning, which is dedicated to solving the problem of detecting malicious traffic containing unknown attacks.

[0011] To solve the above problems, the present invention discloses a new algorithm for network intrusion detection and proposes a network intrusion detection method for identifying encrypted malicious traffic based on ensemble learning, which includes the following steps:

[0012] S1, extract traffic features: extract timing features and statistical features from network encrypted traffic and save the source traffic as input to the next stage detector;

[0013] S2, time series feature clustering processing: The time series features extracted in S1 are fed into the symbolic pattern clustering algorithm (Symbolic Pattern Forest, SPF, a time series clustering algorithm with linear time complexity, which uses randomly selected symbol patterns to divide data areas without using distance metrics. The integration size for obtaining good results does not depend on the input sequence size. It is fast, low-cost and highly accurate. For details, please refer to reference [1]) to obtain the traffic clustering result R1;

[0014] S3, statistical feature classification processing: The statistical features extracted in S1 are fed into CS++SVM (a support vector machine that can simultaneously complete combined classification and anomaly detection. It can accurately classify multiple types of known classes in the data and simultaneously identify newly emerged anomaly classes. It can accurately cut known classes while maximizing the unknown space. It can also ignore the structure of the data stream itself to train the model, effectively solving the combined classification / anomaly detection problem. For details, please refer to reference [2]) to obtain the traffic classification result R2;

[0015] S4, source traffic anomaly detection: The source traffic saved in S1 is fed into the anomaly transformer (a time series anomaly detection algorithm that improves the traditional point-by-point detection and feature extraction method of the continuous traffic model. It finds that the local correlation of anomalies is close but the correlation with the overall sequence is small, while the correlation of normal points with any region of the overall sequence is large. This feature greatly improves the accuracy of the detection model. For details, please refer to reference [3]) to obtain the traffic classification result R3;

[0016] S5, result integration and aggregation: comprehensively consider the traffic classification results of R1, R2, and R3, and use the weighted majority voting method to obtain the final malicious traffic identification result R.

[0017] As an improvement of the present invention, step S1 further includes:

[0018] S11 intercepts the encrypted source traffic at the gateway and performs traffic cleaning, filtering with the five-tuple <source IP, destination IP, source port, destination port, transport layer protocol>, and processes the source traffic into a set of encrypted network flows that can be used as input for the machine learning model;

[0019] S12, perform traffic segmentation on the encrypted stream obtained from traffic cleaning, extract the first third of the data packets of each encrypted stream to represent the original data stream; for each encrypted stream, record the packet lengths in sequence and splice them into a time series feature sequence, calculate the flow duration, the number of forward and reverse data packets, the average length and length standard deviation of the packet header, the maximum arrival time interval of the forward and reverse flows, etc. as statistical feature vectors; and save the representative flow mirror as the original feature set.

[0020] As an improvement of the present invention, in step S2, the time series feature sequence obtained in S1 is sent to the pattern matching clustering algorithm described in [1] to obtain traffic clusters, each of which contains a certain number of encrypted traffic samples, and the clustering results do not have any label information.

[0021] As an improvement of the present invention, in step S3, the statistical feature set obtained in S1 is sent to the pre-trained CS++SVM classifier in [2] to obtain a set of traffic classification results, wherein represents the benign encrypted traffic set obtained by the CS++SVM classifier, represents the malicious encrypted traffic sets of various fine-grained categories obtained by the CS++SVM classifier, such as the DDoS traffic set, the brute force SSH traffic set, the infiltration traffic set, etc., and represents the unknown encrypted traffic set identified by the CS++SVM classifier, that is, the zero-day traffic that does not belong to any known category and represents a potential threat.

[0022] As an improvement of the present invention, in step S4, the original characterization encrypted traffic set saved in S1 is sent to the pre-trained anomaly transformer anomaly detector in [3] to obtain a recognition result set, where represents the benign traffic set obtained by the anomaly detector and represents the abnormal traffic set obtained by the anomaly detector.

[0023] As an improvement of the present invention, step S5 further includes:

[0024] S51 extracts traffic flow judgments in real time from network traffic, or determines the proportion of packet padding flows in the traffic based on experience, where packet padding flows refer to packets in the flow that are padded to mask packet length characteristics for the purpose of protecting user privacy or for the purpose of concealing attack behavior by an adversary; assigns decision weights to the symbolic pattern clustering algorithm, the CS++ SVM classifier, and the anomaly transformer anomaly detector respectively;

[0025] S52, apply the CS++SVM classifier to the traffic clusters obtained in S2, and assign a unified label to each traffic cluster according to the majority voting principle, thereby converting the traffic clusters into a traffic classification set (for example, assuming that the vast majority of samples in are labeled benign by the CS++SVM classifier, then all samples in are temporarily considered to be benign);

[0026] S53, integrate the results obtained in S2, S3, and S4, and first determine whether the traffic is benign or malicious: perform weighted majority voting according to the weights determined in S51, and divide the encrypted traffic set into a benign set and a malicious set

[0027] S54, further divide the malicious traffic set by the integration results: take the intersection of and, that is, when the clustering result and the CS++SVM classification result both believe that a certain malicious traffic belongs to a category, the traffic is determined to belong to the fine-grained malicious category; all traffic in the malicious traffic set that is not identified as the fine-grained malicious category is divided into abnormal traffic together with the unknown traffic set and kept for further analysis.

[0028] Beneficial effects: Compared with the existing technology, the present invention proposes an encrypted traffic network intrusion detection method based on heterogeneous ensemble learning, which only extracts the first one-third of the data packets of each flow as a representation, reducing the processing complexity and being able to alarm before the malicious traffic causes negative effects; it does not rely on the characteristics of non-encrypted traffic and can cope with the growing amount of encrypted traffic in the network; it can detect unknown zero-day threats; it utilizes heterogeneous features and heterogeneous learning machine integration strategies to maximize the difference, and adjusts the integration strategy in real time according to the proportion of data packets filling the flow in the network. This solution combines coarse and fine granularity in the classification task, takes into account the common zero-day attack traffic in the network, and avoids the problem of high false positives of the intrusion detection system to a certain extent; it automatically processes time series data streams, with high speed, low hardware requirements and low processing cost, and is convenient for actual deployment; the integrated model used can simultaneously cope with combined classification and anomaly detection, and ignores the structure of the data itself; it adopts an integrated strategy, and the three models vote instead of a single one, which has good stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the method steps of the present invention;

[0030] Figure 2 It is a working framework diagram of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0032] Implementation example: Figure 1 As shown, a network intrusion detection method for identifying encrypted malicious traffic based on ensemble learning includes the following steps:

[0033] Extract traffic features:

[0034] In the era of the Internet of Things, the network's data storage capacity has been enhanced, data generation and collection technologies have made significant progress, and a large amount of time series data has become available. Although we are able to monitor a wide range of physical objects and environments in fine spatial and temporal details, we are also faced with the problem of processing large and complex evolving data streams.

[0035] The feature extraction step requires considerable consideration and meticulous work. How much traffic length should be retained to ensure sufficient analysis without excessive redundancy? What components are needed to extract time series features? Is the consideration comprehensive and unbiased, based on the overall characteristics? These all require meticulous planning and extensive reference to other work. Using the first third of the encrypted stream as a representative representation of the original data stream has been rigorously proven to be sufficient, and the types of length-based statistical features we use are widely accepted.

[0036] We first obtain network encrypted traffic from server clusters, personal computers (PCs), and other network detection points and send it to the Intrusion Detection System (IDS) for preprocessing. We extract timing features and statistical features and save the source traffic as input for the next stage detector.

[0037] The encrypted traffic preprocessing and feature extraction in step (1) includes the following sub-steps:

[0038] (1.1) Intercept encrypted source traffic at the gateway and perform traffic cleaning, filtering with the five-tuple <source IP, destination IP, source port, destination port, transport layer protocol>, and process the source traffic into a set of encrypted network flows that can be used as input for machine learning models

[0039] (1.2) Perform traffic segmentation on the encrypted stream obtained from traffic cleaning, extract the first n packets of each encrypted stream (n is a hyperparameter, determined in the specific application) to represent the original data stream; for each encrypted stream, record the packet lengths in sequence and splice them into a time series feature sequence, calculate the flow duration, the number of forward and reverse packets, the average length and length standard deviation of the packet header, the maximum arrival time interval of the forward and reverse flows, etc. as statistical feature vectors; and save the representative flow image as the original feature set.

[0040] Time series feature clustering processing:

[0041] Automatically interpreting the time-series data stream obtained in the previous step is a huge challenge. The large amount of data makes manual inspection impractical. The rapidly changing content of the data stream requires data mining methods with low time complexity and reduced hardware requirements, thereby balancing processor performance and processing costs and promoting the further promotion of the solution.

[0042] In 2019, a symbolic pattern clustering algorithm (Symbolic Pattern Forest, SPF) with linear time complexity was proposed [1]. It is an effective solution to the above challenges. It uses randomly selected symbolic patterns to divide data areas without using distance metrics. The integration size for obtaining good results does not depend on the size of the input sequence. It is fast, low-cost and highly accurate.

[0043] Step (2) uses the time series features extracted in (1) and feeds them into the symbol pattern clustering algorithm described in [1] to obtain the traffic clustering result R1.

[0044] The specific symbolic expression is as follows: The time series feature sequence obtained in (1) is fed into the symbolic pattern clustering algorithm described in [1] to obtain the traffic clusters {C1, C2, ..., C m}, each traffic cluster contains a certain amount of encrypted traffic samples, and the clustering results do not have any label information.

[0045] Statistical feature classification processing:

[0046] Because new and unknown events may appear at any time, we must be cautious about the actual situation and not just consider the known types and proportions artificially controlled by laboratory training sets. Classifiers trained using known patterns, such as traditional binary and multi-class support vector machines, cannot recognize or will incorrectly label these events.

[0047] In 2020, a new SVM algorithm: CS++SVM was proposed [2]. It can not only accurately classify multiple types of known classes in the data, but also simultaneously identify newly emerging anomaly classes, accurately cut known classes while maximizing the unknown space, and can also ignore the structure of the data stream itself to train the model, effectively solving the combined classification / anomaly detection problem.

[0048] Step (3) uses the statistical features extracted in (1) and feeds them into the CS++ support vector machine classifier described in [2] to obtain the traffic classification result R2;

[0049] The specific symbolic expression is as follows:

[0050] The statistical feature set obtained in (1) is fed into the pre-trained CS++SVM classifier in [2] to obtain the traffic classification result set: {B, M1, M2......, M n ,U}, where B represents the benign encrypted traffic set obtained by CS++SVM classifier, {M1,M2......,M n} represents the malicious encrypted traffic sets of various fine-grained categories obtained by the CS++SVM classifier, such as the DDoS traffic set, brute-force SSH traffic set, and infiltration traffic set. U represents the unknown encrypted traffic set identified by the CS++SVM classifier, that is, zero-day traffic that does not belong to any known category and represents a potential threat.

[0051] Source traffic anomaly detection:

[0052] The proportion of abnormal data is usually too small, the time distribution is unpredictable and too uneven, making it difficult to be effectively detected, and the sensitivity of the detection method is required to be very high.

[0053] In 2022, in response to the above dilemma, the anomaly transformer anomaly detector was proposed [3]. It abandoned the traditional continuous traffic model point-by-point detection and feature extraction method, and discovered that the local correlation of anomalies is close but the correlation with the overall sequence is very small, while the correlation between normal points and any area of ​​the whole is very large. This feature is used to greatly improve the accuracy of the detection model.

[0054] Step (4) uses the source traffic saved in (1) and feeds it into the anomaly transformer detector described in [3] to obtain the traffic classification result R3;

[0055] The symbolic representation is as follows:

[0056] The original encrypted traffic set stored in (1) is fed into the pre-trained anomaly transformer anomaly detector in [3] to obtain the recognition result set {B2, A}, where B2 represents the benign traffic set obtained by the anomaly detector and A represents the abnormal traffic set obtained by the anomaly detector.

[0057] Result integration aggregation:

[0058] Taking into account the traffic classification results of R1, R2, and R3, we choose to use the weighted majority voting method to obtain the final malicious traffic identification result R, where the weight ratio is manually assigned based on prior experience. If the specific effect is not clear, you can first average it and then rely on expert experience to dynamically adjust the classification results in the future, thereby enhancing the robustness of the model.

[0059] After obtaining the voting results, we first perform a coarse-grained classification of benign and malignant traffic, then perform fine-grained identification of malicious traffic, and retain unknown traffic in the malicious traffic concentration for further analysis, rather than the traditional direct labeling method, which enhances the model's fault tolerance.

[0060] Step (5) specifically includes the following sub-steps:

[0061] (5.1) Extract traffic flow judgments in real time from network traffic, or determine the proportion of packet padding flows in the traffic based on experience, α, where packet padding flows refer to packets in the flow that are padded to mask packet length characteristics for the purpose of protecting user privacy or for the purpose of concealing attack behavior by an adversary; assign decision weights of (1-α) / (3-α), 1 / (3-α), and 1 / (3-α) to the symbolic pattern clustering algorithm, CS++ SVM classifier, and anomaly transformer anomaly detector, respectively;

[0062] (5.2) Apply the CS++SVM classifier to the traffic clusters obtained in (2) and assign a unified label to each traffic cluster according to the majority voting principle, thus classifying the traffic clusters {C1, C2, ..., C m}Convert to traffic classification set {B1 ′ ,M1 ′ ,M2 ′ ......,M n ′ ,U1 ′ (For example, assuming that the vast majority of samples in C1 are labeled benign by the CS++SVM classifier, then all samples in C1 are temporarily considered to be benign).

[0063] (5.3) Integrate the results obtained in (2)(3)(4) and first determine whether the traffic is benign or malicious: perform weighted majority voting according to the weights determined in (5.1) to divide the encrypted traffic set into benign set B and malicious set M. Then further divide the malicious traffic set M based on the integrated results: i With M i ′ Take the intersection (i=1,2,......,n), that is, when the clustering result and the CS++SVM classification result both believe that a malicious traffic belongs to category M i When the traffic is considered to belong to the fine-grained malicious category M i All traffic that is not identified as fine-grained malicious categories in the malicious traffic set and the unknown traffic sets U and U ′ They are classified as abnormal traffic and kept for further expert analysis.

[0064] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation scheme, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A method for detecting intrusion in encrypted traffic network based on ensemble learning, characterized in that: The steps include: S1, extract traffic features: extract timing features and statistical features from network traffic and save the source traffic as input to the next stage detector; S2, time series feature clustering processing: use the time series features extracted in S1 and send them into the symbol pattern clustering algorithm to obtain the traffic clustering result R1; S3, statistical feature classification processing: Utilize the statistical features extracted in S1 and feed them into CS++SVM to obtain the traffic classification result R2; S4, source traffic anomaly detection: Use the source traffic saved in S1 and send it to the anomaly transformer to obtain the traffic classification result R3; S5, result integration and aggregation: Taking into account the traffic classification results of R1, R2, and R3, the final malicious traffic identification result R is obtained using the weighted majority voting method.

2. The encrypted traffic network intrusion detection method based on ensemble learning according to claim 1, characterized in that: The step S1 further comprises: S11 intercepts the encrypted source traffic at the gateway and performs traffic cleaning, filtering it with the five-tuple <source IP, destination IP, source port, destination port, transport layer protocol>, and processing the source traffic into an encrypted traffic set that can be used as input for the machine learning model; S12, perform traffic segmentation on the encrypted stream obtained from traffic cleaning, extract the first third of the data packets of each encrypted stream to represent the original data stream; for each encrypted stream, record the packet lengths in sequence and splice them into a time series feature sequence, calculate the flow duration, the number of forward and reverse data packets, the average length and length standard deviation of the packet header, and the maximum arrival time interval of the forward and reverse flows as statistical feature vectors; and save the representative flow mirror as the source traffic.

3. The encrypted traffic network intrusion detection method based on ensemble learning according to claim 1, characterized in that: In step S2, the time series feature sequence obtained in step S1 is fed into the symbol pattern clustering algorithm to obtain the traffic clusters {C1, C2, ..., C m }, each traffic cluster contains a certain amount of encrypted traffic samples, and the clustering results do not have any label information.

4. The encrypted traffic network intrusion detection method based on ensemble learning according to claim 1, characterized in that: In step S3, the statistical feature set obtained in S1 is fed into the pre-trained CS++SVM classifier to obtain the traffic classification set {B, M1, M2..., M n ,U}, where B represents the benign encrypted traffic set obtained by CS++SVM classifier, {M1,M2......,M n } represents the malicious encrypted traffic set of each fine-grained category obtained by the CS++SVM classifier, including the DDoS traffic set, the brute-force SSH traffic set, and the infiltration traffic set. U represents the unknown encrypted traffic set identified by the CS++SVM classifier, that is, the zero-day traffic that does not belong to any known category and represents a potential threat.

5. The encrypted traffic network intrusion detection method based on ensemble learning according to claim 1, characterized in that: In step S4, the source traffic stored in S1 is fed into the pre-trained anomaly transformer anomaly detector to obtain a recognition result set {B2, A}, where B2 represents the benign traffic set obtained by the anomaly detector, and A represents the abnormal traffic set obtained by the anomaly detector.

6. The encrypted traffic network intrusion detection method based on ensemble learning according to claim 4, characterized in that: The step S5 further comprises: S51 extracts traffic judgment from network traffic in real time, or determines the proportion α of packet padding flow in the traffic based on experience, where packet padding flow refers to the padding of packets in the flow to cover up packet length characteristics for the purpose of protecting user privacy or for the purpose of hiding attack behavior by the adversary; assigns decision weights (1-α) / (3-α), 1 / (3-α), and 1 / (3-α) to the symbolic pattern clustering algorithm, CS++ SVM classifier, and anomaly transformer anomaly detector, respectively; In S52, the CS++SVM classifier is applied to the traffic clusters obtained in S2, and a unified label is assigned to each traffic cluster according to the majority voting principle, thereby classifying the traffic clusters {C1, C2, ..., C m } is converted into a traffic classification set {B1′,M1′,M2′......,M n ′,U1′}; S53, integrating the results obtained in S2, S3, and S4, first determining whether the traffic is benign or malicious: performing weighted majority voting according to the weights determined in S51, and dividing the encrypted traffic set into a benign set B and a malicious traffic set M; S54, further divide the malicious traffic set M by integrating the results: i With M i ′ Take the intersection, i=1,2,......,n, that is, when the clustering result and the CS++SVM classification result both believe that a malicious traffic belongs to category M i When the traffic is considered to belong to the fine-grained malicious category M i All traffic in the malicious traffic set that is not identified as fine-grained malicious categories and the unknown encrypted traffic sets U and U′ are classified as abnormal traffic and retained as suspected zero-day attack traffic for further security incident analysis and academic research.

Images