Encrypted malicious traffic detection and attack identification method based on deep learning

Abstract

The present invention discloses a method for detecting and identifying encrypted malicious traffic based on deep learning, which includes the following steps: S1: an Internet of Things threat perception terminal collects network traffic generated by Internet of Things devices in an industrial Internet of Things in real time; S2: the network traffic collected by the Internet of Things threat perception terminal is sent to an Internet of Things threat perception analysis center, the network traffic is identified through a malicious encrypted traffic detection model and a malicious encrypted traffic attack identification model, and the identification result is sent to an Internet of Things security management platform; S3: the Internet of Things security management platform forms a network attack surface through the identification result and network traffic information, and visually displays the impact of the attack source on the network. The present invention has high detection efficiency and high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of malicious traffic detection, and in particular to an encrypted malicious traffic detection and attack identification method based on deep learning. Background Art

[0002] With the development of information technology, IoT devices are widely popularized, but due to cost and practicality, the network security issues of IoT devices are often ignored by manufacturers. Hackers are good at using security vulnerabilities to infect a large number of IoT devices and then form botnets, and use botnets as a springboard to launch distributed denial of service (DDoS) attacks. Therefore, security issues in the field of IoT need to be solved urgently, and network attack traffic detection algorithms, as a branch of network security, also need to be updated urgently. In addition, with the trend of network traffic encryption, the widespread application of encryption technology and protocols, the network is facing a massive amount of encrypted traffic, and the encrypted traffic presents complex and diversified characteristics, which increases the difficulty of traffic identification in the intrusion detection system in the Internet of Things. Since the characteristics of encrypted traffic have changed, traditional traffic detection methods are difficult to reproduce in an encrypted environment. Methods such as deep packet inspection or pattern matching are helpless against encrypted traffic. Therefore, how to effectively identify malicious traffic on encrypted traffic without decryption has become an important challenge in the field of network security.

[0003] Analyzing the current research status of malicious encrypted traffic classification and combining it with the application scenarios of the Internet of Things, the current encrypted malicious traffic detection technology still has the following problems to be improved: 1) Due to the characteristics of large attack volume and diverse forms of network attacks such as botnets, denial of service attacks, and malicious encrypted traffic, a deeper study of the data features of malicious encrypted traffic data is needed; 2) At the feature extraction level, in the application of the Internet of Things, the current feature method is difficult to cope with massive Internet of Things data, and there is a lag in the extraction of features. It is necessary to consider accurately extracting data features while ensuring the stability of feature extraction; 3) At the detection model classification level, the detection model application scenario is single, the generalization power is weak, and the stability of the model detection accuracy is poor. Therefore, a malicious encrypted traffic detection method with high detection efficiency, high accuracy and strong robustness is needed. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides an encrypted malicious traffic detection and attack identification method based on deep learning with high detection efficiency and high accuracy.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] The encrypted malicious traffic detection and attack identification method based on deep learning includes the following steps:

[0007] S1: IoT threat perception terminal collects network traffic generated by IoT devices in industrial IoT in real time;

[0008] S2: Send the network traffic collected by the IoT threat perception terminal to the IoT threat perception analysis center, identify the network traffic through the malicious encrypted traffic detection model and the malicious encrypted traffic attack identification model, and send the identification results to the IoT security management platform;

[0009] Among them, the malicious encrypted traffic detection model first trains a single classifier on the encrypted normal data set, and then puts the encrypted malicious data set into a single classifier for identification, identifies abnormal encrypted traffic, and re-judges it as encrypted malicious data. Then, the identified encrypted malicious data set and the encrypted normal data set are combined into a new data set for binary classification training;

[0010] The malicious encrypted traffic attack identification model selects a normal traffic data set and a malicious traffic data set, filters out unencrypted network traffic in the data set, extracts the time series features of the data packets of the encrypted network traffic, and establishes a malicious encrypted traffic attack fingerprint library; divides the data set into a test data set and a training data set, inputs the training data set into the GRU-DNN neural network model, and then matches and optimizes the model training results with the known identification results to adjust the model parameters;

[0011] S3: IoT security management platform, which forms a network attack surface through identification results and network traffic information, and visualizes the impact of attack sources on the network.

[0012] Furthermore, the single classifier is a One-Class-SVM algorithm.

[0013] Furthermore, the binary classifier is an Xgboost algorithm.

[0014] Further, in step S2, if the detection results of the malicious encrypted traffic detection model and the malicious encrypted traffic attack identification model are consistent, it can be determined that the network traffic is malicious encrypted traffic;

[0015] If the two results are inconsistent, where the malicious encrypted traffic detection model result is malicious traffic, and the malicious encrypted traffic attack identification model result is normal traffic, then the network traffic is judged to be malicious traffic of unknown attack type, and the malicious traffic is stored in the malicious traffic sample library; if the malicious encrypted traffic attack identification model result is malicious encrypted traffic, and the malicious encrypted traffic detection model result is normal, it is judged to be malicious encrypted traffic.

[0016] The beneficial effects of the present invention are as follows: the present invention designs a malicious encrypted traffic detection model and a malicious encrypted traffic attack identification model, realizes the detection of malicious encrypted traffic, and further detects the type of malicious attack. The test results show that the malicious encrypted traffic detection method designed by the present invention can achieve an accuracy of more than 98% and a recall rate of more than 99.8%, and has better robustness and applicability on the basis of ensuring the accuracy of model classification. Finally, the malicious encrypted traffic detection system designed by the present invention can not only shorten the detection time but also obtain more accurate detection results, and predict unknown threats to realize the network security situation awareness of the Internet of Things. The present invention also designs an Internet of Things threat perception terminal, an Internet of Things threat perception analysis center and an Internet of Things security control platform, realizes real-time and rapid detection of network traffic of Internet of Things devices and visual network threat perception, and establishes a malicious traffic sample database of self-incremental learning to better reduce the harm caused by unknown malicious encrypted traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the malicious encrypted network traffic detection method of the present invention;

[0018] Figure 2 The malicious encrypted traffic detection model of the present invention;

[0019] Figure 3 It is a malicious encrypted traffic attack identification model of the present invention;

[0020] Figure 4 This is a structural diagram of the GRU+DNN neural network model of the present invention;

[0021] Figure 5 This is the GRU layer structure diagram of the present invention;

[0022] Figure 6 This is a system architecture diagram of the present invention;

[0023] Figure 7 It is a network attack analysis visualization interface of the present invention; DETAILED DESCRIPTION

[0024] In order to better understand the present invention, Figure 1-7 The embodiments of the present invention will be explained in detail.

[0025] The present invention provides a system and method for detecting malicious traffic and identifying attacks based on deep learning. Specifically, the system architecture of detecting malicious traffic and identifying attacks based on deep learning is as follows: Figure 6As shown, it includes an IoT threat perception terminal, an IoT threat analysis center, and an IoT security management and control platform. The IoT threat perception terminal collects the network traffic generated by IoT devices in the industrial IoT in real time. An IoT threat perception analysis center consisting of a malicious encrypted network traffic detection model and a malicious network traffic attack identification model is established. The malicious encrypted network traffic is identified without decrypting the network traffic by using the SSL / TLS network protocol characteristics, the time series characteristics of the data packet, and the DGA domain name characteristics. Then, a malicious network attack database is established based on the malicious attack network traffic data characteristics. Network attacks include password attacks, denial of service attacks (DoS), botnets, etc., so as to effectively detect the network threat behaviors of IoT devices in the industrial IoT and establish a real-time, automated, and visual malicious encrypted traffic detection system.

[0026] The IoT Threat Analysis Center conducts in-depth analysis of the network traffic generated by IoT devices in the industrial IoT. By parsing network protocols such as TCP, SSL, and UDP, it deploys multiple detection model working units based on machine learning, deep learning, etc., thereby identifying malicious encrypted network traffic. It further identifies encrypted malicious network traffic formed by export password attacks, SQL injections, denial of service attacks, distributed denial of service attacks, and botnet attacks, and sends the identified network traffic and detection model results to the IoT security control platform to demonstrate the impact of the attack source on the network. It builds a malicious sample database suitable for self-incremental learning, which is used to continuously update the malicious encrypted network traffic detection model and attack identification model, enhance the robustness and applicability of the malicious encrypted traffic detection model, and thus form a real-time, automated, and visual malicious encrypted traffic detection system.

[0027] The detection and attack identification method of the present invention is as follows:

[0028] In the data collection and monitoring control system of the Internet of Things, by deploying the Internet of Things threat perception terminal, the network traffic generated by the Internet of Things devices can be collected in real time;

[0029] The network traffic collected by the IoT threat perception terminal is sent to the IoT threat perception analysis center, and a multi-threaded working mode is adopted to start multiple working units, including a malicious encrypted traffic detection model and a malicious encrypted traffic attack identification model, to identify the network traffic and send the identification results to the IoT security management platform.

[0030] Malicious encrypted traffic detection, after the malicious encrypted traffic attack identification model receives the network traffic, it pre-processes the network traffic, and then extracts features of the network traffic data according to the TLS protocol characteristics, server certificate characteristics, DNS domain name characteristics, and data packet time series characteristics. Finally, it is tested through the detection model and the detection results are sent to the IoT Threat Perception Analysis Center.

[0031] The detection results of each detection model are sent to the IoT Threat Perception Analysis Center. The center makes judgments based on the classification results of multiple classifiers, identifies malicious encrypted network traffic information (source IP address, destination IP address, destination port, and source port), and sends the judgment results to the IoT Security Management Platform.

[0032] The IoT security management and control platform forms a network attack surface through identification results and network traffic information, visually displays the impact of the attack source on the network, and establishes a malicious sample database suitable for self-incremental learning by continuously collecting malicious encrypted network traffic. It is used to update the detection model and enhance the robustness and extensiveness of the model.

[0033] like Figure 1 and Figure 2 As shown in, the encrypted malicious network traffic detection method in this method is to extract effective feature values ​​from the handshake phase information of the TLS protocol, server certificate information, network data packet statistical characteristics and DNS domain name information, combine machine learning and deep learning model frameworks, and use the learning ability and generalization ability of the model to distinguish malicious encrypted traffic and normal encrypted traffic in the network, as well as malicious traffic and normal traffic from the perspective of network data packets, data flows and DNS domain name information, and build a malicious traffic sample database suitable for self-incremental learning to ensure the accuracy of the malicious encrypted traffic detection method and enable the model to have incremental learning capabilities.

[0034] Without decrypting the encrypted network traffic, we conduct an in-depth study on the characteristics of malicious encrypted traffic, extract effective feature values ​​from the handshake phase information, certificate information, HTTP header information, and DNS response information of the TLS protocol, and then combine anomaly detection and binary classification detection methods to effectively distinguish the characteristics of malicious encrypted traffic from normal traffic. Finally, we establish a malicious encrypted traffic detection model and deploy it to the IoT threat perception terminal to perform real-time dynamic differentiation between malicious and benign traffic in the network and detect malicious threats received by IoT devices.

[0035] The malicious encrypted traffic detection model distinguishes malicious encrypted traffic from normal encrypted traffic and establishes a binary malicious encrypted traffic detection model. In this model, the data features are first determined. Based on the in-depth analysis of the malicious encrypted traffic features, combined with TLS handshake features, certificate features, and domain name features, 12 representative and real-time data features are determined, as shown in Table 1:

[0036]

[0037]

[0038] Table 1

[0039] Secondly, after determining the data characteristics, select a suitable model for training and parameter tuning. The model selects the binary classification xgboost model for training parameter tuning:

[0040] The first step is to select a data set, which consists of an encrypted malicious data set and an encrypted normal data set. The encrypted malicious data set is the network traffic generated by malware attacking the IoT device in a sandbox environment. The encrypted normal data set is the network traffic generated by the IoT device without being attacked by malware, and all of them are composed of pcap files.

[0041] The second step is data feature extraction. The data message information in the pcap file is filtered to extract the handshake phase information of the TLS protocol. Combined with the protocol features such as the TLS encryption suite, certificate and domain name information in the handshake phase, data features of the network traffic pcap file are extracted. Finally, the extracted data features are saved as a csv file.

[0042] The third step is to develop a malicious encrypted traffic detection model. Since the selected data set clearly states that the normal data set is the encrypted network traffic generated by IoT devices that are not attacked by malware, we first train a single classifier on the encrypted normal data set, and then put the encrypted malicious data set into a single class classifier to identify abnormal encrypted traffic and re-judge it as encrypted malicious data. The purpose of this is not only to obtain more representative malicious encrypted traffic samples, but also to enhance the robustness of the malicious encrypted traffic detection model. Finally, the identified encrypted malicious data set and the encrypted normal data set are combined into a new data set for binary classification training. Among them, the single classifier is the One-Class-SVM algorithm, and the binary classifier is the Xgboost algorithm, so as to improve the accuracy of detecting malicious encrypted traffic, and on the basis of maintaining classification accuracy, it has better robustness and applicability.

[0043] In the fourth step, according to the selected data feature files, the data set is divided into a training data set and a test data set, of which the training data set accounts for 80% and the test data set accounts for 20%. Using the model described in the third step, the training data set is input into the model, and then the model training results are matched and optimized with the known recognition results, the model parameters are adjusted, and the model detection accuracy is improved.

[0044] The following is a detailed introduction to the malicious network traffic attack identification model and attack identification method based on deep learning.

[0045] The present invention analyzes malicious network traffic data of known malicious attack types such as password attacks, SQL injections, denial of service attacks, distributed denial of service attacks, and botnet attacks in HTTP and HTTPS environments, and uses multiple network data packet statistical features such as network data packet duration, number of messages, number of bytes, and message length to build an Internet of Things network threat attack sample fingerprint database. Combined with the characteristics of the deep learning neural network model that automatically learns data features, the present invention distinguishes the differences in the characteristics of normal traffic and malicious traffic, and effectively identifies the differences between malicious encrypted traffic and non-encrypted malicious traffic, thereby establishing a detection model with high accuracy and robustness, and deploying the model to the Internet of Things network threat analysis center to dynamically detect encrypted malicious attacks on Internet of Things devices in real time.

[0046] The malicious encrypted traffic attack identification method of the present invention learns the data characteristics of different attack types of malicious encrypted traffic based on a deep learning method, including botnets, denial of service attacks, SQL injection attacks, and malicious encrypted traffic attacks, thereby achieving in-depth analysis of malicious encrypted traffic characteristics and improving the accuracy of detecting malicious encrypted traffic.

[0047] Malicious encrypted traffic attack identification method Figure 3 As shown:

[0048] The first step is to select a data set, which consists of a normal traffic data set and a malicious traffic data set. The malicious traffic data set is the network traffic generated by a known malware family attacking IoT devices in a sandbox environment, and the normal traffic is the network traffic generated by IoT devices that are not attacked by malware.

[0049] The second step is data feature selection. First, filter out the unencrypted network traffic in the data set, leaving only the encrypted network traffic. Secondly, based on the TCP protocol characteristics, extract the time series features of the encrypted network traffic data packets, extract multiple network traffic statistical features such as duration, number of messages, number of bytes, and message length, and establish a malicious encrypted traffic attack fingerprint library in the form of an n*80 matrix. Among them, n is the number of network data packets, and 80 is the network data packet data feature. This feature extraction method allows the detection model to fully analyze the data features of each data packet, and there is no risk of delayed detection.

[0050] The data characteristics of the data packet are shown in Table 2:

[0051]

[0052]

[0053]

[0054] Table 2

[0055] The third step is to build a malicious encrypted traffic attack recognition model based on deep learning. Since the data features are extracted based on time series, a time series classification model based on deep learning is proposed. By building a multi-layer learning model, more useful features can be learned. Therefore, we chose a deep learning model based on gated recurrent neural network (GRU) and deep neural network (DNN), which can effectively identify the type of network traffic attack. Figure 4 and Figure 5 The model consists of two GRU layers, four dropout layers, two Dense layers, a fully connected layer, and an activation function. Among them, the GRU neural network layer only has two neural network structures, the update gate and the reset gate, which has fewer parameters. Therefore, using the GRU neural network layer can save time costs and improve the efficiency of model training when the hardware computing power is limited.

[0056] In the fourth step, the data set is divided into a test data set and a training data set, where the training data set accounts for 80% and the test data set accounts for 20%. Using the model described in the third step, the training data set is input into the model, and then the model training results are matched and optimized with the known recognition results, the model parameters are adjusted, and the model detection accuracy is improved.

[0057] The present invention designs an IoT threat perception terminal, an IoT threat perception center, and an IoT security control platform. Among them, the IoT device network threat perception center collects the network traffic generated by the IoT devices in real time, inputs the real-time collected network traffic to the IoT threat perception analysis center, and the IoT threat perception center automatically identifies the network traffic, sends the detected network traffic and the detection results to the IoT security control platform, forms a network attack surface through the detection results and network traffic information, and visualizes the impact of the attack source on the network.

[0058] The present invention designs two detection models in the Internet of Things Threat Perception Analysis Center, namely, a malicious encryption detection model and a malicious encrypted traffic attack identification model, wherein the malicious encrypted traffic detection model is an anomaly detector, which can distinguish between normal encrypted traffic and malicious encrypted traffic. The malicious encryption attack identification model can further identify the attack type of malicious encrypted traffic. The identification result of the malicious encrypted traffic attack identification model on the network traffic is compared with the malicious encrypted traffic detection identification result, and the detection results of the two models are combined to determine whether the network traffic is malicious encrypted traffic. The judgment process is as follows: if the result of the malicious encrypted traffic detection is consistent with the result of the attack model, the network traffic can be judged to be malicious encrypted traffic; if the two results are inconsistent, wherein the malicious encrypted traffic detection result is malicious traffic, and the attack model result is normal traffic, the network traffic is judged to be an unknown attack type malicious traffic, and the malicious traffic is stored in the malicious traffic sample library; if the attack model result is malicious encrypted traffic, and the malicious encrypted traffic detection result is normal, the attack model result shall prevail and it shall be judged to be malicious encrypted traffic.

Claims

1. Encrypted malicious traffic detection and attack identification method based on deep learning, It is characterized in that It includes the following steps: S1: IoT threat perception terminal collects network traffic generated by IoT devices in industrial IoT in real time; S2: Send the network traffic collected by the IoT threat perception terminal to the IoT threat perception analysis center, identify the network traffic through the malicious encrypted traffic detection model and the malicious encrypted traffic attack identification model, and send the identification results to the IoT security management platform; Among them, the malicious encrypted traffic detection model first trains a single classifier on the encrypted normal data set, and then puts the encrypted malicious data set into a single classifier for identification, identifies abnormal encrypted traffic, and re-judges it as encrypted malicious data. Then, the identified encrypted malicious data set and the encrypted normal data set are combined into a new data set for binary classification training; The malicious encrypted traffic attack identification model selects a normal traffic data set and a malicious traffic data set, filters out unencrypted network traffic in the data set, extracts the time series features of the data packets of the encrypted network traffic, and establishes a malicious encrypted traffic attack fingerprint library; divides the data set into a test data set and a training data set, inputs the training data set into the GRU-DNN neural network model, and then matches and optimizes the model training results with the known identification results to adjust the model parameters; S3: IoT security management platform, which forms a network attack surface through identification results and network traffic information, and visualizes the impact of attack sources on the network.

2. The method for detecting and identifying encrypted malicious traffic based on deep learning as claimed in claim 1, It is characterized in that The single classifier is a One-Class-SVM algorithm.

3. The method for detecting and identifying encrypted malicious traffic based on deep learning as claimed in claim 1, It is characterized in that The binary classifier is the Xgboost algorithm.

4. The method for detecting and identifying encrypted malicious traffic based on deep learning as claimed in claim 1, It is characterized in that In step S2, if the detection results of the malicious encrypted traffic detection model and the malicious encrypted traffic attack identification model are consistent, it can be determined that the network traffic is malicious encrypted traffic; If the two results are inconsistent, where the malicious encrypted traffic detection model result is malicious traffic, and the malicious encrypted traffic attack identification model result is normal traffic, then the network traffic is judged to be malicious traffic of unknown attack type, and the malicious traffic is stored in the malicious traffic sample library; If the malicious encrypted traffic attack identification model result is malicious encrypted traffic, and the malicious encrypted traffic detection model result is normal, it is judged to be malicious encrypted traffic.

Images