Forged malicious encrypted traffic detection method based on refined statistical feature analysis

Abstract

The present invention relates to a method for detecting forged malicious encrypted traffic based on refined statistical feature analysis. The present invention extracts statistical features in units of flow, performs traffic purification on the extracted statistical features, performs restoration processing on the extracted features, performs refinement processing on the restored features, and performs deep training on the refined features. Malicious encrypted traffic is classified using a deep learning model, and the classification results of malicious encrypted traffic are output. The present invention can reduce the interference of forged samples through restoration processing, and improve the detection capability of forged samples without affecting the detection accuracy of non-forged samples. In addition, the restored features are refined, which increases the overall characteristics of the attack session and effectively improves the detection accuracy of malicious encrypted traffic.

Description

Technical Field

[0001] The invention belongs to the technical field of network security and relates to a method for detecting forged malicious encrypted traffic based on refined statistical feature analysis. Background Art

[0002] Malicious encrypted traffic detection is an important technology in today's network security field. Malicious encrypted traffic encrypts the data packet payload, making the data packet payload character flow nearly randomly distributed. It is very difficult to analyze the encrypted traffic directly from the data packet payload, which increases the difficulty of malicious encrypted traffic detection. With the popularization of encryption technology, there is more and more encrypted traffic in the network, so it is of great significance to identify malicious encrypted traffic.

[0003] Traditional malicious encrypted traffic detection methods are generally divided into four categories:

[0004] One is to distinguish through blacklists, such as malicious websites or port numbers of data packets. Encryption software is classified by well-known websites or port numbers. However, if malware uses new websites or non-standard port numbers, such as dynamic ports or using common ports that are not used by malware, this restriction can be bypassed.

[0005] The second is the traditional traffic classification method based on Deep Package Inspections (DPI). In theory, DPI can obtain all the information in the data packet, and can also obtain encrypted information data. However, the encryption algorithm is very strong nowadays, and it is difficult to crack it. If you don’t know the encryption algorithm key, you can’t understand the specific content of the packet transmission, so the general practice is to maintain the digital signature, decrypt the data packet, and perform detection. This makes this method very inefficient when facing encrypted traffic, and there are a lot of situations that cannot be detected.

[0006] The third is to apply machine learning to malicious encrypted traffic classification. Machine learning classification uses network traffic statistical features. Classification is based on data packets or stream bytes and statistical features. Such as random forest (RF), support vector machine (SV), K nearest neighbor (KNN). However, classification methods based on machine learning rely on features or patterns extracted by experts. This process is error-prone and costly.

[0007] Fourth, deep learning is applied to the classification of malicious encrypted traffic. In recent years, deep learning has achieved great success in solving complex problems such as image classification, speech recognition, and natural language processing, such as deep neural networks (DNN), convolutional neural networks (CNN), and recurrent neural networks (RNN). Deep neural networks are also applied to the classification of encrypted traffic. They do not rely entirely on port numbers, but on the combination of certain features. For example, deep learning can learn the nonlinear relationship between the original input and the corresponding output.

[0008] Chinese Patent No. 202111363318.2 discloses a "method, device and storage medium for detecting encrypted traffic". The method includes: obtaining the IP address of the encrypted traffic to be detected; determining whether the IP address is an IP address in a preset malicious encrypted traffic information table; obtaining the issuing authority domain name in the public key certificate corresponding to the encrypted traffic to be detected; and determining whether the issuing authority domain name is a domain name in the preset malicious encrypted traffic information table.

[0009] Chinese patent No. 202010058395.6 discloses a "malicious encrypted traffic detection method and system based on behavioral analysis". The method includes: after obtaining network traffic data, the network traffic data will be analyzed for features to obtain a session-based feature sequence; the feature sequence will then be input into a detection model to obtain a detection result score, so as to determine whether the network traffic is malicious encrypted traffic based on the detection result score.

[0010] Chinese patent No. 202011035454.4 discloses a "https malicious encrypted traffic detection method and system based on machine learning". The method includes: inputting the four-tuple feature vector of traffic data (source IP, destination IP, destination port, protocol type) into the trained random forest model, detecting the malicious probability of the traffic data, and realizing rapid online detection of malicious traffic.

[0011] Chinese Patent No. 202110944248.3 discloses a "deep learning-based malicious encrypted traffic detection method and device". The method includes: collecting and parsing the TLS data traffic to be detected, obtaining the ClientHello data packet, ServerHello data packet (the data packet is not encrypted) in the TLS handshake data traffic, and the Subject field and Issuer field in the certificate. Extract several feature sequences of the ClientHello data packet and the ServerHello data packet, and input them into the convolutional neural network CNN for feature extraction. Then one-hot encode the Subject field and Issuer field in the certificate to obtain the corresponding feature sequence, and input them into the LSTM model for feature extraction to obtain the corresponding feature sequence. Combine the obtained feature sequences and input them into the classifier for classification, and output the detection results.

[0012] Chinese Patent No. 202011502419.9 discloses a "malicious encrypted traffic detection method, system and device based on traffic behavior". The method includes: extracting statistical features of data packets (bidirectional data packet length information, bidirectional application layer data packet length information) and part of the unencrypted information of the SSL / TLS protocol field, using deep neural network to perform feature processing to obtain implicit features, and then perform encrypted traffic detection, and adjust the detection results through fingerprint information (client fingerprint information, server fingerprint information. Client fingerprint information: the length of the first data packet sent by the client, the length of the encryption suite provided by the client. Server fingerprint information: the name of the server certificate owner, the encryption algorithm selected by the server.).

[0013] Chinese Patent No. 202010764871.6 discloses a "malicious encrypted traffic detection method and device". The method includes: when obtaining encrypted traffic, obtaining characteristic data corresponding to the encrypted traffic (data when establishing an encrypted communication channel and statistical data when transmitting encrypted data. The data when establishing an encrypted communication channel includes the encryption protocol version, acceptable ciphers, extension lists, elliptic curve ciphers, and elliptic curve cipher formats. The statistical data when transmitting encrypted data includes the number of connections, the total packet size, the mean duration, and the standard deviation of the duration), and the characteristic data does not contain the payload in the encrypted traffic; judging whether there is malicious encrypted traffic in the encrypted traffic based on the characteristic data; and performing corresponding processing on the encrypted traffic based on the judgment result.

[0014] At present, the classification of malicious encrypted traffic basically uses unencrypted data flow information, and does not take into account the possibility of forged samples. The current samples can be forged to deceive the existing detection methods. Attackers can deliberately generate forged samples based on the detection method, which greatly reduces the accuracy of malicious encrypted traffic detection. Although their methods may have a high detection accuracy for unforged malicious encrypted traffic, the accuracy will drop significantly if forged samples are detected. For example, the blacklist mechanism cannot completely cover malicious features and there is a possibility of being bypassed. The existing method of extracting statistical features detects based on the differences in certain statistical features between normal encrypted traffic and malicious encrypted traffic. Therefore, attackers can also forge features to make the statistical features of malicious encrypted traffic similar to those of normal encrypted traffic, which greatly reduces the detection accuracy. Summary of the invention

[0015] The purpose of the present invention is to provide a method for detecting forged malicious encrypted traffic based on refined statistical feature analysis, wherein the statistical features include time statistical features such as flow duration and static statistical features such as data packet size; the refined processing method refers to a traffic processing method that takes attack sessions as units and integrates network traffic statistical features, overall attack session features and individual network flow features; forged malicious encrypted traffic refers to the attacker changing certain features of the malicious code traffic without changing the semantics of the malicious code to generate new malicious encrypted traffic, thereby significantly reducing the accuracy of the malicious encrypted traffic detection method based on time and static statistics.

[0016] Since the existing method of extracting statistical features detects malicious encrypted traffic based on certain statistical feature differences between normal encrypted traffic and malicious encrypted traffic, it is almost impossible to detect forged malicious encrypted traffic. Compared with the traditional malicious encrypted traffic detection based on statistical features, the detection method provided by the present invention can reduce the interference of forged samples through restoration processing, and improve the detection ability of forged samples without affecting the detection accuracy of non-forged samples. In addition, the restored features are refined, which increases the overall features of the attack session and effectively improves the detection accuracy of malicious encrypted traffic.

[0017] The present invention specifically comprises the following steps:

[0018] Step 1: Obtain encrypted traffic data, and extract statistical features from the encrypted traffic data in units of TCP streams; a TCP stream is a TCP (Transmission Control Protocol) connection, a process from three-way handshake to four-way handshake. The stream is bidirectional, and the direction of a TCP stream is determined by the initiator of the first data packet of the TCP handshake. The initiator IP is the source IP of the TCP stream, and the receiver IP is the destination IP of the TCP stream. A stream includes all communication data packets from the time the initiator initiates the first TCP handshake packet to the time the initiator sends the last four-way handshake packet. It includes not only data packets with the initiator as the source IP and the receiver as the destination IP, but also data packets with the receiver as the source IP and the initiator as the destination IP.

[0019] The extracted statistical features of the flow include directly extracted statistical features and features calculated based on the directly extracted statistical features:

[0020] The directly extracted statistical features include: source IP address, destination IP address, source port, destination port, protocol number, flow duration, data packet arrival interval in the flow, data packet arrival interval in the forward flow, data packet arrival interval in the reverse flow, number of data packets in the forward flow, number of data packets in the reverse flow, data packet size in the forward flow, and data packet size in the reverse flow;

[0021] The following features are calculated based on the above features: number of packets per second in the flow, number of packets per second in the forward flow, number of packets per second in the reverse flow, size of transmission per second in the flow, size of transmission per second in the forward flow, size of transmission per second in the reverse flow;

[0022] Step 2: purify the statistical features of the flow extracted in step 1, delete duplicate flows, and delete flows with a specified item less than ε (which will affect the subsequent training results). ε is the minimum value set by the user. The minimum value is generally close to 0.

[0023] Step 3: Group the features of the streams purified in step 2 according to the five-tuple (source IP address, destination IP address, source port number, destination port number, protocol number), and each group contains a feature set of streams with the same five-tuple value. Then restore each group (feature set), specifically: for each group, subtract the minimum value of the corresponding item obtained in step 2 from each feature of each stream in the group. Because the number of streams in the group is a finite number, the minimum value only needs to take the minimum value of all streams in the group on the corresponding item to complete the restoration of the above features. For the problem of zeros that may appear in the subtraction, add 1 to all the result values ​​to obtain the result after restoration.

[0024] Step 4: Refine the features restored in step 3 to obtain the overall features of the quintuple, save the average value of all feature items, and for each flow within the group, add the corresponding item average value to its value.

[0025] Step 5: Train the deep learning model and put the new features obtained in step 4 into the deep learning model training. The deep learning model uses the general classification model in the field, such as Resnet (deep residual network), CNN (convolutional neural network), etc.

[0026] Step 6: Use the deep learning model to classify malicious encrypted traffic and output the malicious encrypted traffic classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the overall process of the present invention;

[0028] Figure 2 for Figure 1 Flowchart for restoring the extracted features;

[0029] Figure 3 for Figure 1 Flow chart of refined processing. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following will be combined with the accompanying drawings in the present invention. The technical solutions in the present invention are clearly and completely described. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0031] like Figure 1 As shown, a method for detecting forged malicious encrypted traffic based on refined statistical feature analysis specifically includes the following steps:

[0032] Step 1: Obtain encrypted traffic data and extract statistical features of encrypted traffic:

[0033] 1.1 Obtain traffic data. Use common methods in this field such as port mapping to obtain traffic data. For example, use WireShark to capture the traffic data of normal HTTPS communication and the traffic data of malicious Trojans and ransomware communications.

[0034] 1.2 Preprocess the acquired data. This embodiment mainly considers encrypted traffic. Taking the TLS protocol as an example, the port is filtered during the preprocessing process, and only the data of port 443 is retained.

[0035] 1.3 Extract statistical features from the encrypted traffic data in units of flows.

[0036] Currently, encrypted communications in a large number of networks use symmetric keys, and the size of data streams generally does not change too much. Therefore, the time and static statistical characteristics of encrypted traffic data can reflect the characteristics of encrypted traffic. The stream: A stream is composed of a series of data packets. A TCP stream is a TCP connection, a process from three handshakes to four waves; the stream is bidirectional, and the direction of a TCP stream is determined by the initiator of the first data packet of the TCP handshake. The initiator IP is the source IP of this TCP stream, and the receiver IP is the destination IP of this TCP stream. A stream includes all communication data packets from the initiator initiating the first TCP handshake packet to the initiator sending the last four-wave packet. It not only includes data packets with the initiator as the source IP and the receiver as the destination IP, but also includes data packets with the receiver as the source IP and the initiator as the destination IP.

[0037] The extracted statistical features include: time statistical features and static statistical features;

[0038] Time statistics features include:

[0039] Flow duration: Starting from the TCP handshake packet using the same five-tuple, it is counted as the start of the flow, and the timestamp of this TCP handshake packet is recorded as the start time of the flow. It lasts until the TCP handshake packet with the same five-tuple appears or the TCP RST ends. The difference between the timestamps of the two data packets is recorded as the flow duration.

[0040] Arrival interval of packets in a flow: The difference in timestamp between a packet in a flow and the previous packet is the flow arrival interval of the packet. Extract the maximum and minimum values ​​of the arrival interval of packets in the flow. Calculate the average value, variance, and the total time of the arrival interval.

[0041] Packet arrival interval in the forward flow: The timestamp difference between a packet and the previous packet in the forward flow (determined by the source IP and destination IP of the first TCP handshake packet) is the forward flow arrival interval of the packet. Extract the maximum and minimum values ​​of the packet arrival interval in the forward flow. Calculate the interval sum, average value, variance, and total time of the forward flow arrival interval.

[0042] Reverse flow packet arrival interval: The timestamp difference between a packet and the previous packet in the reverse flow (determined by the destination IP and source IP of the first TCP handshake packet) is the reverse flow arrival interval of the packet. Extract the maximum and minimum values ​​of the reverse flow packet arrival interval. Calculate the interval sum, average, variance, and total time from reverse flow to interval.

[0043] Static statistical features include:

[0044] Number of forward packets: The number of packets that make up the forward flow. Number of reverse packets: The number of packets that make up the reverse flow.

[0045] Total size of forward packets: the sum of the sizes of each packet that makes up the forward flow. From these forward packets, the maximum length and minimum length of the forward packets in the flow can also be extracted. The average size and variance of the forward packets in the flow are calculated.

[0046] Total size of reverse packets: the sum of the sizes of each packet that makes up the reverse flow. From these reverse packets, we can also extract the maximum and minimum lengths of the reverse packets in the flow. Calculate the average size and variance of the reverse packets in the flow. And from the forward and reverse packets, extract the maximum and minimum lengths of the packets in the flow. Calculate the average size and variance of the packets in the flow.

[0047] The above features are directly extracted features. Based on the above features, the following features can be indirectly calculated:

[0048] Number of packets per second for a flow: (number of forward packets + number of reverse packets) / flow duration

[0049] Forward flow packets per second: number of forward packets / flow duration

[0050] Reverse flow packets per second: number of reverse packets / flow duration

[0051] The size of the flow transmitted per second: (total size of forward packets + total size of reverse packets) / flow duration

[0052] The size of the forward flow transmitted per second: the total size of the forward data packet / flow duration

[0053] The size of the reverse flow transmitted per second: the total size of the reverse data packet / flow duration

[0054] The above features are all based on flow units. What is considered is the individual statistical features of a single flow.

[0055] Step 2: Data purification of the extracted features:

[0056] Remove duplicate flows. Flows with exactly the same extracted features are considered duplicate flows.

[0057] Remove the flows whose specified items are less than ε. The specified items refer to certain items defined by the user, and ε is the minimum value of the specified items defined by the user. These flows will be removed, as they will affect the final training results.

[0058] Step 3: Figure 2 As shown, the features purified in step 2 are restored:

[0059] The features of the flow purified in step 2 are grouped according to the five-tuple (source IP address, destination IP address, source port number, destination port number, protocol number), and each group contains a feature set of flows with the same five-tuple value. Then, each group (feature set) is restored, specifically: for each group, the minimum value of the corresponding item is subtracted from each feature of each flow in the group. Because the number of flows in the group is a finite number, the minimum value only needs to take the minimum value of all flows in the group on the corresponding item to complete the restoration of the above features. For the problem of zero that may appear in the subtraction, add 1 to all the result values ​​to obtain the result after restoration.

[0060] In this example, malicious traffic is detected in units of five tuples:

[0061] Features directly extracted from data traffic are called directly extracted features. Directly extracted features can be directly forged by attackers. Time statistical features and static statistical features are independent of each other. Changing time statistical features does not affect static statistical features: for example, changing the flow duration does not affect the size of the data packet in the forward flow. Forged malicious encrypted traffic targeting statistical features only changes its time statistical features or static statistical features without changing the code semantics, thereby generating forged malicious traffic.

[0062] Taking flow duration as an example, the attacker can make the program sleep for a period of time after each data packet is sent, so that the flow duration of the malicious encrypted traffic characteristics will become longer, and the static statistical characteristics will not be affected. According to the behavior and analysis of normal and malicious encryption software, the flow duration of normal encrypted traffic is generally longer than that of malicious encrypted traffic. Therefore, the forged malicious encrypted traffic generated above may be regarded as normal encrypted traffic, thereby bypassing detection. It can be seen from the analysis of the forged malicious encrypted traffic above. The key to restoring the forged sample features is to restore the directly extracted features. Because the forgery process above extends the flow duration, the flow duration is restored by reducing the flow duration.

[0063] Subtract the minimum value of the corresponding item from all the flow items in the group, and set the duration of an initial malicious encrypted traffic flow to be E i , the sleep time is S, and the duration of the forged malicious encrypted traffic flow is E i +S, let the duration of a normal encrypted traffic flow be Z i , Z i ≈E i +S.

[0064] Suppose a malicious encrypted traffic has the shortest flow duration E j , the sleep time is S, and the duration of the fake flow is E j +S, the shortest flow duration Z for normal encrypted traffic j After restoration, the duration of malicious encrypted traffic flow is E i +S-(E j + S) = E i -E j . Normal encrypted traffic flow duration Z i -Z j , eliminating the interference of sleep time S. Even forged samples can be distinguished without affecting the detection of ordinary samples.

[0065] Step 4: Figure 3As shown, the overall characteristics of the five-tuple are obtained by fine processing, and the average values ​​of all characteristic items are saved. For each flow in the group, the corresponding item average value is added to its value. For example, the average flow duration is added to its flow duration.

[0066] Taking the flow duration as an example, suppose a malicious encrypted traffic flow duration E i , the average flow duration E of malicious encrypted traffic j , the duration of the processed malicious encrypted traffic flow is E i +E j While retaining the individual characteristics of a single flow, the overall characteristics of the attack session are introduced. Considering the overall characteristics, it is easier to distinguish the encrypted traffic of different groups. In addition, a single attack or session may have data that is significantly different from the overall characteristics of the group, which is called extreme data. If only a single network data flow is considered, these data are likely to be misclassified, but after considering the overall characteristics, these extreme data can also be accurately classified.

[0067] Step 5: Train the deep learning model and put the new features obtained in step 4 into the deep learning model training. The deep learning model can be a general classification model in the field, such as Resnet, CNN, etc.

[0068] Step 6: Use the deep learning model to classify malicious encrypted traffic and output the deep learning model classification results.

[0069] The above detection method adds overall characteristics to improve the accuracy of detection. At the same time, possible forged samples are also processed to enhance the reliability of the method.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting forged malicious encrypted traffic based on refined statistical feature analysis, characterized in that: The specific steps include: Step 1: Obtain encrypted traffic data, and extract statistical features from the obtained encrypted traffic data in units of TCP flows; A TCP flow is a TCP connection, a process from three-way handshake to four-way handshake. The flow is bidirectional. The direction of a TCP flow is determined by the initiator of the first data packet of the TCP handshake. The initiator's IP is the source IP of the TCP flow, and the receiver's IP is the destination IP of the TCP flow. A flow includes all communication data packets from the initiator's first TCP handshake packet to the initiator's sending of the last four-way handshake packet. It includes not only data packets with the initiator as the source IP and the receiver as the destination IP, but also data packets with the receiver as the source IP and the initiator as the destination IP. The extracted statistical features of the flow include directly extracted statistical features and features calculated based on the directly extracted statistical features: The directly extracted statistical features include: source IP address, destination IP address, source port, destination port, protocol number, flow duration, data packet arrival interval in the flow, data packet arrival interval in the forward flow, data packet arrival interval in the reverse flow, number of data packets in the forward flow, number of data packets in the reverse flow, data packet size in the forward flow, and data packet size in the reverse flow; The following features are calculated based on the above features: number of packets per second in the flow, number of packets per second in the forward flow, number of packets per second in the reverse flow, size of transmission per second in the flow, size of transmission per second in the forward flow, size of transmission per second in the reverse flow; Step 2: purify the statistical features of the flow extracted in step 1, delete duplicate flows, and delete flows with a specified item less than ε, where ε is the minimum value set by the user; Step 3: Group the features of the flow purified in step 2 according to five-tuples, where the five-tuples include source IP address, destination IP address, source port number, destination port number, and protocol number; each group includes a feature set of flows with the same five-tuple value; then perform restoration processing on each group, specifically: for each group, subtract the minimum value of the corresponding item obtained in step 2 from each feature of each flow in the group. Since the number of flows in the group is a finite number, the minimum value only needs to be the minimum value of all flows in the group on the corresponding item, thus completing the restoration processing of the above features; For the problem that zero may appear in the subtraction, add 1 to all the result values ​​to get the restored result; Step 4: refine the features after the reduction process in step 3 to obtain the overall features of the quintuple, save the average value of all feature items, and for each flow within the group, add the corresponding item average value to its value; Step 5: Train the deep learning model and put the new features obtained in step 4 into the deep learning model training; Step 6: Use the deep learning model to classify malicious encrypted traffic and output the malicious encrypted traffic classification results.

2. The method for detecting forged malicious encrypted traffic based on refined statistical feature analysis as claimed in claim 1, characterized in that: The deep learning model described in step 5 adopts Resnet or CNN model.

Images