Botnet traffic detection method and device based on multi-granularity statistical characteristics

Abstract

The invention discloses a botnet flow detection method and device based on multi-granularity statistical characteristics, and belongs to the technical field of network security. In order to solve the problems that an existing botnet detection method is low in accuracy and insufficient in detection capability, deep learning analysis is mainly carried out by adopting multi-granularity statistical feature extraction, source IP-oriented global fine-granularity portrait construction and an LSTM model with a multi-head attention mechanism. By combining the coarse-grained statistical features and the fine-grained portrait, the time sequence features of the Botnet traffic can be effectively extracted, and the accuracy and robustness of detection are improved. According to the invention, accurate detection of Botnet traffic in different time periods and a full life cycle can be realized, and the identification capability of the network traffic is significantly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and particularly relates to a botnet traffic detection method and device based on multi-granularity statistical features. Background Art

[0002] With the development of quantum computing and 5G Internet of Things, the network supply side has been further expanded, and network attacks have shown characteristics of high frequency, high harm, and high technology. Attackers use these attack surfaces to bring many threats to the national cyberspace. Typical threats such as botnets. A botnet is a collection of hosts infected by bot programs. The botnet is controlled by a bot master through a control protocol and can be distributed inside a local area network and on the Internet. Attackers can use botnets to carry out distributed denial-of-service (DDOS) attacks, spam delivery, ransomware propagation, sensitive information theft, etc., which have great harm. For example, the Emotet botnet spreads through malicious emails, distributes the Ryuk ransomware, infects more than 1.6 million computers globally, and the total loss exceeds $2.5 billion. Timely and effective botnet detection methods can curb other derivative attacks launched by attackers based on botnets, ensure the security of the cyberspace, and reduce the impact of network attacks on society and the economy.

[0003] Current botnet detection methods can be divided into host behavior-based detection methods, network behavior-based detection methods, and comprehensive correlation behavior-based methods. Host behavior-based detection methods and comprehensive correlation behavior-based methods need to collect and aggregate system call information related to modifying the system registry, creating network connections, etc. on the host side. The collection of system call information will consume the computing resources of the host side, and information aggregation will occupy a large amount of network link bandwidth. Network behavior-based detection methods mostly carry out detection work based on network traffic. Existing network traffic-based detection methods include network traffic load content-based detection methods and network flow statistical feature-based detection methods. Among them, with the increase in the amount of network data packets and the development of end-to-end encryption, it is difficult for network traffic load content-based detection methods to carry out botnet detection work in a timely and effective manner; network flow statistical feature-based detection methods aggregate network data packets with the same attributes within a period of time into network flows (such as the Netflow protocol) and perform detection based on the statistical information of the network flows, which alleviates the contradiction between the large amount of network data packets and the requirements of end-to-end encryption and botnet detection to a certain extent.

[0004] With the development of machine learning or deep learning, researchers have tried to introduce machine learning or deep learning algorithms such as support vector machine (SVM), K-nearest neighbors (KNN), decision tree, random forest, and convolutional neural network into the botnet detection process, which has improved the scalability and accuracy of botnet detection methods to a certain extent. However, there are still problems such as a single feature selection perspective that can be easily bypassed by attackers by adjusting the network data packet structure or a high false alarm rate. Summary of the invention

[0005] The purpose of the present invention is to provide a botnet traffic detection method and device based on multi-granularity statistical features, by obtaining coarse-grained statistical features of local network flows and fine-grained portraits of global network flows oriented to source IP as network traffic features, and using an LSTM model based on a multi-head attention mechanism to deeply explore the differences in statistical features between botnet and benign network traffic at different time series, and effectively realize intelligent detection of botnet traffic.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is as follows:

[0007] A method for detecting botnet traffic based on multi-granularity statistical features, the steps of which include:

[0008] 1) Extracting network flow metadata from network traffic, performing data preprocessing, and obtaining standardized network flow metadata;

[0009] 2) Based on the set time window, in the standardized network flow metadata, count the features of other network flows with the same attributes as the network flow to be detected, and extract the coarse-grained statistical features of the local network flow;

[0010] 3) In the standardized network flow metadata, obtain all historical network flows with the same source IP as the network flow to be detected, count the access behavior information of the source IP, and build a global network flow fine-grained portrait for the source IP;

[0011] 4) Convert the obtained local network flow coarse-grained statistical features and global network flow fine-grained portraits into vectors respectively, and concatenate them to generate a network flow feature vector for detection;

[0012] 5) The network flow feature vector is input into the botnet traffic detection model constructed based on the long short-term memory network with multi-head attention mechanism, sequence modeling and feature attention analysis are performed, and the network flow detection results are output.

[0013] Further, the metadata of the network flow in step 1) includes the start time, duration, source IP, source port, destination IP, destination port, the number of data packets and the number of bytes sent from the source IP to the destination IP.

[0014] Further, the data preprocessing in step 1) includes: filling missing values in the metadata of the network flow, converting outliers, and converting data types.

[0015] Further, the steps of extracting the coarse-grained statistical features of the local network flow in step 2) include:

[0016] Counting the number of network flows with the same source-end attributes as the network flow to be detected;

[0017] Counting the number of network flows with the same source-end and destination-end attributes as the network flow to be detected;

[0018] Counting the number of network flows with the same destination-end attributes as the network flow to be detected;

[0019] Taking the above-mentioned counted numbers of various network flows as the coarse-grained statistical features of the local network within the time window.

[0020] Further, the source-end attributes include the source IP, source port, and the combination of source IP and source port, and the destination-end attributes include the destination IP, destination port, and the combination of destination IP and destination port.

[0021] Further, the access behavior information of the source IP in step 3) includes the number of target ports accessed, average session duration, average number of data packets, and average number of bytes.

[0022] Further, the steps of obtaining the fine-grained portrait of the global network flow for the source IP in step 3) include:

[0023] In all historical network flows, counting the number of network flows with the same source IP and target port as the network flow to be detected, the average number of data packets, the average number of sent bytes, the average communication duration, and the communication protocol distribution;

[0024] Using the source IP as an index, counting all the target ports accessed in its history, and calculating the average communication duration, average number of sent bytes, and protocol distribution for each target port to construct the global network behavior portrait corresponding to the source IP.

[0025] Further, the steps of generating the network flow feature vector in step 4) include:

[0026] Constructing a feature vector based on the coarse-grained statistical features of the local network flow according to a preset arrangement order;

[0027] Constructing corresponding feature vectors for each field in the fine-grained portrait of the global network flow through dictionary mapping;

[0028] Concatenate the above two feature vectors in dimension to generate a unified network flow feature vector.

[0029] Furthermore, the preset arrangement order is: source - end attribute statistics, combined source - end and destination - end attribute statistics, destination - end attribute statistics.

[0030] A botnet traffic detection device based on multi - granularity statistical features includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs to implement the steps of the above - mentioned method.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1) Based on the extraction of local network flow coarse - granularity statistical features, the present invention further constructs a global network flow fine - granularity portrait oriented to the source IP, comprehensively depicts network behavior characteristics, and improves the modeling ability of the long - term behavior patterns of botnets;

[0033] 2) By fusing coarse - granularity and fine - granularity features, the present invention enriches the network flow representation information, which helps deep models to identify complex behavior differences;

[0034] 3) The present invention introduces a detection model combining the multi - head attention mechanism and LSTM, which can dynamically focus on key positions in the network flow sequence and enhance the modeling effect of temporal dependence and global context;

[0035] 4) The present invention can significantly improve the effectiveness and robustness in botnet traffic detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a framework diagram of the botnet traffic detection method based on multi - granularity statistical features of the present invention.

[0037] Figure 2 is a cumulative statistical data graph of Rbot and Neris botnet flows and benign network flows accessing port 53.

[0038] Figure 3 is a structural diagram of the botnet traffic detection model based on LSTM with multi - head attention mechanism. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the embodiments of the present invention in detail.

[0040] The embodiments of the present invention specifically disclose a botnet traffic detection method based on multi - granularity statistical features, as Figure 1As shown in the figure, the method mainly includes the following three parts: extraction of network flow statistical features, vectorization of network flow features, and detection of botnet traffic using a detection model.

[0041] 1) Extraction of network flow statistical features

[0042] The network flow statistical features consist of two types of sub-features: one is the coarse-grained statistical features of local network flows based on a time window, and the other is the fine-grained behavior portrait of global network flows facing the source IP.

[0043] 1.1) Extraction of coarse-grained statistical features of local network flows based on a time window

[0044] Extract the local statistical features of the current network flow to be detected within a fixed time window δ (i.e., t - δ to t). Suppose the network flow sequence within the time window is nfList = [nf0, nf1, …, nf k , where it satisfies the condition t - δ ≤ nf j [startTime] ≤ t, and j ∈ {0, 1, …, k}. The network flow to be detected at the source end (i.e., the request initiating end) is denoted as nf Γ , where nf Γ ∈ {nf0, nf1, …, nf k}.

[0045] Within this time window, count the number of network flows with the same attributes or attribute combinations as nf Γ . Specifically, it includes the following three types of statistical features:

[0046] ① Only count the number of network flows with the same source-end attributes (including source IP, source port, and the combination of source IP and source port) as nf Γ , as shown in formula (1):

[0047]

[0048] ② Count the number of network flows with the same source-end attributes and destination-end (request receiving end) attributes as nf Γ . The source-end attributes include source IP, source port, and the combination of source IP and source port, and the destination-end attributes include destination IP, destination port, and the combination of destination IP and destination port, as shown in formula (2):

[0049]

[0050] ③ Only count the number of network flows with the same destination-end attributes (including destination IP, destination port, and the combination of destination IP and destination port) as nf Γ , as shown in formula (3):

[0051]

[0052] Based on the above statistical logic, the specific steps for extracting the coarse-grained statistical features of local network flows are as follows:

[0053] First, initialize the statistical feature dictionary featDict within the time window;

[0054] Then, obtain the network flow sequence nfList that has been preprocessed within the time window with a start time of st and a duration of δ;

[0055] Finally, compare each network flow in nfList one by one and count the following three types of feature values: the number of network flows with the same source-end attribute value as nf Γ the number of network flows with the same destination-end attribute value as nf Γ and the number of network flows with the same source-end and destination-end attribute values as nf Γ

[0056] 1.2) Fine-grained Portrait of Global Network Flows Oriented to Source IP

[0057] Although the coarse-grained statistical features of local network flows based on time windows can effectively reflect the behavioral characteristics of botnet traffic within a specific time period, it is difficult to comprehensively depict the cumulative behavior patterns of network traffic generated by bot hosts throughout their entire life cycle. As Figure 2 shown in the two figures, when hosts infected by botnet malware (such as Rbot, Neris) access the destination port 53, their long-term cumulative statistical features are significantly different from those of normal hosts. Therefore, the present invention introduces a fine-grained portrait mechanism for global network flows oriented to source IP to supplement the behavioral features at the life cycle level.

[0058] The specific method is as follows:

[0059] First, extract the statistical information (access frequency, number of data packets, average number of bytes, communication time, and protocol distribution) of network flows with the same source IP and destination port as the current network flow to be detected nf Γ (where nf Γ = nf n ) from all historical network flow sequences. Let all historical network flow sequences be: allNfList = [nf0, nf1,..., nf n , where nf0[startTime] < nf1[startTime] <... < nf n [startTime].[[]END]]

[0060] On this basis, calculate the following fine-grained statistical features:

[0061] ① Count, among all historical network flows, the number of network flows with the same source IP as nf Γ ​The same (denoted as vsip = nf Γ [sip]) and the target ports are the same (denoted as vdport = nf Γ [dport]), the number of network flows is shown in formula (4):

[0062]

[0063] ② Statistically analyze all historical network flows, and for the network flows with the same source IP and target ports as nf Γ , count the total number of data packets sent, as shown in formula (5):

[0064]

[0065] ③ Calculate for all historical network flows, for the network flows with the same source IP and target ports as nf Γ , calculate the average number of bytes per data packet sent, as shown in formula (6):

[0066]

[0067] ④ Calculate for all historical network flows, for the network flows with the same source IP and target ports as nf Γ , calculate the average communication duration, as shown in formula (7):

[0068]

[0069] ⑤ Statistically analyze all historical network flows, for the network flows with the same source IP and communication protocol as nf Γ (denoted as vproto = nf Γ [proto]), count the distribution of the communication protocols used, as shown in formula (8):

[0070]

[0071] Then, further construct a multi-feature fine-grained portrait in the source IP dimension. By analyzing the interaction behavior of the source IP of the network flow to be detected with different target ports in the historical network flows, extract its global portrait in terms of communication characteristics, including the access frequency of target ports, average communication duration, average number of bytes, and protocol usage distribution, etc.

[0072] Specifically, it includes the following five aspects of statistical analysis:

[0073] ① As shown in formula (9), when fea = dport, statistically analyze the network flows in the historical network flows with the same source IP as the current network flow nf Γ , and count the access distribution in the dimension of target ports to construct a portrait of the source IP in the dimension of accessing target ports;

[0074]

[0075] Among them, vport0, vport1, …, vport r ∈ {nf i [dport] | nf i [sip] == vsip and 0 ≤ i ≤ n}

[0076]

[0077] and nf j [dport] == port q and 0 ≤ q ≤ r and 0 ≤ j ≤ n})

[0078] and fea ∈ {dport, toTPkts, ave_toTBytes, ave_dur}

[0079] ② As shown in formula (9), when fea = toTPkts, count the number of data packets sent when accessing each destination port for the historical network flows with the same source IP vsip, which is used to construct the access portrait of the source IP in the dimension of "number of data packets".

[0080] ③ As shown in formula (9), when fea = ave_dur, calculate the average number of bytes of data sent by the source IP vsip during the process of accessing each destination port, which is used to construct the access portrait of the source IP in the dimension of the average value of the number of bytes of sent data.

[0081] ④ As shown in formula (9), when fea = ave_dur, calculate the average communication duration when the source IP vsip accesses each destination port, which is used to construct the access portrait of the source IP in the dimension of the average value of the communication duration.

[0082] ⑤ As shown in formula (10), count the communication protocols used in the historical network flows with the same source IP vsip, and construct the portrait of the source IP in the dimension of "protocol usage distribution".

[0083]

[0084] Among them, proto0, vproto1, …, vproto p ∈ {nf i [proto] | nf i [sip] == nf[sip] and 0 ≤ i ≤ n}

[0085]

[0086] and nf i[proto] == vproto q and 0 ≤ q ≤ n and 0 ≤ i ≤ n}

[0087] On this basis, the steps for constructing the fine-grained portrait of the global network flow facing the source IP are as follows:

[0088] First, initialize the historical network flow statistical structure globalFeatDict;

[0089] Then, traverse all historical network flows, and count the number of networks, total communication duration, total number of sent data packets, and total number of sent bytes that have the same source IP and destination port as the current network flow nf Γ and update them to globalFeatDict;

[0090] Next, calculate the average communication duration and average number of sent bytes of the above network flows in the dimension of the destination port;

[0091] Finally, based on the above statistical results, extract the fine-grained behavior portrait corresponding to the source IP of the current network flow.

[0092] 2) Vectorization of network flow features

[0093] The vectorization of network flow features aims to transform network flow features of different granularities and dimensions into mathematical vectors that can be used for model training and inference. The specific process is as follows:

[0094] First, arrange the coarse-grained statistical features of network flows extracted based on time windows in a preset order to construct a mathematical vector. As shown in formula (11), the feature arrangement order is: source-end statistical features, two-end statistical features, and destination-end statistical features.

[0095]

[0096] where att1 ∈ {sip, sport, sip:sport, any} and att2 ∈ {any, dip, dip:dport, dport}

[0097] Secondly, for each feature dimension of the fine-grained portrait of the global network flow extracted facing the source IP, vector construction is performed respectively by means of dictionary mapping. As shown in formula (12) and formula (13), different dimensions (such as destination port distribution, protocol distribution, average communication duration, etc.) are respectively mapped to feature vectors of a fixed length.

[0098]

[0099] where

[0100] Among them,

[0101] Finally, the coarse-grained feature vector generated based on formula (11) and the fine-grained feature vectors generated based on formulas (12) and (13) are horizontally stacked to obtain the final network flow feature vector featureVector for subsequent model input, where

[0102] 3) Detection by the botnet traffic detection model

[0103] Construct a botnet traffic detection model (Multi-HeadAttention-based LSTM Botnet Detection Model, abbreviated as M-AttLSTMBotModel) based on the long short-term memory network with multi-head attention mechanism. Its structure is as Figure 3 shown. This model is used to model the input network flow sequence, dynamically focus on the information at different positions in the sequence, effectively capture the dependencies between network flows, extract the network flow context features, and thus identify the differences between botnet flows and benign network flows in the statistical feature dimension.

[0104] The LSTM network has the ability to model time series features and can capture the temporal variation features of botnet flows and normal network flows in terms of coarse-grained statistical features and fine-grained portraits, thereby assisting in detection. The multi-head attention mechanism has the ability to simultaneously focus on multiple key positions in the input sequence and has significant advantages in capturing long-range dependencies and global features, which helps to further improve the model's ability to extract key traffic features.

[0105] The core structure of LSTM includes the input gate I t , the forget gate F t , the output gate O t , the candidate memory unit the memory unit C t and the hidden state H t . Among them, the input gate I t controls which information in the current input X t will be introduced into the memory unit C t , the forget gate F t decides which information in the memory unit C t-1 from the previous moment is retained, the output gate O t is responsible for controlling the information output in the hidden state H t , and the candidate memory unit is based on the current input X tAbstract representation, and finally the memory cell C t Comprehensive F t and I t The screening result generation of, the hidden state H t Then it reflects the model's understanding of the current input. The calculation process of LSTM is shown in formulas (14) to (17):

[0106] I t = σ(W i X t + U i H t-1 + b i ) (14)

[0107] F t = σ(W f X t + U f H t-1 + b f ) (15)

[0108] O t = σ(W o X t + U o H t-1 + b o ) (16)

[0109]

[0110] H t = O t * tanh(C t ) (19)

[0111] Among them, X t ∈ R n×|featureVector| ; W i , W f , W o , W c ∈ R |featureVector|×h ; U i , U f , U o , U c ∈ R h×h ; b i , b f , b o , b c ∈ R 1×h .

[0112] After the network flow passes through the LSTM layer, it enters δ parallel attention calculation models as shown in formula (20):

[0113]

[0114] The calculation process of each attention calculation model is shown in Formulas (21) to (24):

[0115]

[0116] H = (H1, H2, … H t ) (25)

[0117] Among them, Q l represents the attention focus that the current network flow hopes to obtain, K l represents the degree of association of each position network flow with the current attention target, V l represents the attention value actually obtained by the current network flow, l = 1, 2, 3, … δ; the weight matrix d k = h × 64.

[0118] To sum up, the overall calculation process of the model is shown in Formulas (26) to (27), and the definitions of each parameter in the formula are the same as those in Formulas (14) to (25).

[0119]

[0120] The present invention combines the sequence modeling ability of LSTM and the global modeling ability of the multi-head attention mechanism, significantly improving the representation ability and classification performance of the model in the zombie network traffic detection task.

[0121] Experimental test:

[0122] To verify the effectiveness of the method proposed by the present invention, the CTU-13 dataset released by the Czech Technical University in Prague and the ISCX botnet dataset released by the Canadian Centre for Information Security are used for experimental evaluation. The zombie network traffic detection task is modeled as a binary classification problem: the model input is the statistical features extracted from the network flow, and the output is whether the traffic belongs to the zombie network. The experimental results on the two datasets are shown in Table 1, indicating that the method of the present invention exceeds 0.99 in terms of accuracy, precision, recall, and F1 value, verifying the excellent detection performance and robustness of the method of the present invention.

[0123] Table 1 Model detection analysis results

[0124]

[0125] Although specific embodiments and drawings of the present invention are disclosed for illustrative purposes, which are intended to assist in understanding the content of the present invention and implementing it accordingly, those skilled in the art can understand that: within the spirit and scope of the present invention and the appended claims, various substitutions, changes, and modifications are possible. Therefore, the present invention should not be limited to the content disclosed in the preferred embodiments and drawings, and the scope of protection claimed by the present invention shall be defined by the scope defined in the claims.

Claims

1. A method for detecting botnet traffic based on multi-granularity statistical features, characterized in that the steps Including: 1) Extract the metadata of network flows from network traffic, perform data preprocessing, and obtain standardized network flow metadata; 2) Based on a set time window, in the standardized network flow metadata, count the features of other network flows with the same attributes as the network flow to be detected, and extract the coarse-grained statistical features of local network flows; 3) In the standardized network flow metadata, obtain all historical network flows with the same source IP as the network flow to be detected, count the access behavior information of this source IP, and construct a fine-grained portrait of the global network flow for the source IP; 4) Convert the obtained coarse-grained statistical features of local network flows and the fine-grained portrait of global network flows into vectors respectively, and splice them to generate a network flow feature vector for detection; 5) Input the network flow feature vector into a botnet traffic detection model constructed by a long short-term memory network based on the multi-head attention mechanism, perform sequence modeling and feature attention analysis, and output the detection result of the network flow.

2. The method according to claim 1, characterized in that, In step 1), the metadata of the network flow includes the start time, duration, source IP, source port, destination IP, destination port, the number of data packets and the number of bytes sent from the source IP to the destination IP.

3. The method according to claim 2, wherein In step 1), the data preprocessing includes: filling missing values in the metadata of the network flow, converting outliers, and converting data types.

4. The method according to claim 1, characterized in that The steps of extracting the coarse-grained statistical features of local network flows in step 2) include: Count the number of network flows with the same source-end attributes as the network flow to be detected; Count the number of network flows with the same source-end and destination-end attributes as the network flow to be detected; Count the number of network flows with the same destination-end attributes as the network flow to be detected; Take the above-mentioned counted numbers of various network flows as the coarse-grained statistical features of the local network within the time window.

5. The method according to claim 4, wherein The source-end attributes include the source IP, source port, and the combination of source IP and source port, and the destination-end attributes include the destination IP, destination port, and the combination of destination IP and destination port.

6. The method according to claim 1, wherein In step 3), the access behavior information of the source IP includes the number of target ports accessed, average session duration, average number of data packets, and average number of bytes.

7. The method according to claim 1, characterized in that, The steps of obtaining the fine-grained portrait of the global network flow for the source IP in step 3) include: In all historical network flows, count the number of network flows with the same source IP and target port as the network flow to be detected, average number of data packets, average number of sent bytes, average communication duration, and communication protocol distribution; Using the source IP as an index, count all the target ports it has accessed historically, and calculate the average communication duration, average number of sent bytes, and protocol distribution of each target port to construct a global network behavior portrait corresponding to the source IP.

8. The method according to claim 1, wherein The steps of generating the network flow feature vector in step 4) include: Construct a feature vector for the coarse-grained statistical features of local network flows according to a preset arrangement order; Construct corresponding feature vectors for each field in the fine-grained portrait of the global network flow through dictionary mapping; Splice the above two feature vectors in dimension to generate a unified network flow feature vector.

9. The method according to claim 8, wherein The preset arrangement order is: source-end attribute statistics, source-end and destination-end combined attribute statistics, destination-end attribute statistics.

10. A botnet traffic detection device based on multi-granularity statistical features, characterized in that, It includes a memory and a processor. The memory is used to store a computer program, and the processor is used to execute the computer program to implement the steps of the method according to any one of claims 1-9.

Images