An explainable abnormal traffic detection method for closed internal networks

Abstract

The present invention discloses an interpretable abnormal traffic detection method for closed internal networks, belonging to the field of cyberspace security technology. First, a deep autoencoder is used to learn the deep pattern of benign traffic and reconstruct the benign traffic. Under the guidance of the intermediate output of the deep autoencoder, the gradient ascent method is used to extract whitelist rules from the reconstructed benign traffic features. The whitelist rule library obtained by the present invention can not only retain the good abnormality detection performance of the deep autoencoder, but also provide users with clear decision explanations, achieving a balance between detection accuracy and interpretability. At the same time, the present invention significantly reduces software and hardware overhead and is more suitable for closed internal networks.

Description

Technical Field

[0001] The present invention belongs to the field of cyberspace security technology, and relates to deep learning, anomaly detection and rule extraction, and specifically to an explainable abnormal traffic detection method for closed internal networks. Background Art

[0002] Closed internal networks with specialized purposes, characterized by a highly closed topology and highly regular node communication behavior, are vulnerable to complex and diverse network threats. According to the kill-chain theory, attacks on closed internal networks involve multiple steps, including reconnaissance, infiltration, and payload delivery. This complex attack process often leads to abnormal fluctuations in internal network traffic. Therefore, abnormal traffic detection is a necessary prerequisite for identifying network threats.

[0003] In practice, due to the ever-changing nature of possible attack methods, defenders inherently lag behind in the cyberattack-defense game. Currently used blacklist-based rule bases and supervised learning models can only detect known types of anomalous traffic, but cannot effectively identify new, unknown types of anomalous traffic. Therefore, unsupervised anomalous traffic detection technology based on benign traffic analysis is gaining increasing attention.

[0004] Compared with abnormal traffic, the characteristics of benign traffic are relatively stable and change less. Moreover, due to the different purposes of implementing network behaviors, no matter how new the abnormal traffic is, its characteristic distribution is still different from that of benign traffic. Unsupervised learning models can fit the deep distribution pattern of benign traffic, and inputs that do not conform to this pattern are judged as abnormal. This method has been proven to be effective in identifying unknown new abnormal traffic. Typical related work includes: density-based methods LOF 【1】 , ECOD based on empirical cumulative distribution method 【2】 , ABOD method based on eigenvector angle 【3】 ; Partition-based method IForest 【4】 , OCSVM method based on single-class hyperplane 【5】 ; Auto Encoder based on reconstruction error 【6】 .

[0005] However, existing unsupervised anomaly detection technologies have some shortcomings. Methods based on density, empirical cumulative distribution, and eigenvector angles all make prior assumptions about the distribution of benign samples. For example, normal samples are always densely distributed, normal samples are always numerically common rather than rare, and the angles between the eigenvectors of normal samples are small. The introduction of prior assumptions makes these methods easy to understand and explain. However, the distribution of normal traffic samples is complex, and abnormal traffic detection methods based on specific prior assumptions are not always effective. This is unless a sufficiently universal prior assumption can be found, which is almost impossible. Methods based on partitioning and single-class hyperplanes have the disadvantages of high computational complexity, difficulty in convergence, and easy overfitting. Methods based on reconstruction error are more robust in principle, but their internal decision-making process is almost completely incomprehensible.

[0006] The most serious shortcoming of existing technologies lies in their lack of explainability. The decision-making process of a typical unsupervised learning model is difficult or even impossible for humans to understand, acting more like a black box model. The rationality of the decision-making results cannot be fully evaluated, and security operators are unable to assess and correct the model's decision-making results based on prior knowledge of benign traffic. For internal networks with strong closed structures and strict communication behavior constraints, the interpretability of anomaly traffic detection models is highly demanding. Anomaly detection results based on black box models are difficult to trust, and clear, explainable whitelist rules are clearly the most ideal anomaly traffic detection solution. However, traditional whitelist rule bases are based on expert knowledge, and approaching the detection performance of complex unsupervised learning models requires a significant investment of manpower and time.

[0007] The deployment of some typical unsupervised learning methods for detecting abnormal traffic requires specialized hardware and software support, such as graphics cards and the CUDA parallel computing library. However, the BM requirements for closed internal networks are high, and the introduction of graphics cards and open-source CUDA libraries from overseas manufacturers hinders the security management and control of the intranet's network asset supply chain.

[0008] Existing anomaly traffic detection models based on unsupervised learning often include hyperparameters that require tuning based on anomaly samples. However, attacks against closed internal networks often originate from adversarial nation-state cyberattack and defense organizations. The anomaly traffic samples they generate are highly concealed and difficult to collect, making hyperparameter tuning extremely challenging.

[0009] References

[0010] [1]Markus M.Breunig,Hans-Peter Kriegel,Raymond T.Ng,and Sander.2000.LOF:identifying density-based local outliers.SIGMOD Rec.29,2(June2000),93–104.

[0011] [2]Z.Li,Y.Zhao,X.Hu,N.Botta,C.Ionescu and G.H.Chen,"ECOD:UnsupervisedOutlier Detection Using Empirical Cumulative Distribution Functions,"in IEEETransactions on Knowledge and Data Engineering,vol.35,no.12,pp.12181-12193,1Dec.2023,doi:10.1109 / TKDE.2022.3159580.

[0012] [3]Hans-Peter Kriegel,Matthias Schubert,and Arthur Zimek.2008.Angle-based outlier detection in high-dimensional data.In Proceedings of the 14thACM SIGKDD international conference on Knowledge discovery and data mining(KDD'08).Association for Computing Machinery,New York,NY,USA,444–452.

[0013] [4]F.T.Liu,K.M.Ting and Z.-H.Zhou,"Isolation Forest,"2008Eighth IEEEInternational Conference on Data Mining,Pisa,Italy,2008,pp.413-422,doi:10.1109 / ICDM.2008.17.

[0014] [5] A.Bounsiar and MGMadden, "One-Class Support Vector Machines Revisited," 2014 International Conference on Information Science & Applications (ICISA), Seoul, Korea (South), 2014, pp.1-4, doi:10.1109 / ICISA.2014.6847442.

[0015] [6] J.Zhai, S.Zhang, J.Chen and Q.He, "Autoencoder and Its VariousVariants," 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan, 2018, pp.415-419, doi:10.1109 / SMC.2018.00080. Summary of the Invention

[0016] In response to the problems of poor interpretability and limited hyperparameter tuning in the existing technology for abnormal traffic detection in closed internal networks, the present invention proposes an explainable abnormal traffic detection method for closed internal networks. This method uses the intermediate output of unsupervised deep learning to extract whitelist rules, uses deep learning technology to ensure detection accuracy, and provides a clear decision-making process in the form of an IF-THEN rule base, taking into account both anomaly detection accuracy and the interpretability of detection results.

[0017] The method for detecting explainable abnormal traffic in a closed internal network includes the following steps:

[0018] Step 1: Train a deep autoencoder using several benign traffic samples to learn deep patterns in benign traffic.

[0019] The deep autoencoder consists of two parts: encoder and decoder. The training process is:

[0020] Step 101: Map the original benign traffic sample input vector into a low-dimensional vector through the encoder and output it to the decoder;

[0021] Step 102: The decoder reconstructs the low-dimensional vector into the original input vector and calculates the reconstruction loss between the reconstructed vector and the original vector.

[0022] The original training dataset is denoted as Initialize the autoencoder model M, record the model parameters (number of hidden layer neurons, learning rate, optimizer, etc.) as θ, use the mean square error as the loss function and record it as Π(), the formula is as follows:

[0023]

[0024] Step 103: Determine whether the reconstruction loss has reached convergence. If so, the iteration stops and the deep autoencoder training is completed. Otherwise, the decoder and encoder parameters are updated based on the reconstruction loss guidance, and the process returns to step 101 to re-perform encoding mapping and decoding reconstruction and continue iteration.

[0025] Step 2: Benign traffic samples After the trained deep autoencoder is reconstructed and expanded, the expanded feature space is clustered and divided into clusters, each cluster representing a sub-distribution;

[0026] Specifically:

[0027] First, benign traffic samples After the trained deep autoencoder, the reconstruction loss is π 1 i , using reconstruction loss to convert the original feature vector Expand to The formula is:

[0028]

[0029] Then, after the features of benign traffic are expanded from m dimension to m+1 dimension, the new feature space is recorded as

[0030] Finally, the new feature space is divided into clusters using the clustering algorithm model Cluster_Model, and the cluster set is obtained as follows: Each cluster Represents a sub-distribution of benign traffic.

[0031] The formula for dividing clusters is:

[0032]

[0033] Step 3: Based on the principle of gradient ascent of reconstruction loss, boundary rules are extracted for each sub-distribution;

[0034] The specific process of boundary rule extraction is as follows:

[0035] In step 301, all sub-distributions are traversed to generate a minimum hypercube for each cluster, which is recorded as Mini-Hyper-Cube. At the same time, the maximum reconstruction loss of benign samples in each cluster is recorded as thres.

[0036] In step 302 , several sample instances are selected as Explorers in the shallow neighborhood inside each hyperplane of each Mini-Hyper-Cube.

[0037] Step 303: Calculate the direction of the gradient increase of the reconstruction loss in each cluster and move the Explorer along this direction until the reconstruction loss is greater than thres. Record the feature corresponding to the gradient direction at this time. The value of and It is a boundary threshold of this feature.

[0038] Step 304: Repeat the above process on each hyperplane to obtain the boundary threshold of each feature, and combine them into the boundary rule of the current cluster, which is expressed as:

[0039]

[0040] Among them, 1, 2,…, k are the hyperplane numbers.

[0041] In step 4, the boundary rules of each sub-distribution are merged to form a complete whitelist rule base that describes the global distribution of benign traffic.

[0042] The merging process is: read the boundary rules of a sub-distribution, add them to the overall rule base, and perform redundancy removal operations on the overall rule base until the boundary rules of all sub-distributions are traversed.

[0043] The redundancy removal operation is as follows: if in the overall rule base, there is a boundary rule of a sub-distribution a that is completely contained in the boundary rule of another sub-distribution b, then the boundary rule of sub-distribution b is removed and the boundary rule of sub-distribution a is retained.

[0044] Step 5: Detect unknown traffic data in a closed internal network using a whitelist rule library to obtain abnormal traffic conditions.

[0045] The advantages of the present invention are:

[0046] (1) The present invention uses a benign traffic sub-distribution delineation method that incorporates the reconstruction error and the original feature distribution, resulting in a more accurate and in-depth analysis of benign traffic. Compared to sub-distribution delineation methods based solely on feature distribution clustering, this method is more scientific and precise, with lower coupling between the delineated sub-distributions, and can more accurately reflect the underlying distribution pattern of benign traffic.

[0047] (2) The boundary rule extraction method based on the gradient ascent method in the present invention significantly enhances the interpretability of abnormal traffic detection while taking into account the detection accuracy. This method draws on the idea of ​​adversarial sample generation and continuously explores the distribution boundary points along the direction of the gradient ascent of the reconstruction loss within each sub-distribution to extract the boundary rules of each sub-distribution. The extracted rules are guided by the intermediate output of the autoencoder (reconstruction error), which can not only retain the strong anomaly detection ability of the autoencoder but also facilitate user understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of the overall steps of the explainable abnormal traffic detection method for a closed internal network of the present invention;

[0049] Figure 2 This is a flowchart of training a deep autoencoder in the interpretable abnormal traffic detection method for a closed internal network of the present invention;

[0050] Figure 3 A flowchart of demarcating sub-distributions by fusing reconstruction errors and original features in the interpretable abnormal traffic detection method for closed internal networks of the present invention;

[0051] Figure 4 This is a flowchart of extracting sub-distribution boundary rules based on the gradient ascent method in the explainable abnormal traffic detection method for closed internal networks of the present invention;

[0052] Figure 5 This is a flow chart of merging sub-distribution boundary rules in the explainable abnormal traffic detection method for closed internal networks of the present invention;

[0053] Figure 6 This is a comparison chart of the true positive rates of different anomaly detection methods in real intranet traffic in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0055] The purpose of this invention is to provide an explainable abnormal traffic detection method for a closed internal network. The overall process is as follows: Figure 1As shown. A deep autoencoder is used to learn the deep patterns of benign traffic and reconstruct benign traffic with a smaller error. Under the guidance of the intermediate output of the deep autoencoder, the gradient ascent method is used to extract whitelist rules from the benign traffic features, so that the originally unexplainable decision-making process of the black box model is converted into easy-to-understand IF-THEN rules. The whitelist rule base obtained based on the above method can not only retain the good anomaly detection performance of the deep autoencoder, but also provide users with clear decision explanations, achieving a balance between detection accuracy and explainability, and has the advantages of accurate detection, transparent decision-making, and easy deployment. In addition, users can choose their own technical solutions to implement the storage and matching of the rule base, which makes it easy to maintain the security of the software supply chain, significantly reduces software and hardware costs, and is more suitable for closed internal networks.

[0056] A method for detecting explainable abnormal traffic in a closed internal network, comprising the following steps:

[0057] Step 1: Train a deep autoencoder with benign traffic samples to learn deep patterns in benign traffic.

[0058] The deep autoencoder consists of two parts: encoder and decoder. Figure 2 As shown, the training process is:

[0059] Step 101: The original benign traffic sample is mapped into a low-dimensional vector through an encoder and output to a decoder;

[0060] Step 102: The decoder reconstructs the low-dimensional vector into the original input vector and calculates the reconstruction loss between the reconstructed vector and the original vector.

[0061] The original training dataset is denoted as Initialize the autoencoder model M, record the model parameters (number of hidden layer neurons, learning rate, optimizer, etc.) as θ, use the mean square error as the loss function and record it as Π(), the formula is as follows:

[0062]

[0063] In step 103, it is determined whether the reconstruction loss has reached convergence. If so, the iteration stops and the deep autoencoder training is completed. Otherwise, the decoder and encoder parameters are updated based on the reconstruction loss, and the process returns to step 101 to re-perform encoding mapping and decoding reconstruction with the goal of minimizing the reconstruction loss. The iteration continues until the loss value converges to a stable state.

[0064] A deep autoencoder model, fully trained on benign traffic, can fit the deep patterns of benign traffic and reconstruct the input with minimal error. If the input is a benign traffic sample, the reconstruction error output by the autoencoder in step 1 will be relatively small; if the input is an abnormal traffic sample, the reconstruction error output by the autoencoder in step 1 will be relatively large. In other words, the reconstruction error calculated by the autoencoder in step 1 for the input sample represents the abnormality index of the input sample.

[0065] Step 2: Based on the reconstruction error provided by the autoencoder, the original feature distribution is collected and the sub-distribution is delineated from the overall benign traffic, such as Figure 3 shown.

[0066] Benign traffic as a whole follows a complex global distribution. Extracting rules directly from this global distribution is undoubtedly difficult. Dividing this global distribution into smaller analysis objects is clearly more conducive to rule extraction. The global distribution can be viewed as a collection of multiple, uncoupled sub-distributions, each with a smaller scope and more accessible for rule extraction. The rules of multiple sub-distributions can be combined to form a whitelist rule base capable of describing the global distribution of benign traffic. Therefore, accurately delineating sub-distributions is a necessary step in the present invention, reducing the difficulty of rule extraction and refining the granularity of the rule base.

[0067] In order to delineate sub-distributions from the overall benign traffic, the present invention adopts a clustering method that combines reconstruction error and original features. Benign traffic samples from the same sub-distribution have similar reconstruction errors and similar feature distributions, and the deep autoencoder trained in step 1 will calculate similar reconstruction errors for benign traffic samples from the same sub-distribution. The reconstruction loss is Use reconstruction loss to transform the original feature vector Expand to As shown in formula (2)

[0068]

[0069] After using the reconstruction error to expand the features of benign traffic from m dimensions to m+1 dimensions, the new feature space is recorded as In the expanded feature space, clusters are divided according to the clustering algorithm model Cluster_Model as shown in formulas (3) and (4). Each cluster Represents a sub-distribution of benign traffic.

[0070]

[0071] In this method, reconstruction error is introduced to expand the original features from K dimensions to K+1 dimensions. Clustering is then performed based on the expanded feature space to define subdistributions. Benign traffic samples from the same subdistribution have similar reconstruction errors, making reconstruction error an important factor in defining subdistributions.

[0072] Step 3: Extract the boundary rules of sub-distribution based on the gradient ascent method, such as Figure 4 shown.

[0073] Through the operation in step 2, the entire benign traffic is divided into multiple different sub-distributions. In this step, boundary rules are extracted for each sub-distribution. During actual detection, the test sample is determined to belong to a specific sub-distribution by rule matching. If the test sample does not belong to any sub-distribution of benign traffic, it is determined to be an abnormal traffic sample. The extraction of sub-distribution boundary rules is the basis for the subsequent organization of the overall rule base. The specific operations are as follows:

[0074] First, a minimum hypercube is generated for each cluster, denoted as Mini-Hyper-Cube, and the maximum reconstruction loss of benign samples in each cluster is denoted as thres. A certain number of sample instances are selected as Explorers in the shallow neighborhood inside each hyperplane of the Mini-Hyper-Cube. The Explorer is moved along the direction of the reconstruction loss gradient until its reconstruction loss is greater than thres, and the feature corresponding to the gradient direction at this time is recorded. The value of and Repeat the above process on each hyperplane to obtain the boundary threshold of each feature and combine them into the boundary rule of the current cluster, as shown in (5).

[0075]

[0076] Step 4: Merge rules, such as Figure 5 shown.

[0077] Merge the boundary rules of each sub-distribution extracted in step 3 to form a complete whitelist rule base that describes the global distribution of benign traffic.

[0078] The results show that:

[0079] Based on the whitelist rule base established by the present invention, anomaly detection is performed on real intranet traffic. At the same time, anomaly detection is performed on real intranet traffic using density-based method, partition-based method and empirical cumulative distribution-based method respectively. The results are as follows: Figure 6 shown. Figure 6 In , the horizontal axis represents the observed abnormal traffic detection method, and the vertical axis represents the true positive rate of the observed method in detecting anomalies in real intranet traffic.

[0080] from Figure 6 As can be seen from the results, the proposed method achieves a higher true positive rate than existing typical methods, reaching over 90%. This demonstrates that the proposed method not only provides good interpretability in the form of an IF-THEN structure, but also retains the deep autoencoder's precise anomaly detection capabilities, achieving a balance between detection accuracy and interpretability.

Claims

1. A method for detecting explainable abnormal traffic in a closed internal network, characterized in that: The following steps are involved: Step 1: Train a deep autoencoder using several benign traffic samples to learn deep patterns in benign traffic. The deep autoencoder consists of two parts: encoder and decoder. The training process is: Step 101: Map the original benign traffic sample input vector into a low-dimensional vector through the encoder and output it to the decoder; Step 102: The decoder reconstructs the low-dimensional vector into the original input vector and calculates the reconstruction loss between the reconstructed vector and the original vector. The original training dataset is denoted as Initialize the autoencoder model M, record the model related parameters as θ, use the mean square error as the loss function and record it as Π(), the formula is as follows: Step 103: determine whether the reconstruction loss has reached convergence. If so, the iteration stops and the deep autoencoder training is completed; Otherwise, update the decoder and encoder parameters based on the reconstruction loss, return to step 101, re-perform encoding mapping and decoding reconstruction, and continue iteration; Step 2: Benign traffic samples After the trained deep autoencoder is reconstructed and expanded, the expanded feature space is clustered and divided into clusters, each cluster representing a sub-distribution; Step 3: Based on the principle of gradient ascent of reconstruction loss, boundary rules are extracted for each sub-distribution; The specific process of boundary rule extraction is as follows: Step 301: traverse all sub-distributions and generate a minimum hypercube for each cluster, denoted as Mini-Hyper-Cube. At the same time, the maximum reconstruction loss of benign samples in each cluster is recorded as thres. Step 302: Select several sample instances as Explorers in the shallow neighborhood inside each hyperplane of each Mini-Hyper-Cube; Step 303: Calculate the direction of the gradient increase of the reconstruction loss in each cluster and move the Explorer along this direction until the reconstruction loss is greater than thres. Record the feature corresponding to the gradient direction at this time. The value of and That is a boundary threshold of the feature; Step 304: Repeat the above process on each hyperplane to obtain the boundary threshold of each feature, and combine them into the boundary rule of the current cluster, which is expressed as: Among them, 1, 2, ..., k are hyperplane numbers; Step 4: Merge the boundary rules of each sub-distribution to form a complete whitelist rule base that describes the global distribution of benign traffic; Step 5: Detect unknown traffic data in a closed internal network using a whitelist rule library to obtain abnormal traffic conditions.

2. The method for detecting explainable abnormal traffic in a closed internal network according to claim 1, characterized in that: After reconstruction and expansion, the benign traffic samples are divided into clusters as follows: First, benign traffic samples After the trained deep autoencoder, the reconstruction loss is Use reconstruction loss to transform the original feature vector Expand to The formula is: Then, after the features of benign traffic are expanded from m dimension to m+1 dimension, the new feature space is recorded as Finally, the new feature space is divided into clusters using the clustering algorithm model Cluster_Model, and the cluster set is obtained as follows: Each cluster Represents a sub-distribution of benign traffic; The formula for dividing clusters is:

3. The method for detecting explainable abnormal traffic in a closed internal network according to claim 1, characterized in that: The merging process of each sub-distribution boundary rule is: Read the boundary rules of a sub-distribution, add them to the overall rule base, and perform redundancy removal on the overall rule base until the boundary rules of all sub-distributions are traversed.

4. The method for detecting explainable abnormal traffic in a closed internal network according to claim 3, characterized in that: The redundancy removal operation is as follows: if in the overall rule base, there is a boundary rule of a sub-distribution a that is completely contained in the boundary rule of another sub-distribution b, then the boundary rule of sub-distribution b is removed and the boundary rule of sub-distribution a is retained.

5. The method for detecting explainable abnormal traffic in a closed internal network according to claim 1, characterized in that: The whitelist rule base established by the present invention performs anomaly detection on real intranet traffic, and the true positive rate reaches more than 90%.

Images