Network alarm data fusion method based on improved hierarchical clustering

Abstract

The application relates to the technical field of alarm fusion, in particular to a network alarm data fusion method based on an improved hierarchical clustering, which comprises the following steps: collecting original alarm information and performing format processing through a data preprocessing module; dividing time windows according to time characteristics of an ordered alarm data set; using an improved hierarchical clustering method in a fusion module to aggregate alarms with high similarity together; and finally generating a simplified advanced alarm according to the redundancy and correlation between alarms. The application fuses alarms based on the hierarchical clustering idea, considers the diversity of network data, improves the applicability of the method, solves the information loss problem by using a hybrid similarity distance measurement method, and can effectively remove redundant data in network alarms.

Description

Technical Field

[0001] This invention relates to the field of alarm fusion technology, and specifically to a network alarm data fusion method based on improved hierarchical clustering. Background Technology

[0002] The rapid development of information technology has brought about many security risks, forcing network systems to deploy a variety of network security devices and improve the sensitivity of threat detection, resulting in a large number of redundant alarms in the system.

[0003] Traditional alarm fusion methods typically rely on pre-built knowledge bases, membership functions, and other rules, which require strong expert experience and have certain limitations. Summary of the Invention

[0004] The purpose of this invention is to provide a network alarm data fusion method based on improved hierarchical clustering. It uses flexible hierarchical clustering to avoid parameter selection and employs a hybrid distance metric in the hierarchical clustering process to reduce the loss of useful information.

[0005] To achieve the above objectives, this invention provides a network alarm data fusion method based on improved hierarchical clustering, comprising the following steps:

[0006] The raw alarm data is preprocessed using the preprocessing module.

[0007] An improved hierarchical clustering method was used to aggregate alarm data and obtain alarm clusters.

[0008] The fusion module is used to merge similar alarms of the same cluster to obtain concise and effective alarm information.

[0009] The preprocessing module is used to complete the tasks of data collection and formatting, and the fusion module is used to complete the tasks of data aggregation and fusion.

[0010] The process of preprocessing the raw alarm data using the preprocessing module includes the following steps:

[0011] Fill in missing values ​​based on redundancy between data attributes;

[0012] Standardize continuous data and perform feature encoding on discrete data;

[0013] The alarms are sorted according to their occurrence time to obtain a preprocessed data set.

[0014] The process of aggregating alarm data using an improved hierarchical clustering method includes the following steps:

[0015] Each alarm sample is selected as a separate initial cluster, and the initial similarity between the initial clusters is calculated;

[0016] Select the two most similar clusters to merge to complete one round of iteration;

[0017] Iterative updates are performed using an improved distance metric.

[0018] Specifically, the iterative update process involves using a hybrid similarity distance metric to update the similarity between related clusters.

[0019] The same alarm cluster refers to a set of alarms that contain similar and redundant alarms.

[0020] This invention provides a network alarm data fusion method based on improved hierarchical clustering. First, a data preprocessing module collects and formats raw alarm information. Then, the ordered alarm data set is divided into time windows according to time characteristics. Next, in the fusion module, an improved hierarchical clustering method is used to aggregate alarms with high similarity. Finally, based on the redundancy and correlation between alarms, a simplified high-level alarm is generated. This invention uses hierarchical clustering for alarm fusion, considers the diversity of network data, improves the applicability of the method, and uses a hybrid similarity distance metric to solve the information loss problem, effectively removing redundant data from network alarms. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a network alarm data fusion method based on improved hierarchical clustering according to the present invention.

[0023] Figure 2 This is a schematic diagram of the structure of each module in the method model of this invention.

[0024] Figure 3 This is a comparison chart of the similarity, fusion rate, and accuracy between the present invention and the random forest model in a specific embodiment of the present invention.

[0025] Figure 4 This is a comparison chart of experimental results between hierarchical clustering with mixed similarity measures and hierarchical clustering with single similarity measures in a specific embodiment of the present invention.

[0026] Figure 5 This is a comparison chart of the experimental results of the present invention with those of GMM and K-means methods in a specific embodiment of the present invention.

[0027] Figure 6 This is a comparison chart of alarm aggregation rate and accuracy on the DARPA dataset for a specific embodiment of the present invention. Detailed Implementation

[0028] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0029] Please see Figure 1 This invention proposes a network alarm data fusion method based on improved hierarchical clustering, comprising the following steps:

[0030] S1: Use the preprocessing module to preprocess the raw alarm data;

[0031] S2: Use an improved hierarchical clustering method to aggregate alarm data and obtain alarm clusters;

[0032] S3: Use the fusion module to merge similar alarms of the same cluster to obtain concise and effective alarm information.

[0033] Please see Figure 2 The model of the network alarm data fusion method based on improved hierarchical clustering of the present invention is shown in the figure. Specifically, it obtains simplified data after processing by the original alarm input data preprocessing module and the fusion module.

[0034] (1) Data Preprocessing Module: This module completes the tasks of data collection and formatting. First, it fills in missing values ​​based on the redundancy between data attributes; then it distinguishes between continuous and discrete data, performs standardization on continuous data, and performs feature encoding or relation mapping operations on discrete data; finally, it sorts the data according to alarm time to obtain the preprocessed data set.

[0035] (2) Fusion Module: This module completes the tasks of data aggregation and fusion. First, the ordered data set is divided into time windows according to time characteristics; then, the hierarchical clustering method of this invention is used to aggregate alarms with high similarity together; finally, based on the redundancy and correlation between alarms, a simplified high-level alarm is generated by fusion.

[0036] The effectiveness of the method of the present invention is verified below by combining the dataset and specific implementation steps. The distribution of UNSW-NB15 alarm data is shown in the table below:

[0037]

[0038]

[0039] Furthermore, this invention also verified the alarm aggregation efficiency of the proposed method on the DARPA2000 dataset. By replaying the tcpdump traffic data of the LLDOS1.0 scenario in this dataset, and extracting 30,459 alarm samples based on Snort, each sample has 13-dimensional features, including 10-dimensional nominal attributes and 3-dimensional numerical attributes. The alarm distribution is shown in the table below:

[0040] Alarm Categories Sample size GenericICMPevent 875 RPCsadmindquery 352 Networktrojan 553 Attemptedadministratorprivilegegain 27 Attemptedinformationleak 19 Attempteduserprivilegegain 10 Detectionofadenialofserviceattack 9602 Detectionofanon-standardprotocolorevent 3542 TCPSYN floodattack detected 12593 Deletedinfotelnetaccess 112 DeletedICMPredirecthost 2219 Other 555

[0041] Based on the above dataset, the specific process of alarm fusion based on improved hierarchical clustering is as follows:

[0042] 1. Data Preprocessing

[0043] Step 1: Convert the data to a uniform format, fill in missing values ​​in the dataset by utilizing the redundancy between data, and remove attributes with unique values;

[0044] Step 2: Standardize the continuous data and perform feature encoding on the discrete data;

[0045] Step 3: Sort the alarms according to the time they were generated.

[0046] 2. Hierarchical clustering

[0047] Step 1: Sample Initialization. In order to perform hierarchical clustering, the samples must first be initialized. Each alarm sample is treated as a separate initial cluster, and the initial similarity between these initial clusters must also be calculated.

[0048] Step 2: Merge clusters. Based on the idea of ​​hierarchical clustering, each iteration selects the two most similar clusters to merge. In this invention, the two clusters with the highest similarity are selected each time.

[0049] Step 3: Iterative Update. Cluster merging alters the relationships between alarm clusters, so the hybrid similarity distance metric method described in this invention must be used to update the similarity between related clusters.

[0050] The following is a description of the hybrid similarity distance metric method used in this invention.

[0051] For numerical attributes in alarm data, the Error Sum of Squares (ESS) is used to measure the dispersion within the set. The error sum of squares is essentially related to variance and primarily reflects the deviation of the current sample from the expected sample. For nominal attributes in the network system, since nominal data has no numerical meaning, this invention uses JS divergence to represent the difference between two sets.

[0052] The calculation method for the ESS of a sample set:

[0053]

[0054] Where ESS represents the sum of squared deviations of a sample set, a i Let be the i-th alarm sample in the set, and n represent the number of samples in the set. However, network alarm data is multi-dimensional, and it is usually necessary to use Euclidean distance to map the multi-dimensional attribute vector of the sample to a distance variable.

[0055] Combining the idea of ​​hierarchical clustering, each time the two sets with the smallest change in dispersion are selected for merging, that is, the pair of sets with the smallest ESS increment, and I is used. ess This represents the ESS increment after merging two clusters:

[0056]

[0057] Among them, C a C b Let C represent the two alarm sets before merging. a ∪C b This represents the merged cluster. Combining the idea of ​​hierarchical clustering, each time the two sets with the smallest change in dispersion are selected for merging, i.e., the pair of sets with the smallest ESS increment.

[0058] JS divergence is calculated based on KL divergence, which measures the degree of difference between two separate probability distributions. KL divergence is appropriate for representing the difference between two intrusion methods. The KL divergence is expressed as:

[0059]

[0060] Here, s(a) represents the domain of a, and p(a) and q(a) represent the probabilities of a appearing in the corresponding sets. The smaller the value of KL(p||q), the more similar the probability distributions p and q are, meaning the two sets are more similar. However, KL entropy does not satisfy symmetry, which is inconvenient in clustering algorithms. Therefore, JS divergence is introduced to calculate similarity.

[0061] How to calculate the JS divergence between sample sets:

[0062]

[0063] Here, (p+q) / 2 represents the superposition of two probability distributions, forming a new distribution. A smaller JS(p||q) indicates a more similar probability distribution. Therefore, during clustering, the two clusters with the smallest JS divergence are merged each time to minimize the information difference after merging. Alarm data may also have multi-dimensional nominal attributes. This invention uses the following method to calculate the JS divergence of these multi-dimensional attributes:

[0064]

[0065] Among them, H js (C a C b ) is a set C a With set C b JS divergence between them, JS j (C a ||C b ) represents the JS divergence of the two sets on attribute j, and m' represents the number of nominal attributes of the network alarm data.

[0066] To more accurately measure the similarity between network alarm data, this invention designs a similarity measurement method that combines the advantages of the two methods mentioned above:

[0067] M(C a C b )=αI ess (C a C b )+(1-α)H js (C a C b (1≤a,b≤n)

[0068] Among them, M(C a C b ) represents the computational cluster C a and C b Mixed similarity between them, I ess and H js These are the ESS increments and JS divergences for the two cluster types, respectively. The parameter α controls the relative importance between numerical and nominal attributes; setting α according to the characteristics of different data types improves the robustness of the algorithm.

[0069] After updating the inter-cluster similarity according to the above formula, select min(M(C)) before the next round of merging. a C b C a C b Merge the clusters and iterate through steps 2 and 3 of the hierarchical clustering process until the clustering state satisfies |C|=1, forming a complete hierarchical clustering tree structure diagram.

[0070] 3. Intra-cluster fusion

[0071] After grouping similar alarms together using hierarchical clustering, this invention introduces the downsampling concept from machine learning to reduce redundant data. A random downsampling method is used to select samples. First, a downsampling window is set based on the clustering results. The alarm data within each window are highly similar. Samples are randomly selected and retained within each window, thus removing redundancy while avoiding information loss.

[0072] Experimental results:

[0073] The UNSW-NB15 alarm data, after preprocessing, hierarchical clustering, and fusion, outputs simplified data. This invention uses the fusion rate δ = (n-n') / n to evaluate the efficiency of the fusion method. Generally, the similarity threshold directly affects the alarm fusion rate; higher similarity means stricter fusion conditions and a lower fusion rate; lower similarity means more data can be fused, and a higher fusion rate. If δ represents the fusion rate and ε represents the similarity, generally speaking, δ and ε have an inverse proportional relationship. To demonstrate that the method of this invention can effectively remove redundant data, the original alarm data is divided into a training set and a test set. Simplified training set data under different ε values ​​are selected, and the same test set is used to verify and compare the classification accuracy.

[0074] See Figure 3 This graph effectively illustrates the relationship between similarity, fusion rate, and accuracy. The horizontal axis ε represents the similarity threshold, while the vertical axis, a dimensionless percentage, reflects accuracy and fusion rate. It can be seen that the similarity threshold is indeed inversely proportional to the fusion rate. As the similarity threshold increases, the clustering conditions become increasingly stringent, leading to a decrease in the fusion rate. Regarding the fusion rate δ, when ε ≤ 0.2, δ is high, but the classification accuracy is low, indicating that some information is lost during the aggregation process. When ε = 0.5, δ is relatively stable, with a fusion rate of approximately 95%. If ε continues to increase, the aggregation conditions become very stringent, and the fusion rate drops sharply.

[0075] Furthermore, in order to measure whether information is lost after alarm fusion, this invention trains a Random Forest (RF) classification model using simplified data, tests the same test set, and compares the classification performance of fused data using different methods.

[0076] To demonstrate the effectiveness of the alarm fusion method of the present invention, the experimental results will be analyzed from the following aspects:

[0077] 1. Compare the impact of different clustering algorithms on alarm fusion efficiency

[0078] To verify the effectiveness of hierarchical clustering using a hybrid similarity metric, experiments were conducted to compare its performance with other clustering algorithms. For example... Figure 4The experimental results of hierarchical clustering using a mixed similarity metric and hierarchical clustering using a single similarity metric are compared. The experiment used multiple sets of simplified alarm data with the same fusion rate to compare classification accuracy. It can be seen that the accuracy of the method in this invention is generally higher than that of the single similarity metric method. This is because the original method uses a single similarity metric to measure the differences between samples, leading to information loss. The method in this invention, however, treats nominal data and numerical data differently, making the clustering results more reasonable and accurate.

[0079] This invention also compares the classification efficiency of other clustering methods. Gaussian mixture models (GMMs) can discover data distributions of arbitrary shapes by observing the probability distribution of the data. K-means, as a classic clustering algorithm, is widely used in data preprocessing in various fields. Please refer to... Figure 5 , Figure 5 The experimental results of the proposed method, GMM, and K-means are shown, all using simplified data selected when the fusion rate tends to stabilize to train the classification model. It can be seen that the fusion rates of the three methods are not significantly different, indicating that there is indeed about 90% redundant data in the dataset. However, the accuracy and false positive rate obtained by the proposed method are better than the other two methods, indicating that the data selected by the proposed method is more effective.

[0080] 2. Compare the efficiency of different alarm fusion methods

[0081] To further verify the effectiveness of the fusion method of this invention, a comparison was made with fusion methods based on attribute similarity and rough sets, as shown in the table below. The table shows that, without losing too much information, the method of this invention achieves the highest fusion rate and removes the most redundant data.

[0082] Alarm fusion method Fusion rate accuracy False alarm rate Method of the present invention 0.9693 0.8846 0.0334 Attribute similarity-based methods 0.9406 0.8790 0.037 Rough set-based methods 0.9397 0.8676 0.0377

[0083] 3. Fusion efficiency across different datasets

[0084] For experimental results on the DARPA2000 dataset, please refer to [link / reference]. Figure 6 As can be seen, the alarm fusion rate and accuracy also follow this pattern: the alarm fusion rate decreases as the similarity threshold increases, while the accuracy increases. Before using the fusion method described in this chapter, the calculated accuracy was around 0.84, while the alarm fusion rate calculated using this method was mostly around 0.87. This improvement in accuracy indicates that the simplified data is more precise. The alarm fusion rate was mostly around 88%, meaning that using only 12% of the alarm data could achieve the effect of all the original data, demonstrating that the method of this invention can effectively remove redundant data.

[0085] In summary, the network alarm data fusion method based on improved hierarchical clustering proposed in this invention has the following advantages:

[0086] 1. As an unsupervised learning method, clustering algorithms can process data directly with almost no prior preparation. This invention uses the idea of ​​hierarchical clustering for alarm fusion, taking into account the diversity of network data and improving the applicability of the method.

[0087] 2. In response to the problems of clustering algorithms and the characteristics of network data, a hybrid JS divergence and ESS increment are used to replace the single similarity metric to solve the problem of information loss. This can effectively remove redundant data in network alarms and highlight the role of key information.

[0088] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A network alarm data fusion method based on improved hierarchical clustering, characterized in that, The method comprises the following steps: preprocessing the original alarm data using a preprocessing module; after preprocessing, first dividing time windows for the ordered data set according to time characteristics; then using a hierarchical clustering method to aggregate alarms with high similarity together; using the hierarchical clustering method to aggregate alarm data to obtain alarm clusters; improving the process of using the hierarchical clustering method to aggregate alarm data, comprising the following steps: selecting each alarm sample as a separate initial cluster, and calculating the initial similarity between the initial clusters; selecting two most similar clusters for merging to complete a round of iteration; iteratively updating using an improved distance measurement method; wherein the process of iterative updating is to update the similarity between related clusters using a hybrid similarity distance measurement method; the hybrid similarity distance measurement method is obtained by combining the sum of squared deviations and JS divergence, and the expression is: M(C a ,C b ) = αI ess (C a ,C b )+(1-α)H js (C a ,C b )(1≤a,b≤n) where M(C a ,C b ) represents the hybrid similarity between the calculated clusters C a and C b , I ess and H js are the increments of the sum of squared deviations and JS divergence of the two clusters, respectively, and the parameter a is set according to the characteristics of different types of data; After updating the inter-cluster similarity, the C a ,C b with min(M(C a ,C b )) are selected to merge before the next round of merging, and the hierarchical clustering is iterated until the clustering state satisfies |C| = 1, forming a complete hierarchical clustering tree structure diagram; using a fusion module to fuse similar alarms in the same cluster to obtain concise and effective alarm information; the preprocessing module is used to complete the tasks of data collection and formatting, and the fusion module is used to complete the tasks of data aggregation and fusion; in the cluster fusion process, a random down-sampling method is used to reduce redundant data, and representative samples are retained in the cluster.

2. The network alarm data fusion method based on improved hierarchical clustering according to claim 1, wherein the process of preprocessing the original alarm data using a preprocessing module comprises the following steps: filling in missing values according to the redundancy between data attributes; standardizing continuous data and feature encoding discrete data; sorting alarms according to alarm generation time to obtain a preprocessed data set.

3. The network alarm data fusion method based on improved hierarchical clustering according to claim 1, wherein the same alarm cluster is an alarm set containing similar alarms and mutual redundancy.

Images