A Canopy protocol identification method based on maximum close distance

Abstract

The invention belongs to the technical field of computer machine learning, and in particular to a Canopy protocol identification method based on maximum close distance. Aiming at the problems of poor clustering accuracy and stability, inaccurate algorithm parameter selection and inability to determine the center point, the following scheme is proposed, comprising the following steps: S1: performing data preprocessing on a protocol data set to be classified and identified, and using PCA for dimensionality reduction; S2: extracting a small number of labeled protocols to calculate the average value of the average distance in each subset, and treating it as the distance threshold T2 in the Canopy algorithm; S3: selecting the sample point with the largest density value as the first clustering center for the first clustering, performing Canopy iteration, and then calculating the close distance value r of the remaining sample points in the data set; The purpose of the invention is to improve the clustering accuracy and stability of the K-means algorithm, accurately select algorithm parameters, determine the center point, improve the clustering effect and improve the accuracy of protocol identification, so as to adapt to the environment of non-uniform network protocols.

Description

Technical Field

[0001] The invention relates to relevant algorithms of protocol identification and semi-supervised learning, belonging to the technical field of computer machine learning, and in particular to a Canopy protocol identification method based on maximum close distance. Background Art

[0002] In the field of network protocol identification, machine learning-based methods use the external statistical features of network flows to identify network applications, getting rid of the dependence on protocol content. Clustering algorithms are classic machine learning algorithms and have always attracted the attention of many scholars. Clustering algorithms belong to unsupervised learning. Their input is an unlabeled data set. Clustering is divided into several classes based on the distance or similarity of the data itself. The goal is to minimize the distance within the class and maximize the distance between classes. Through clustering, the judgment of known and unknown classes is achieved. It has a certain ability to identify unknown protocols, so it is often used for identification. The literature (Li Jintao, Ai Ping, Yue Zhaoxin, Ma Mengmeng, Bian Shizhe. Improvement of K-means clustering algorithm [J]. Foreign Electronic Measurement Technology, 2017, 36(06): 9-13+21. DOI: 10.19652 / j.cnki.femt.2017.06.002.) proposed a density-based K-means improved algorithm, which selects the point with the largest density in the data set as the initial center point, and selects the point farthest from this point as the second center point based on this point, and then finds the point farthest from these two points in the remaining points as the third point, and so on until k initial center points are found. This method is better than the traditional K-means method in terms of accuracy and stability, but has weak noise resistance. The literature (Huang Xiaohui, Wang Cheng, Xiong Liyan, Zeng Hui. A weighted k-means clustering method integrating intra-cluster and inter-cluster distances [J]. Journal of Computers, 2019, 42(12): 2836-2848. Mao Dianhui.) proposed a weighted K-means method integrating intra-cluster and inter-cluster distances to solve the high-dimensional data clustering problem, while making each cluster center not only represent the data objects in this cluster, but also as far away as possible from the data objects that do not belong to this cluster. This method can achieve better clustering performance in most cases, but it is not suitable for situations involving various complex cluster structures. The literature (Improved Canopy-Kmeans Algorithm Based on MapReduce [J]. Computer Engineering and Applications, 2012, 48(27): 22-26+68.) proposed a Canopy-Kmeans algorithm based on the maximum-minimization principle. It uses the boundary recognition idea and introduces the depth index to determine the T1 value and the optimal clustering center, while ignoring the assignment of T2. However, it does not process the noise and its operating efficiency needs to be improved.The literature (Lin GZ, Xin Y, Niu XX, et al. Network traffic classification based on semi-supervised clustering [J] The Journal of China Universities of Posts and Telecommunications, December 2010. 17 (Suppl. 2): 84-88.) optimized the selection of the initial clustering center and determined the initial clustering center using feature variance. However, the "simple majority" principle was used in the mapping relationship between clusters and application types, resulting in a poor recognition rate of the algorithm in an environment with uneven network protocols.

[0003] In summary, although relevant scholars have achieved many results in clustering algorithms, there are limitations and shortcomings in the K value and initial cluster center points of the K-means clustering algorithm. With the increase in the complexity of experimental data, the improvement of the clustering algorithm is still a long-term optimization problem. Summary of the invention

[0004] In order to improve the clustering accuracy and stability of the K-means algorithm and solve the problems of inaccurate algorithm parameter selection and center point determination, the present invention proposes a Canopy protocol identification method based on maximum close distance to adapt to the environment of non-uniform network protocols.

[0005] To achieve the above object, the present invention adopts a Canopy protocol identification method based on maximum close distance, comprising the following steps:

[0006] S1: preprocess the protocol data set to be identified and use PCA to reduce the dimension;

[0007] S2: Extract a small number of labeled protocols to calculate the average distance avg in each subset i The average value is taken as the distance threshold T2 in the Canopy algorithm;

[0008] S3: For the first clustering, the sample point with the largest density value is selected as the first cluster center, and Canopy is iterated once. Then, the density distance r of the remaining sample points in the data set is calculated, and the point with the maximum value is found as the next center point for iteration until the data set is empty.

[0009] S4: The sample average value calculated in the Canopy clustering data subset is used as its cluster center, thereby obtaining the number of clusters k and the corresponding initial cluster center, which are then brought into K-means for clustering to obtain the recognition result.

[0010] Preferably, in S1, the data preprocessing process includes digitization, standardization and normalization, and standardization is:

[0011]

[0012] in is the average value of each attribute, S k is the mean absolute error, Z ik is a standardized measure.

[0013] Normalization makes the contribution of each dimension equal, that is:

[0014]

[0015] Preferably, in S2, the average distance avg within each marker subset is i The average value of is defined as given p sets of labeled protocols, in the i-th (i∈p) set, there are m protocol samples, then the average distance avg of the samples in the set i Denoted as:

[0016]

[0017] Where d(i x ,i y ) represents the Euclidean distance between samples x and y in set i.

[0018] And according to the p sets of labels avg i The value of sets the Canopy distance threshold T2, that is:

[0019]

[0020] Preferably, S3 specifically includes the following steps:

[0021] A1: Calculate the density of sample elements, take the point with the maximum density as the first center point, and perform Canopy clustering;

[0022] A2: Calculate the distance value determined by the sample element density, the distance between samples, and the average distance of elements within the sample density range, identify the outliers and find the maximum value as the center point of the next iteration and perform Canopy clustering;

[0023] A3: If the data set is not empty, jump to step A2; if it is empty, jump to step A4;

[0024] A4: Output the Canopy clustering subsets and the number of subsets.

[0025] Preferably, in A3, the sample spacing value r is expressed as the sample element density ρ i The distance between samples b iThe product is divided by the average distance a of the elements within the sample density range i , that is: r = ρ i b i / a i , taking its maximum value is the maximum close distance method. i The calculation formula is:

[0026]

[0027] in

[0028] where d ij Represents the Euclidean distance between sample elements j and i.

[0029] The distance b between samples i represents the distance between sample i in the data set and another sample j with a larger local density than it. If the density of sample i is the global maximum, then b i It is defined as the distance between the farthest elements from the element. If there is a sample j with a higher density than sample i, then b i It is defined as the minimum distance between samples with higher density than sample i, and the calculation formula is:

[0030]

[0031] where ρ j Indicates that element j has a higher density than element i, d ij Represents the distance between sample elements j and i;

[0032] The average distance a of elements within the sample density range i The calculation formula is:

[0033]

[0034] where ρ i is the number of samples in the range, d(i x ,i y ) represents the Euclidean distance between samples x and y in the range.

[0035] Preferably, S4 specifically includes the following steps:

[0036] B1: Calculate the sample average in each subset as its initial cluster center;

[0037] B2: In the K-means algorithm, the Euclidean distance from each sample to the initial cluster center is calculated, and the sample is classified into the category of the cluster center closest to it according to the minimum principle;

[0038] B3: Calculate the average distance of sample elements in a category and update it as the new cluster center of the category:

[0039] B4: Compare the new cluster center with the previous cluster center to see if it has changed or if the error is within the requirements. If there is no change, the maximum number of iterations or the allowable error is reached, then output the clustering result, otherwise go to step B2.

[0040] The Canopy protocol recognition method based on the maximum close distance of the present invention has strong noise resistance. Since outliers and abnormal points have low density and discrete characteristics, when calculating the sample close distance value, when the density value is 1 or much smaller than the normal value, and the distance b between samples is i However, if the value is greater than the normal value, the sample can be regarded as an outlier or an isolated point generated in the clustering process and is removed before the clustering.

[0041] In contrast to related studies, the present invention proposes a Canopy semi-supervised clustering algorithm using close distance values ​​to address the defect that the distance parameters of the existing Canopy algorithm are difficult to determine. The center point of the Canopy algorithm is selected by the maximum close distance method, which solves the randomness of the initial center point selection and the problem of decreased algorithm accuracy caused by local optimality during the implementation process. At the same time, the traditional Canopy random selection of threshold parameters and related studies that determine threshold parameters by global solution will have a greater impact on the clustering results. The algorithm threshold is determined by experimental samples based on the close distance value method proposed in the present invention, and the global solution is converted into a local solution without manual setting. This can reduce the instability of the threshold parameters on the clustering results, while improving the accuracy of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the algorithm flow chart of the present invention;

[0043] Figure 2 This is a schematic diagram of the data standardization and normalization feature changes of the present invention;

[0044] Figure 3 It is a schematic diagram of the maximum close distance method of the present invention;

[0045] Figure 4 A two-dimensional schematic diagram of determining T2 using a local sample according to the present invention;

[0046] Figure 5 It is a schematic diagram of comparing the accuracy of the algorithms of the present invention;

[0047] Figure 6 It is a schematic diagram of comparing the protocol recall rates of the present invention;

[0048] Figure 7 Schematic diagram of the sum of squared errors of the present invention. DETAILED DESCRIPTION

[0049] In order to make the technical solutions and advantages of the embodiments of the present invention more clear, the present invention is further described below with reference to the accompanying drawings.

[0050] Reference Figure 1 , a Canopy protocol identification method based on maximum close distance, comprising the following steps:

[0051] S1: preprocess the protocol data set to be identified and use PCA to reduce the dimension;

[0052] S2: Extract a small number of labeled protocols to calculate the average distance avg in each subset i The average value is taken as the distance threshold T2 in the Canopy algorithm;

[0053] S3: For the first clustering, the sample point with the largest density value is selected as the first cluster center, and Canopy is iterated once. Then, the density distance r of the remaining sample points in the data set is calculated, and the point with the maximum value is found as the next center point for iteration until the data set is empty.

[0054] S4: The sample average value calculated in the Canopy clustering data subset is used as its cluster center, thereby obtaining the number of clusters k and the corresponding initial cluster center, which are then brought into K-means for classification again to obtain the recognition result.

[0055] In this embodiment, the data preprocessing process includes digitization, standardization and normalization. Since the distance is used to judge the sample similarity in the clustering process, the smaller the unit of the feature attribute, the greater the change in the value, which will lead to inaccurate calculation of the distance of the dimension and have a greater impact on the final clustering, so it is necessary to standardize the data features, that is:

[0056]

[0057] in is the average value of each attribute, S k is the mean absolute error, Z ik is a standardized measure.

[0058] When calculating the distance, if the range of feature vector values ​​is too large, it will have a greater impact on the final distance and affect the distance accuracy. Therefore, it is necessary to normalize all data so that the contribution of each dimension is the same, that is:

[0059]

[0060] Reference Figure 2 ,Standardization and normalization change the data features, limiting the range of the same features to the range of [0, 1], and do not change the feature shape between different samples.

[0061] In this embodiment, the steps for calculating the threshold parameter T2 of Canopy are as follows: given p sets of labeled protocols, in the i-th (i∈p) set, there are m protocol samples, then the average distance avg of the samples in the set i Denoted as:

[0062]

[0063] Where d(i x ,i y ) represents the Euclidean distance between samples x and y in set i.

[0064] According to the p sets of labels avg i The value of sets the Canopy distance threshold T2, that is:

[0065]

[0066] Reference Figure 3 , under two-dimensional intuitive sample observation, use T2 to perform an iterative effect.

[0067] On the basis of the above, S3 specifically includes the following steps:

[0068] A1: Calculate the density of sample elements, take the point with the maximum density as the first center point, and perform Canopy clustering;

[0069] A2: Calculate the distance value determined by the sample element density, the distance between samples, and the average distance of elements within the sample density range, identify the outliers and find the maximum value as the center point of the next iteration and perform Canopy clustering;

[0070] A3: If the data set is not empty, jump to step A2; if it is empty, jump to step A4;

[0071] A4: Output the Canopy clustering subsets and the number of subsets.

[0072] Reference Figure 4 , the close distance value r is expressed as the sample element density ρ i The distance between samples b i The product is divided by the average distance a of the elements within the sample density range i , that is: r = ρ i b i / a i , taking its maximum value is the maximum close distance method, for the sample element density ρ i The calculation formula is:

[0073]

[0074] in d ijRepresents the Euclidean distance between sample elements j and i.

[0075] For the distance b between samples i The calculation formula is:

[0076]

[0077] where ρ j Indicates that element j has a higher density than element i, d ij Represents the distance between sample elements j and i.

[0078] For the average distance a of elements within the sample density range i The calculation formula is:

[0079]

[0080] where ρ i is the number of samples in the range, d(i x ,i y ) represents the Euclidean distance between samples x and y in the range.

[0081] In this embodiment, S4 specifically includes the following steps:

[0082] B1: Calculate the sample average in each subset as its initial cluster center;

[0083] B2: In the K-means algorithm, the Euclidean distance from each sample to the initial cluster center is calculated, and the sample is classified into the category of the cluster center closest to it according to the minimum principle;

[0084] B3: Calculate the average distance of sample elements in a category and update it as the new cluster center of the category:

[0085] B4: Compare the new cluster center with the previous cluster center to see if there is any change or if the error is within the requirement. If there is no change, the maximum number of iterations is reached or the allowable error is reached, then output the clustering result, otherwise go to step B2.

[0086] The data set of the present invention is provided by Andrew Moore et al. of Cambridge University. The data set is collected from a research institute for 24-hour high-performance network detection, where each sample represents a TCP data stream, including 248 flow statistical features and corresponding protocol categories. Six network protocols are selected during the experiment, two of which are selected to remove labels as unknown protocols, and 10% of the remaining protocols are randomly selected and marked as known protocols, and the remaining protocols are also removed from the labels.

[0087] Reference Figure 5,Comparison results of algorithm accuracy, four algorithms are used for simulation parameters under the same data set, and the threshold parameter T2 is taken in the same way, among which the ICK algorithm is the algorithm proposed by this invention, and the CK algorithm is the traditional algorithm. In the center point selection method, the MDCK algorithm uses the maximum and minimum principle, and the DCK algorithm uses the maximum density principle.

[0088] Reference Figure 5 , it can be seen from the algorithm accuracy that compared with the MDCK and CK algorithms, the ICK algorithm performs better in terms of accuracy, with an increase of 3%. In repeated experiments, it can be concluded that the ICK algorithm and the MDCK algorithm have good stability, while the CK algorithm has a large fluctuation in accuracy. The stability of the MDCK algorithm's accuracy is due to the fact that one of the two initial center points has the largest density and the other has the farthest distance, so the accuracy is not affected by random labeled samples. The reason why the traditional CK algorithm is unstable is that the distribution discreteness of the sampled samples affects the clustering results, and the probability of low-density samples being selected is low. At the same time, it is affected by the randomness and noise of the center point selection. The DCK algorithm has a lower accuracy than other algorithms. In the result analysis, it is found that the DCK algorithm can distinguish large-class samples, but fails to identify small-class samples. Instead, it classifies small-class samples as high-density samples, which leads to a decrease in overall accuracy. Therefore, the DCK algorithm is not suitable for non-uniform protocol data sets.

[0089] Reference Figure 6 ,It can be seen from the protocol recall rate that the recall rates of the three algorithms for http and service are similar, while the ICK algorithm has a higher recall rate for smtp, ftp-c and multimedia. Since the accuracy of the DCK algorithm is lower than that of other algorithms, the recall rate of this algorithm is not compared here.

[0090] Reference Figure 7 The smaller the sum of squared errors is, the better the clustering effect is. From the figure, it can be concluded that the sum of squared errors of the algorithm proposed by the present invention is smaller than that of other algorithms, which means that the clustering effect of the algorithm is improved.

[0091] The specific calculation process of the Canopy protocol identification system based on the maximum close distance can be found in the above embodiment, and the embodiment of the present invention will not be repeated here.

[0092] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A Canopy protocol identification method based on maximum close distance, characterized in that: The following steps are involved: S1: Preprocess the protocol data set to be identified and use PCA to reduce the dimension; each sample in the data set represents a TCP data flow, including 248 flow statistical features and corresponding protocol categories; S2: Extract a small number of labeled protocols to calculate the average distance avg in each subset i The average value is taken as the distance threshold T2 in the Canopy algorithm; S3: For the first clustering, the sample point with the largest density value is selected as the first cluster center, and Canopy is iterated once. Then the distance value r of the remaining sample points in the data set is calculated, and the point with the maximum value is found as the next center point for iteration until the data set is empty; the sample distance value r is the sample element density ρ i The distance between samples b i The product is divided by the average distance a of the elements within the sample density range i , that is: r = ρ i b i / a i , taking its maximum value is the maximum close distance method; the sample element density ρ i The calculation formula is: in where d ij Represents the Euclidean distance between sample elements j and i; The distance b between samples i The calculation formula is: where ρ j Indicates that element j has a higher density than element i, d ij Represents the distance between sample elements j and i; The average distance a of elements within the sample density range i The calculation formula is: where ρ i is the number of samples in the range, d(i x ,i y ) represents the Euclidean distance between samples x and y within the range; S4: The sample average value calculated in the Canopy clustering data subset is used as its cluster center, thereby obtaining the number of clusters k and the corresponding initial cluster center, which are then brought into K-means for clustering to obtain the recognition result.

2. A Canopy protocol identification method based on the maximum close distance method according to claim 1, characterized in that: The data preprocessing includes digitization, standardization and normalization, and standardization is: in is the average value of each attribute, S k is the mean absolute error, Z ik is a standardized measure; normalization makes the contribution of each dimension equal, that is:

3. The Canopy protocol identification method based on the maximum close distance method according to claim 1 is characterized in that: In S2, the average distance avg within each marker subset i The average value of is defined as: given p sets of labeled protocols, in the i-th (i∈p) set, there are m protocol samples, then the average distance avg of the samples in the set is i Denoted as: Where d(i x ,i y ) represents the Euclidean distance between samples x and y in the i-th set, and according to the p-set avg i The value of sets the Canopy distance threshold T2, that is:

4. The Canopy protocol identification method based on maximum close distance according to claim 1, characterized in that: The S4 specifically comprises the following steps: A1: Calculate the density of sample elements, take the point with the maximum density as the first center point, and perform Canopy clustering; A2: Calculate the distance value determined by the sample element density, the distance between samples, and the average distance of elements within the sample density range, identify the outliers and find the maximum value as the center point of the next iteration and perform Canopy clustering; A3: If the data set is not empty, jump to step A2; if it is empty, jump to step A4; A4: Output the Canopy clustering subsets and the number of subsets.

5. The Canopy protocol identification method based on maximum close distance according to claim 4, characterized in that: The S4 specifically further comprises the following steps: B1: Calculate the sample average in each subset as its initial cluster center; B2: In the K-means algorithm, the Euclidean distance from each sample to the initial cluster center is calculated, and the sample is classified into the category of the cluster center closest to it according to the minimum principle; B3: Calculate the average distance of sample elements in a category and update it as the new cluster center of the category: B4: Compare the new cluster center with the previous cluster center to see if there is any change or if the error is within the requirement. If there is no change, the maximum number of iterations is reached or the allowable error is reached, then output the clustering result, otherwise go to step B2.

6. The Canopy protocol identification method based on maximum close distance according to claim 1, characterized in that: When calculating the sample density value, when the density value is 1 or much smaller than the normal value, and the distance between samples b i However, if the value is greater than the normal value, the sample can be regarded as an outlier or an isolated point generated in the clustering process and is removed before the clustering.

Images