Unknown protocol clustering method and system based on frequent item extraction and bi-layer autoencoder

Abstract

The present application relates to the technical field of information security, in particular to a kind of unknown protocol clustering method and system based on frequent item extraction and double-layer self-encoder, through the preprocessing process, the original data is converted from bit form to byte form, and then the first 32 bytes are intercepted, then the frequency of frequent item and the number of frequent item of each byte are obtained by carrying out frequent item statistics to the preprocessed data byte by byte;Frequency self-encoder and quantity self-encoder are used to extract features from the byte with larger frequency of frequent item and larger number of frequent item respectively;The features extracted by frequency self-encoder are coarsely clustered to obtain coarse clustering label, and the coarse clustering label and the features extracted by quantity self-encoder are merged and clustered to obtain the final fine clustering label.The present application can not only retain protocol-level clustering, but also realize class-level clustering, with small amount of calculation, while ensuring the real-time of clustering, it can effectively solve the under-partition problem of unknown protocol clustering and improve the performance of clustering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, and in particular to an unknown protocol clustering method and system based on frequent item extraction and a double-layer autoencoder. BACKGROUND

[0002] Internet has deeply embedded every corner of social life, and network security problem has become the cornerstone of protecting personal and organizational information security in the digital age. Data in 2023 shows that organizations encounter network attacks an average of 1,168 times per week, among which attackers frequently use non-standard or private network protocols. These protocols are difficult to be effectively identified by traditional security mechanisms due to the lack of public specifications. In this context, protocol reverse engineering, which reveals the syntax and semantics of unknown protocol data packets through in-depth analysis, has become increasingly important in the field of network security. Protocol reverse engineering is an important way to analyze private protocols, and it can infer the constraints and specifications of private protocols based on little or no prior knowledge. It has high practical value in malicious application regulation, protocol fuzz testing, vulnerability detection, and communication behavior understanding. However, in actual network situations, there are usually multiple protocol generated message data, and protocol reverse engineering often works on single message data, which will limit the application of protocol reverse engineering in actual network situations. Therefore, it is necessary to cluster the message data before performing protocol reverse engineering, so as to process the message data of the same class. Unknown protocol clustering is a prerequisite for protocol reverse engineering, and the clustering effect will directly affect the performance of protocol reverse engineering.

[0003] Traditional methods based on machine learning, multiple sequence alignment and its variants have significant effects. Multiple sequence alignment converts binary protocol data frames into quaternary gene form, then calculates the k-seed of base combination to get the distance matrix, and finally performs clustering. Traditional protocol clustering based on frequent item extraction, machine learning algorithms, etc. has significant effects. Frequent item extraction uses Apriori algorithm to mine frequent items from data frames to generate protocol feature files, and then performs unknown protocol clustering based on the feature files. However, frequent item extraction needs to scan the data set multiple times and is prone to candidate set combination explosion when dealing with longer sequences, resulting in high computational complexity and inability to meet the real-time processing needs of network data; machine learning algorithms use principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE) and other algorithms to reduce the dimension of protocol data frames before clustering. Among them, principal component analysis finds the main direction or "principal component" of the data, which can maximize the variance of the data to achieve dimension reduction. Although it is simple and fast to operate, it cannot capture the nonlinear structure and complex patterns in the data because it is a linear dimension reduction; t-distributed stochastic neighbor embedding is a nonlinear dimension reduction, but it needs to calculate the distance matrix between samples, resulting in high time complexity and memory demand of the algorithm.

[0004] With the rapid development of deep learning, the cross application of deep learning and unknown protocol identification brings new possibilities to unknown protocol clustering. For example, the protocol data frame is converted into a gray scale form, and then the classic convolutional neural network is used to extract features from the gray scale form through transfer learning, and finally the features are clustered. However, due to the simple use of pre-trained models without model fine-tuning, the clustering performance is poor. For another example, the self-encoder is used to extract features of unknown protocols, which obtains good clustering performance and is superior to traditional methods in terms of operation time and calculation speed. However, this method can only achieve rough clustering of protocols, and there is little research on different types within the same protocol, and the actual network received data frames often contain multiple protocols and multiple types, so the algorithm for processing unknown protocol clustering using deep learning often has the problem of under-classification when processing actual network traffic data. SUMMARY

[0005] Therefore, the present application provides an unknown protocol clustering method and system based on frequent item extraction and double-layer self-encoder, which solves the problems of under-classification, high algorithm complexity and inability to meet real-time processing requirements in existing unknown protocol clustering analysis.

[0006] According to the design scheme provided by the present application, on the one hand, an unknown protocol clustering method based on frequent item extraction and double-layer self-encoder is provided, which comprises:

[0007] The received bit form network protocol data frame is converted into byte representation and data preprocessing is performed, the data preprocessing including: data frame de-duplication processing and data frame length alignment processing;

[0008] The preprocessed data frame is counted byte by byte to obtain the frequency and number of frequent items of each byte of the data frame, the frequent item being an element with a frequency greater than a preset minimum support degree in the byte;

[0009] The protocol key byte is screened out using the frequency of the frequent item, the protocol category key byte is screened out using the number of the frequent item, and the pre-trained frequency self-encoder and the pre-trained number self-encoder are used to extract features of the protocol key byte and the protocol category key byte respectively;

[0010] The features extracted by the frequency self-encoder are roughly clustered to obtain a protocol rough clustering label, the protocol rough clustering label and the features extracted by the number self-encoder are merged, and the merged features are clustered to obtain a final output protocol fine clustering label, wherein the protocol rough clustering label is a label containing protocol information obtained by clustering the protocol key byte, and the protocol fine clustering label is a label containing protocol category information obtained by clustering the category key byte.

[0011] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present application, further, data preprocessing is performed on the data frames, including:

[0012] The data frames are taken as processing units in specified bits, and the multi-bit is taken as one-dimensional features for clustering in feature clustering, wherein the specified bits are the number of bits corresponding to a single or multiple bytes;

[0013] The length of the data frame is aligned by zero padding and / or truncation for each data frame.

[0014] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present application, further, the preprocessed data frames are counted by byte for frequent items, including:

[0015] The frequency of occurrence of all elements contained in each byte of each data frame is counted;

[0016] If the frequency of occurrence of the element is greater than the preset minimum support, the element is added to the frequent item set to obtain the frequency of occurrence of each frequent item in the frequent item set of each byte.

[0017] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present application, further, the protocol key bytes are screened by using the frequency of frequent items, and the protocol category key bytes are screened by using the number of frequent items, including:

[0018] The frequency of frequent items and the number of frequent items of each byte in the data frame are sorted by size;

[0019] According to the sorting result, the protocol key bytes and the protocol category key bytes are selected from the maximum frequent item frequency and the maximum frequent item number.

[0020] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present application, further, the frequency autoencoder and the number autoencoder both adopt a five-layer encoder structure, the first layer and the last layer of the five-layer encoder structure both include 8 neurons, the second layer and the penultimate layer both include 4 neurons, and the middle layer includes 2 neurons, so that the five-layer encoder structure is used to reduce the error between the reconstructed data and the input data, and the middle layer is used to complete the data dimension reduction processing by learning the low-dimensional representation of the input data.

[0021] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present application, further, the training process of the frequency autoencoder and the number autoencoder, including:

[0022] A weight matrix is constructed according to the protocol data frames and using the frequent item elements, and the mean square error is optimized using the weight matrix to obtain an autoencoding loss function.

[0023] The auto-encoder is iteratively trained based on a self-encoding loss function and sample data, and a weight matrix is initialized using a frequent item frequency before the first round of training, so that the auto-encoder learns to focus on the features of the protocol key bytes.

[0024] As the unknown protocol clustering method based on frequent item extraction and double-layer auto-encoder, further, in the feature clustering, a density-based clustering algorithm DBSCAN is used to cluster the features, and the clustering process includes:

[0025] The radius and minimum point number two optimal parameters in the clustering algorithm DBSCAN are set according to the contour coefficient by using a grid search;

[0026] A data point is randomly selected from the feature set, all neighbor points of the unvisited data point in the radius parameter neighborhood are obtained, if the data point has at least a neighbor point meeting the minimum point number parameter in the neighborhood, the data point is classified as a core point;

[0027] The core point is taken as a clustering center, and the neighbor points of the core point in density connectivity are recursively visited, the neighbor points in density connectivity are neighbor points in the radius parameter neighborhood;

[0028] The core point and the neighbor points in density connectivity are assigned to the same cluster, and the boundary points are assigned to the same cluster as the associated core point.

[0029] In still another aspect, the application further provides an unknown protocol clustering system based on frequent item extraction and double-layer auto-encoder, comprising a data preprocessing module, a frequent item statistics module, a feature extraction module and a clustering analysis module, wherein,

[0030] The data preprocessing module is used for converting the received bit form network protocol data frame into byte representation and performing data preprocessing, the data preprocessing includes data frame de-duplication processing and data frame length alignment processing;

[0031] The frequent item statistics module is used for performing frequent item statistics on the preprocessed data frame byte by byte, to obtain the frequent item frequency and the frequent item number of each byte of the data frame, the frequent item is an element with a frequency greater than a preset minimum support degree in the byte;

[0032] The feature extraction module is used for screening out the protocol key bytes using the frequent item frequency, screening out the protocol category key bytes using the frequent item number, and performing feature extraction on the protocol key bytes and the protocol category key bytes using the pre-trained frequency auto-encoder and the pre-trained number auto-encoder respectively;

[0033] The clustering analysis module is used for rough clustering of the features extracted by the frequency autoencoder, obtaining protocol rough clustering labels, merging the protocol rough clustering labels with the features extracted by the quantity autoencoder, and clustering the merged features to obtain final output protocol fine clustering labels, wherein the protocol rough clustering labels are labels containing protocol information obtained by protocol key byte clustering, and the protocol fine clustering labels are labels containing protocol category information obtained by category key byte clustering.

[0034] The present application has the following advantages:

[0035] The present application performs deduplication, format conversion and equal-length truncation on the collected network data frames, performs byte-by-byte frequent item extraction on the processed data frames to obtain the frequency and quantity of frequent items of each byte, filters protocol key bytes and category key bytes based on the frequency and quantity of frequent items, then inputs the protocol key bytes and the category key bytes into the frequency autoencoder and the quantity autoencoder respectively, and optimizes the loss function of the autoencoder using the frequency of frequent items of each byte so that they can learn more targeted features, performs clustering on the features extracted by the frequency autoencoder to obtain rough clustering labels, and finally merges the features extracted by the quantity autoencoder with the rough clustering labels and performs clustering to obtain fine clustering labels, which can retain protocol-level clustering while realizing category-level clustering through the combination of the rough clustering labels and the fine clustering labels. Compared with the traditional unknown protocol clustering invention, the present application has small calculation amount, ensures the real-time of clustering, effectively solves the under-partitioning problem of unknown protocol clustering, and improves the performance of clustering. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The figure is a schematic of the unknown protocol clustering process based on frequent item extraction and double-layer autoencoder in the embodiment.

[0037] Figure 2 The figure is a schematic of the autoencoder model structure in the embodiment.

[0038] Figure 3 The figure is a schematic of the weight matrix calculation process in the embodiment.

[0039] Figure 4 The figure is a schematic of the rough and fine clustering labels in the embodiment.

[0040] Figure 5 The figure is a schematic of the experimental result comparison in the embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the drawings and technical scheme.

[0042] In view of the problems of under-classification and high complexity in the existing protocol clustering analysis process, an unknown protocol clustering method based on frequent item extraction and double-layer self-encoder is provided, which comprises:

[0043] S101, convert the received bit-form network protocol data frame into byte representation and perform data preprocessing, the data preprocessing includes: data frame de-duplication processing and data frame length alignment processing;

[0044] S102, byte-by-byte frequent item statistics is performed on the preprocessed data frame to obtain the frequency and number of frequent items of each byte of the data frame, the frequent item being an element with a frequency of occurrence in a byte greater than a preset minimum support degree;

[0045] S103, the protocol key bytes are screened out by using the frequency of frequent items, the protocol category key bytes are screened out by using the number of frequent items, and the pre-trained frequency self-encoder and the pre-trained number self-encoder are used to extract features of the protocol key bytes and the protocol category key bytes respectively;

[0046] S104, the features extracted by the frequency self-encoder are coarsely clustered to obtain protocol coarse clustering labels, the protocol coarse clustering labels are combined with the features extracted by the number self-encoder, and the combined features are clustered to obtain the final output protocol fine clustering labels, wherein the protocol coarse clustering labels are labels containing protocol information obtained by clustering the protocol key bytes, and the protocol fine clustering labels are labels containing protocol category information obtained by clustering the category key bytes.

[0047] As Figure 1 shown in the algorithm flow diagram, the original data is converted from bit form to byte form through the preprocessing process, and the first 32 bytes are intercepted, then the frequent items are counted byte by byte for the preprocessed data to obtain the frequency and number of frequent items of each byte, so as to find the protocol key bytes and the category key bytes, and the clustering accuracy is improved by using the two; the frequency self-encoder and the number self-encoder are used to extract features of the bytes with larger frequency and larger number of frequent items respectively, the features suitable for clustering can be obtained by the self-encoder for targeted feature extraction; then the features extracted by the frequency self-encoder are coarsely clustered to obtain coarse clustering labels, the coarse clustering labels and the features extracted by the number self-encoder are combined and clustered to obtain the final fine clustering labels.

[0048] Among them, the data preprocessing of the data frame can be designed to include:

[0049] The data frame is taken as a processing unit in a specified bit, and the multi-bit is taken as a one-dimensional feature for clustering in feature clustering, the specified bit being a bit number corresponding to a single or multiple bytes;

[0050] The data frame length is aligned by zero padding and / or truncation for each data frame.

[0051] During the transmission of network data frames, retransmission of data frames may occur due to network congestion, retransmission mechanism or error detection mechanism, so that a large number of duplicate data frames exist in the actual network protocol data frames received. If each frame is directly processed without deduplication, unnecessary bandwidth is occupied, calculation is wasted, the accuracy of performance evaluation is affected, and the uneven distribution of data set is deepened, which affects the clustering performance. Therefore, the data frames need to be deduplicated first.

[0052] For protocol data frames, a bit is taken as a processing unit and recorded as a one-dimensional feature. The dimension of the data frame is usually hundreds or even thousands of bits, and a large amount of redundant information is contained therein. In order to improve the performance of clustering and reduce the calculation amount, multi-bit is taken as a one-dimensional feature for clustering. Generally, 4 bits or 8 bits are considered. However, when 4 bits are taken as a one-dimensional feature, there are 16 possible values for each one-dimensional feature, i.e. 0-15. At this time, the number of possible values for each one-dimensional feature is small. When the unknown protocol data contains a large number of protocols, some features that are not frequent items may be treated as frequent items, which may result in incorrect extraction of protocol keywords and category keywords. Therefore, in the embodiments of the present case, 8 bits, i.e. 1 byte, can be selected as a one-dimensional feature.

[0053] In addition, for the same protocol, there are multiple frame lengths due to different frame purposes. Even the same type of data frame may have an extensible field, resulting in variable frame length. The byte-by-byte frequent item extraction and the input of the autoencoder are both fixed-length sequences, so the data frames need to be zero-padded or truncated. Existing experiments show that almost all possible protocol features can be obtained when the first 20 bytes are used for data mining. Therefore, in the embodiments of the present case, the first 32 bytes of each data frame can be truncated to achieve length alignment.

[0054] Specifically, the byte-by-byte frequent item statistics of the preprocessed data frames can be designed to include:

[0055] The frequency of occurrence of all elements contained in each byte of each data frame is counted.

[0056] If the frequency of occurrence of an element is greater than a preset minimum support degree, the element is added to the frequent item set to obtain the frequency of occurrence of each frequent item of each byte of the frequent item set.

[0057] Among them, the protocol key bytes are screened out by using the frequent item frequency, and the protocol category key bytes are screened out by using the number of frequent items, which can include:

[0058] The frequent item frequency and the number of frequent items of each byte of the data frame are sorted by size.

[0059] According to the sorting result, the maximum frequent item frequency and the maximum frequent item number are selected to obtain the protocol key bytes and the protocol category key bytes.

[0060] Although the traditional frequent item extraction based on Apriori and Eclat algorithms has good effects in association rule learning, they have the disadvantages of large memory and large calculation amount. Moreover, these algorithms can mine different unknown frequent items, but for protocol data frame clustering, only the frequent items located in the same byte are meaningful, which leads to the fact that the traditional frequent item extraction algorithm will obtain a lot of redundancy. In addition, the frequent item extraction algorithm obtains variable-length sequences, while the autoencoder model generally needs to input fixed-length sequences. Therefore, the traditional frequent item extraction algorithm has low applicability in unknown protocol clustering. Therefore, in the embodiments of the present case, byte-by-byte frequent item extraction is used. The byte-by-byte frequent item extraction only considers the frequent items in each byte and does not consider the correlation between the bytes. Compared with the traditional algorithm, it has the characteristics of small calculation amount and is more suitable for unknown protocol clustering.

[0061] The byte-by-byte frequent item extraction respectively counts the number of occurrences of all elements contained in each byte. If the occurrence frequency is greater than the set minimum support, it is included in the frequent item set. Through the above operation, the frequent item set of each byte and the frequency of each frequent item can be obtained. The specific calculation formula can be expressed as follows:

[0062] IT = {x e A | (num(x) / N) > minsupport}

[0063] Where IT is the set of all frequent item elements of the byte, A is the set of all elements of the byte, and minsupport is the set minimum support.

[0064] For any protocol, there are usually some fixed bytes that have corresponding rules that can distinguish them from other protocols. These bytes can greatly assist us in inferring the protocol category, and therefore, they can be referred to as protocol key bytes. Similarly, for any category in the same protocol, there are usually some fixed bytes that have corresponding rules that can distinguish them from other categories. These bytes can also assist in category inference, and therefore, they can be referred to as category key bytes. For protocol key bytes, since they are common to a certain protocol, the frequency of their frequent item elements is usually large, and the protocol key bytes can be screened out through the frequent item frequency. For category key bytes, since different protocols have different categories, this will result in a large number of frequent items contained in the category key bytes. Therefore, the category key bytes can be screened out according to the size of the frequent item number. The calculation formulas of the frequent item frequency and the frequent item number can be expressed as follows:

[0065]

[0066] n it = len(IT)

[0067] where f it is the frequency of frequent item, n it is the number of frequent item, it i is the i-th frequent item in the byte, num(it i ) is the number of times it i appears in the byte.

[0068] After obtaining f it and n it of each byte, they are sorted by size, and the 8 bytes with the largest f it are taken as the protocol key bytes, and the 8 bytes with the largest n it are taken as the category key bytes.

[0069] Specifically, both the frequency autoencoder and the quantity autoencoder adopt a five-layer encoder structure, the first layer and the last layer of the five-layer encoder structure each contain 8 neurons, the second layer and the penultimate layer each include 4 neurons, and the middle layer includes 2 neurons, so as to use the five-layer encoder structure to reduce the error between the reconstructed data and the input data, so that the middle layer learns the low-dimensional representation of the input data to complete the data dimension reduction processing.

[0070] After obtaining the protocol key bytes and the category key bytes, they are respectively input into two autoencoders, and the two autoencoders have the same structure, and the only difference is that the input data is different. The autoencoder inputting the feature key bytes is called the frequency autoencoder, and the autoencoder inputting the category key bytes is called the quantity autoencoder. As shown in FIG. 2, the first layer and the last layer of the autoencoder are 8 units, the second layer and the penultimate layer are 4 units, and the middle embedding layer is 2 units. The autoencoder reduces the error between the reconstructed data and the input data, so that the embedding layer of the autoencoder can learn the low-dimensional representation of the input data and complete the dimension reduction of the data. Figure 2

[0071] wherein the training process of the frequency autoencoder and the quantity autoencoder is designed to include:

[0072] constructing a weight matrix according to the protocol data frame and using the frequent item elements, and optimizing the mean square error using the weight matrix to obtain an autoencoding loss function;

[0073] iteratively training the autoencoder based on the autoencoding loss function and using sample data, and initializing the weight matrix using the frequency of frequent items before the first round of training, so that the autoencoder learns to focus on the features of the protocol key bytes.

[0074] ​Autoencoders can map high-dimensional data to low-dimensional space while preserving the main features of the data, so they are widely used in data dimensionality reduction. Compared with traditional machine learning dimensionality reduction algorithms, autoencoder models can learn more complex data patterns.

[0075] Traditional autoencoder models usually use mean square error as their loss function. However, for autoencoders used for unknown protocol clustering, using mean square error as the loss function directly will cause the autoencoder to learn all features of the input data indiscriminately and fail to focus on frequent items that can distinguish unknown protocols. Therefore, it is necessary to optimize the loss function according to the frequent items to make it more suitable for the task of unknown protocol clustering. Specifically, a weight matrix is constructed for a batch of protocol data frames. If the element at this bit of the data frame is a frequent item element, the weight at the corresponding position is 1, otherwise it is 0. As shown in the weight matrix calculation diagram. Figure 3

[0076] The mean square error is optimized using the weight to obtain a new loss function, and the specific calculation formula can be represented as follows:

[0077]

[0078] where w i is the i-th element of the weight vector of the data frame, x i is the i-th element of the frame data, is the i-th element of the reconstructed data, and N is the number of input data frames.

[0079] In addition, any neural network will have a weight initialization process before training, and the same applies to autoencoders. Weight initialization will directly affect the learning ability and training efficiency of the network. If the initialization is not good, problems such as gradient vanishing, gradient explosion, and inaccurate learned features may occur. The most commonly used initialization method today is He uniform initialization, whose weight matrix w follows a uniform distribution. For unknown protocol clustering, it is hoped that the autoencoder will learn more frequent item features while avoiding gradient vanishing and gradient explosion. Therefore, in the embodiments of the present case, the frequency of frequent items is used to optimize the first layer weights of the autoencoder, and the calculation formula can be represented as follows:

[0080]

[0081] where w ij is the weight of the i-th neural network connection from the first layer to the j-th neuron in the second layer of the autoencoder, f it is a set of frequent item frequencies, f iti is the first element in the set, and n in ​is the number of neurons in the model. Through this weight initialization method, the self-encoder can effectively learn the features of frequent items, and then perform subsequent clustering.

[0082] Specifically, when clustering features, a density-based clustering algorithm DBSCAN can be used to cluster the features, and the clustering process can be designed to include:

[0083] The radius and minimum point number two optimal parameters in the clustering algorithm DBSCAN are set according to the silhouette coefficient using grid search;

[0084] A random unvisited data point is selected from the feature set, and all neighbor points of the unvisited data point within the radius parameter neighborhood are obtained. If the data point has at least a neighbor point meeting the minimum point number parameter in the neighborhood, the data point is classified as a core point.

[0085] The core point is taken as a clustering center, and the neighbor points of the core point that are density connected are recursively visited starting from the core point, the density connected neighbor points being neighbor points that are in the radius parameter neighborhood of each other.

[0086] The core point and the neighbor points density connected thereto are assigned to the same cluster, and the boundary point is assigned to the same cluster as the associated core point.

[0087] The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm clusters by defining core points (points having enough neighbors within a given radius), boundary points (located within the neighborhood of a core but not a core point), and noise points. The algorithm first finds all core points, and then connects all adjacent core points according to density reachability to form a cluster group.

[0088] The features extracted by the frequency self-encoder are clustered by the DBSCAN algorithm to obtain coarse clustering labels, and then the coarse clustering labels and the features extracted by the quantity self-encoder are merged to perform a second DBSCAN clustering to obtain fine clustering labels. The coarse clustering labels are obtained by clustering the protocol key bytes, and mainly contain protocol information, while the fine clustering labels are obtained by clustering the category key words, and mainly contain fine-grained category information. The two kinds of labels are as shown in Figure 4 .

[0089] The coarse clustering labels classify the same protocol data frames in unknown protocol data frames into the same category, while the fine clustering labels classify different categories of the same protocol into different categories. By combining the fine clustering labels and the coarse clustering labels, the protocol clustering and the classification of the internal categories of the protocol can be realized at the same time.

[0090] Further, based on the above method, the embodiment of the application also provides an unknown protocol clustering system based on frequent item extraction and double-layer autoencoder, comprising a data preprocessing module, a frequent item statistics module, a feature extraction module and a clustering analysis module, wherein,

[0091] The data preprocessing module is used for converting the received bit-form network protocol data frame into byte representation and performing data preprocessing, wherein the data preprocessing comprises data frame deduplication processing and data frame length alignment processing.

[0092] The frequent item statistics module is used for performing frequent item statistics on the preprocessed data frame byte by byte to obtain the frequent item frequency and the frequent item quantity of each byte of the data frame, wherein the frequent item is an element with a frequency of occurrence in a byte greater than a preset minimum support degree.

[0093] The feature extraction module is used for screening out protocol key bytes by using the frequent item frequency, screening out protocol category key bytes by using the frequent item quantity, and performing feature extraction on the protocol key bytes and the protocol category key bytes by using a pre-trained frequency autoencoder and a pre-trained quantity autoencoder respectively.

[0094] The clustering analysis module is used for performing coarse clustering on the features extracted by the frequency autoencoder to obtain protocol coarse clustering labels, combining the protocol coarse clustering labels with the features extracted by the quantity autoencoder, and performing clustering on the combined features to obtain finally outputted protocol fine clustering labels, wherein the protocol coarse clustering labels are labels containing protocol information obtained by clustering the protocol key bytes, and the protocol fine clustering labels are labels containing protocol category information obtained by clustering the category key bytes.

[0095] To verify the effectiveness of the scheme, the following experimental data are used for further explanation and description:

[0096] The data set used in the experiment is shown in the following table:

[0097]

[0098] The AIS is received from the actual communication network, and the ARP, DNS, NBNS and SMB are from the MACCDC 2012 data set, which is the network traffic data collected during the network defense competition of the University of Atlantic in the United States national network observation. The PCAP file of the data set is deduplicated and preliminarily processed to obtain the above-mentioned 8027 data frames. After the protocol data frames are extracted by using the traditional machine learning principal component analysis (PCA) algorithm, the autoencoder and the double-layer autoencoder based on frequent items in the scheme, the DBSCAN clustering algorithm is used for clustering. The ε and MinPts of the DBSCAN algorithm are used, and the commonly used F and purity in clustering are used as evaluation indexes.

[0099] The experimental results are shown in Table 2. Figure 5 As can be seen from the experimental results, the scheme based on the frequent item extraction self-encoder has higher F value and purity value and better overall performance compared with the other two algorithms in unknown protocol clustering, and the protocol level information can be retained after clustering based on the frequent item extraction self-encoder, which has great significance for subsequent protocol reverse work.

[0100] In addition, since network communication data has strong real-time performance, the running time of the algorithm is also an important performance indicator. The running time of the three algorithms is tested using the above test set, and the specific results are shown in the following table:

[0101] Table Running time statistics

[0102]

[0103] As can be seen from the actual running data loss time, the traditional machine learning algorithm has certain advantages in running time compared with the deep learning, and the running time of the PCA is significantly shortened compared with the self-encoder, and although there is still certain advantage compared with the scheme based on the frequent item extraction self-encoder, the difference is not large. However, the clustering performance of the PCA is obviously different from that of the scheme based on the frequent item extraction self-encoder, and therefore, in general, the scheme based on the frequent item extraction self-encoder has certain advantages in unknown protocol clustering compared with the traditional algorithm.

[0104] In addition, the time calculated in the above table is calculated using the CPU, and nowadays, the GPU is widely used in the model training process of deep learning due to its powerful parallel computing capability. If a better performance GPU is used to replace the CPU for calculation, the operation time of the scheme based on the frequent item extraction self-encoder will also be shortened.

[0105] Through the above experimental data, it can be shown that the scheme combines the coarse clustering label and the fine clustering label, which can not only retain the protocol level clustering but also realize the class level clustering. In addition, compared with the traditional unknown protocol clustering invention, the calculation amount is small, which can effectively solve the under-segmentation problem of unknown protocol clustering, improve the performance of clustering, and has good application prospect.

[0106] Unless otherwise specified, the relative steps, numerical expressions and values of the components and steps set forth in these examples do not limit the scope of the present application.

[0107] The various embodiments are described in the specification in a progressive manner, each embodiment focusing on different aspects of the other embodiments, and the same or similar parts between the embodiments can be mutually referred to. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method.

[0108] The units and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been described in the above description in general terms. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation does not exceed the scope of the present application.

[0109] Those skilled in the art can understand that all or part of the steps in the above method can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, etc. Alternatively, all or part of the steps of the above embodiments can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiments can be implemented in the form of hardware or in the form of a software function module. The present application is not limited to any specific form of combination of hardware and software.

[0110] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, which are used to illustrate the technical solutions of the present application, and are not limiting. The protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A clustering method for unknown protocols based on frequent item extraction and a two-layer autoencoder, characterized in that, Include: The received bit-form network protocol data frames are converted into byte representations and preprocessed. The data preprocessing includes: data frame deduplication and data frame length alignment. The frequent items are counted byte by byte in the preprocessed data frame to obtain the frequency and number of frequent items in each byte of the data frame. The frequent items are elements in the byte whose frequency of occurrence is greater than the preset minimum support. Protocol keyword segments are selected by frequency of frequent items, and protocol category keyword segments are selected by number of frequent items. Pre-trained frequency autoencoders and pre-trained quantity autoencoders are used to extract features from protocol keyword segments and protocol category keyword segments, respectively. The features extracted by the frequency autoencoder are coarsely clustered to obtain protocol coarse cluster labels. The protocol coarse cluster labels are merged with the features extracted by the quantity autoencoder, and the merged features are clustered to obtain the final output protocol fine cluster labels. The protocol coarse cluster labels are labels containing protocol information obtained by clustering using protocol keyword nodes, and the protocol fine cluster labels are labels containing protocol category information obtained by clustering using category keyword nodes.

2. The unknown protocol clustering method based on frequent item extraction and two-layer autoencoder according to claim 1, characterized in that, Data preprocessing of data frames includes: The data frame is processed in a specified bit unit so that it can be clustered with multiple bits as one-dimensional features during feature clustering. The specified bit is the number of bits corresponding to a single or multiple bytes. Data frame length alignment is achieved by padding each data frame with zeros and / or truncating it.

3. The unknown protocol clustering method based on frequent item extraction and two-layer autoencoder according to claim 1, characterized in that, The preprocessed data frame is analyzed byte by byte for frequent item statistics, including: Statistical analysis was performed on the frequency of all elements contained in each byte of each data frame. If the frequency of an element is greater than the preset minimum support, the element is added to the frequent itemset to obtain the frequency of each frequent item in the frequent itemset for each byte.

4. The unknown protocol clustering method based on frequent item extraction and two-layer autoencoder according to claim 1 or 3, characterized in that, The protocol keyword byte is filtered out by the frequency of frequent items, and the protocol category keyword byte is filtered out by the number of frequent items. This includes sorting the frequency of frequent items and the number of frequent items in each byte of the data frame by size. Based on the sorting results, select the protocol keyword section and protocol category keyword section from the highest frequency of frequent items and the highest number of frequent items.

5. The unknown protocol clustering method based on frequent item extraction and two-layer autoencoder according to claim 1, characterized in that, Both the frequency autoencoder and the quantity autoencoder adopt a five-layer encoder structure. The first and last layers of the five-layer encoder structure each contain 8 neurons, the second and penultimate layers each contain 4 neurons, and the middle layer contains 2 neurons. This five-layer encoder structure can reduce the error between the reconstructed data and the input data, so that the middle layer can complete the data dimensionality reduction process by learning the low-dimensional representation of the input data.

6. The unknown protocol clustering method based on frequent item extraction and two-layer autoencoder according to claim 1 or 5, characterized in that, The training process for the frequency autoencoder and the quantity autoencoder includes: Based on the protocol data frames and using the frequent terms to construct a weight matrix, the mean square error is optimized using the weight matrix to obtain the autoencoding loss function; The autoencoder is trained iteratively using an autoencoder loss function and sample data. The weight matrix is ​​initialized with the frequency of frequent terms before the first round of training so that the autoencoder learns the features of the protocol key section.

7. The unknown protocol clustering method based on frequent item extraction and two-layer autoencoder according to claim 1, characterized in that, During feature clustering, the density-based clustering algorithm DBSCAN is used to cluster the features. The clustering process includes: Use grid search and set the two optimal parameters, radius and minimum number of points, in the DBSCAN clustering algorithm based on the contour coefficient; Randomly select an unvisited data point from the feature set, obtain all neighbor points of the unvisited data point within the radius parameter neighborhood, and classify the data point as a core point if the data point has at least one neighbor point that meets the minimum number of points parameter within the neighborhood. Using the core point as the cluster center, starting from the core point, recursively visit the density-connected neighbor points of the core point, where the density-connected neighbor points are the neighbor points of each other within the radius parameter neighborhood. Core points and their density-connected neighboring points are assigned to the same cluster, and boundary points are assigned to the same cluster as the associated core points.

8. An unknown protocol clustering system based on frequent item extraction and a two-layer autoencoder, characterized in that, It includes: a data preprocessing module, a frequent item statistics module, a feature extraction module, and a cluster analysis module. The data preprocessing module is used to convert the received bit-form network protocol data frames into byte representations and perform data preprocessing, which includes: data frame deduplication and data frame length alignment. The frequent item statistics module is used to perform frequent item statistics on each byte of the preprocessed data frame to obtain the frequency and number of frequent items for each byte of the data frame. The frequent item is the element in the byte whose frequency of occurrence is greater than the preset minimum support. The feature extraction module is used to filter out protocol keyword segments by the frequency of frequent items, filter out protocol category keyword segments by the number of frequent items, and extract features from protocol keyword segments and protocol category keyword segments by using pre-trained frequency autoencoders and pre-trained quantity autoencoders, respectively. The clustering analysis module is used to perform coarse clustering on the features extracted by the frequency autoencoder to obtain protocol coarse cluster labels. The protocol coarse cluster labels are then merged with the features extracted by the quantity autoencoder, and the merged features are clustered to obtain the final output protocol fine cluster labels. The protocol coarse cluster labels are labels containing protocol information obtained by clustering using protocol keyword nodes, while the protocol fine cluster labels are labels containing protocol category information obtained by clustering using category keyword nodes.

9. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1 to 7.

Images