Unknown protocol clustering method and system based on frequent item extraction and double-layer auto-encoder

Abstract

The invention relates to the technical field of information security, in particular to an unknown protocol clustering method and system based on frequent item extraction and a double-layer auto-encoder, and the method comprises the steps: converting original data from a bit form into a byte form through a preprocessing process, intercepting the first 32 bytes, carrying out the frequent item statistics of the preprocessed data byte by byte, and obtaining a frequent item clustering result; respectively obtaining the frequent item frequency and the frequent item number of each byte; utilizing a frequency auto-encoder and a number auto-encoder to respectively carry out feature extraction on bytes with larger frequent item frequency and larger frequent item number; and performing coarse clustering on the features extracted by the frequency auto-encoder to obtain coarse clustering tags, and merging the coarse clustering tags and the features extracted by the number auto-encoder and performing clustering to obtain final fine clustering tags. According to the method, not only can protocol-level clustering be reserved, but also category-level clustering can be realized, the calculation amount is small, the clustering instantaneity is ensured, meanwhile, the under-division problem of unknown protocol clustering can be effectively solved, and the clustering performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to an unknown protocol clustering method and system based on frequent item extraction and a double-layer autoencoder. Background Art

[0002] The Internet has been deeply embedded in every corner of social life, and network security issues have become the cornerstone of protecting the information security of individuals and organizations in the digital age. Data from 2023 show that the average frequency of cyber attacks encountered by organizations per week is as high as 1,168 times. Among them, attackers frequently use non-standard or private network protocols, which are difficult to be effectively identified by traditional security mechanisms due to the lack of public specifications. In this context, protocol reverse engineering - a technology that reveals the syntax and semantics of unknown protocol data packets through deep analysis - has become increasingly important in the field of network security. Protocol reverse engineering technology is an important way to analyze private protocols, and it infers the constraints and specifications of private protocols based on little or no prior knowledge. It has high practical value in malicious application supervision, protocol fuzz testing, vulnerability detection, and communication behavior understanding. However, in actual network situations, there are usually message data generated by multiple protocols, and protocol reverse engineering often acts on a single message data, which will limit protocol reverse technology in actual network situations. Therefore, before performing protocol reverse work, it is necessary to cluster the message data so that the same type of message data can be processed. Clustering of unknown protocols is a prerequisite for protocol reverse engineering. The quality of clustering will directly affect the performance of protocol reverse engineering.

[0003] Traditional methods based on machine learning, multiple sequence comparison and their variants have significant effects. Multiple sequence comparison converts binary protocol data frames into quaternary gene form, and then calculates the k-seed of the base combinations to obtain the distance matrix, and finally clusters them. Traditional protocol clustering is based on frequent item extraction, machine learning algorithms and other methods and has significant effects. Frequent item extraction uses algorithms such as Apriori to mine frequent items on data frames to generate protocol feature files, and implements unknown protocol clustering based on feature files. However, frequent item extraction requires multiple scans of the data set and is prone to candidate set combinatorial explosion when processing longer sequences, resulting in high computational complexity and unable to meet the needs of real-time network data processing; machine learning algorithms use principal component analysis (PCA), t-distributed neighbor embedding (t-SNE) and other algorithms to reduce the dimension of protocol data frames before clustering. Among them, principal component analysis achieves dimensionality reduction by finding the main directions or "principal components" of the data. These directions can maximize the variance of the data. Although its operation is simple and the processing is fast, it cannot capture the nonlinear structure and complex patterns in the data because it is a linear dimensionality reduction. Although t-distribution neighborhood embedding is a nonlinear dimensionality reduction, it needs to calculate the distance matrix between the samples, which leads to high time complexity and memory requirements of the algorithm.

[0004] With the rapid development of deep learning, the cross-application of deep learning and unknown protocol identification has brought new possibilities to unknown protocol clustering. For example, the protocol data frame is converted into a grayscale image, and then the classic convolutional neural network is used to extract features from the grayscale image through transfer learning, and finally the features are clustered. However, due to the simple use of pre-trained models without fine-tuning the model, the clustering performance is poor. For another example, the use of autoencoders to extract features of unknown protocols has achieved good clustering while also outperforming traditional methods in terms of computing time and speed. However, this type of method can only achieve coarse clustering of protocols, and there are few studies on the problem of different types within the same protocol. The data frames received by the actual network often contain multiple protocols and multiple types. Therefore, the algorithm that uses deep learning to process unknown protocol clustering often has the problem of under-classification when processing actual network traffic data. Summary of the invention

[0005] To this end, the present invention provides an unknown protocol clustering method and system based on frequent item extraction and double-layer autoencoder to solve the problems existing in the existing unknown protocol clustering analysis, such as under-classification, high algorithm complexity, and inability to meet real-time processing requirements.

[0006] According to the design scheme provided by the present invention, on the one hand, a method for clustering unknown protocols based on frequent item extraction and a double-layer autoencoder is provided, comprising:

[0007] Convert the received bit-form network protocol data frame into a byte representation and perform data preprocessing, wherein the data preprocessing includes: data frame deduplication processing and data frame length alignment processing;

[0008] Perform frequent item statistics on the preprocessed data frame byte by byte to obtain the frequency and number of frequent items of each byte of the data frame, wherein the frequent item is an element whose occurrence frequency in the byte is greater than the preset minimum support;

[0009] The frequency of frequent items is used to filter out the key bytes of the protocol, the number of frequent items is used to filter out the key bytes of the protocol category, and the pre-trained frequency autoencoder and the pre-trained number autoencoder are used to extract features of the key bytes of the protocol and the key bytes of the protocol category respectively;

[0010] The features extracted by the frequency autoencoder are coarsely clustered to obtain the coarse protocol clustering labels, the coarse protocol clustering labels are merged with the features extracted by the quantity autoencoder, and the merged features are clustered to obtain the final output fine protocol clustering labels, where the coarse protocol clustering labels contain protocol information obtained by clustering protocol keyword bytes, and the fine protocol clustering labels contain protocol category information obtained by clustering category keyword bytes.

[0011] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present invention, further, data preprocessing is performed on the data frame, including:

[0012] The data frame is processed with a designated bit as a unit, and multiple bits are clustered as one-dimensional features during feature clustering, wherein the designated bit is the number of bits corresponding to a single byte or multiple bytes;

[0013] The data frame length is aligned by padding each data frame with zeros and / or truncating the data frame.

[0014] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present invention, further, the pre-processed data frame is subjected to frequent item statistics byte by byte, including:

[0015] Count the frequency of occurrence of all elements contained in each byte of each data frame;

[0016] If the frequency of occurrence of an element is greater than the preset minimum support, the element is added to the frequent itemset to obtain the frequency of occurrence of each frequent item in the frequent itemset of each byte.

[0017] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present invention, further, the protocol key bytes are screened out by using the frequent item frequency, and the protocol category key bytes are screened out by using the frequent item quantity, including:

[0018] Sort the frequency and number of frequent items of each byte in the data frame;

[0019] According to the sorting results, the protocol keyword byte and the protocol category keyword byte are selected from the largest frequent item frequency and the largest frequent item quantity.

[0020] As an unknown protocol clustering method based on frequent item extraction and two-layer autoencoder of the present invention, further, the frequency autoencoder and the quantity autoencoder both adopt a five-layer encoder structure, the first layer and the last layer of the five-layer encoder structure each include 8 neurons, the second layer and the penultimate layer each include 4 neurons, and the middle layer includes 2 neurons, so as to utilize the five-layer encoder structure to reduce the error between the reconstructed data and the input data, so that the middle layer completes the data dimensionality reduction processing by learning the low-dimensional representation of the input data.

[0021] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present invention, further, the training process of the frequency autoencoder and the quantity autoencoder includes:

[0022] A weight matrix is ​​constructed according to the protocol data frame and using frequent item elements, and the mean square error is optimized using the weight matrix to obtain an autoencoder loss function;

[0023] The autoencoder is iteratively trained based on the autoencoder loss function and using sample data. The weight matrix is ​​initialized using the frequent item frequency before the first round of training so that the autoencoder can learn the features that focus on the key bytes of the protocol.

[0024] As the unknown protocol clustering method based on frequent item extraction and double-layer autoencoder of the present invention, further, when clustering features, a density-based clustering algorithm DBSCAN is used to cluster the features, and the clustering process includes:

[0025] Use grid search to set the two optimal parameters of radius and minimum number of points in the clustering algorithm DBSCAN based on the silhouette coefficient;

[0026] Randomly select an unvisited data point from the feature set, obtain all neighboring points of the unvisited data point within the radius parameter neighborhood, and classify the data point as a core point if the data point has at least one neighboring point that meets the minimum number of points parameter in the neighborhood;

[0027] Taking the core point as the cluster center, starting from the core point, recursively visit the core point's density-connected neighbor points, where the density-connected neighbor points are neighbor points within the radius parameter neighborhood;

[0028] The core points and their density-connected neighboring points are assigned to the same cluster, and the boundary points are assigned to the same cluster as the associated core points.

[0029] On the other hand, the present invention also provides an unknown protocol clustering system based on frequent item extraction and double-layer autoencoder, comprising: a data preprocessing module, a frequent item statistics module, a feature extraction module and a clustering analysis module, wherein:

[0030] A data preprocessing module, used for converting the received bit-form network protocol data frame into a byte representation and performing data preprocessing, wherein the data preprocessing includes: data frame deduplication processing and data frame length alignment processing;

[0031] A frequent item statistics module is used to perform frequent item statistics on the preprocessed data frame byte by byte to obtain the frequent item frequency and the number of frequent items of each byte of the data frame, wherein the frequent item is an element whose occurrence frequency in the byte is greater than a preset minimum support;

[0032] A feature extraction module is used to filter out protocol key bytes using the frequency of frequent items, filter out protocol category key bytes using the number of frequent items, and extract features of protocol key bytes and protocol category key bytes using a pre-trained frequency autoencoder and a pre-trained number autoencoder, respectively;

[0033] The clustering analysis module is used to coarsely cluster the features extracted by the frequency autoencoder to obtain a coarse protocol clustering label, merge the coarse protocol clustering label with the features extracted by the quantity autoencoder, and cluster the merged features to obtain a final output fine protocol clustering label, wherein the coarse protocol clustering label is a label containing protocol information obtained by clustering protocol keyword bytes, and the fine protocol clustering label is a label containing protocol category information obtained by clustering category keyword bytes.

[0034] Beneficial effects of the present invention:

[0035] The invention removes duplicates, converts formats and performs equal-length interception on collected network data frames; extracts frequent items byte by byte on the processed data frames to obtain the frequent item frequency and the number of frequent items of each byte; screens protocol key bytes and category key bytes based on the frequent item frequency and the number of frequent items; then inputs the protocol key bytes and the category key bytes into a frequency autoencoder and a quantity autoencoder respectively; and can use the frequent item frequency of bytes to optimize the loss function of the autoencoder so that they can learn more targeted features; clusters the features extracted by the frequency autoencoder to obtain a coarse clustering label; finally, merges the features extracted by the quantity autoencoder and the coarse clustering label and performs clustering to obtain a fine clustering label, and by combining the coarse clustering label and the fine clustering label, it is possible to retain the protocol level clustering while realizing the category level clustering; compared with the traditional unknown protocol clustering invention, the amount of calculation is small, and while ensuring the real-time performance of clustering, it is possible to effectively solve the problem of under-scoring of unknown protocol clustering and improve the performance of clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The figure is a schematic diagram of the unknown protocol clustering process based on frequent item extraction and double-layer autoencoder in the embodiment;

[0037] Figure 2 This is a schematic diagram of the autoencoder model structure in the embodiment;

[0038] Figure 3 This is a schematic diagram of the weight matrix calculation process in the embodiment;

[0039] Figure 4 This is a schematic diagram of coarse and fine clustering labels in the embodiment;

[0040] Figure 5 The following is a comparison diagram of the experimental results in the examples. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0042] In view of the problems of under-classification and high complexity in the existing protocol clustering analysis process, an embodiment of the present invention provides an unknown protocol clustering method based on frequent item extraction and a double-layer autoencoder, comprising:

[0043] S101, converting the received bit-form network protocol data frame into a byte representation and performing data preprocessing, wherein the data preprocessing includes: data frame deduplication processing and data frame length alignment processing;

[0044] S102, performing frequent item statistics on the preprocessed data frame byte by byte to obtain the frequency and number of frequent items of each byte of the data frame, wherein the frequent item is an element whose occurrence frequency in the byte is greater than a preset minimum support;

[0045] S103, filter out protocol key bytes using the frequency of frequent items, filter out protocol category key bytes using the number of frequent items, and extract features of protocol key bytes and protocol category key bytes using a pre-trained frequency autoencoder and a pre-trained number autoencoder, respectively;

[0046] S104. Coarsely cluster the features extracted by the frequency autoencoder to obtain a coarse protocol clustering label, merge the coarse protocol clustering label with the features extracted by the quantity autoencoder, and cluster the merged features to obtain a final output fine protocol clustering label, wherein the coarse protocol clustering label is a label containing protocol information obtained by clustering protocol keyword bytes, and the fine protocol clustering label is a label containing protocol category information obtained by clustering category keyword bytes.

[0047] like Figure 1 The algorithm flow shown is as follows: the original data is converted from bit form to byte form through a preprocessing process, and the first 32 bytes are intercepted. Then, the preprocessed data is subjected to frequent item statistics byte by byte to obtain the frequent item frequency and the number of frequent items of each byte, respectively, so as to find the protocol key bytes and the category key bytes, and use both to improve the clustering accuracy; the frequency autoencoder and the quantity autoencoder are used to extract features of bytes with large frequent item frequencies and large frequent item numbers, respectively, and the autoencoder is used to extract features of the data in a targeted manner to obtain features that are more suitable for clustering; then, the features extracted by the frequency autoencoder are coarsely clustered to obtain coarse clustering labels, and the coarse clustering labels and the features extracted by the quantity autoencoder are merged and clustered to obtain the final fine clustering labels.

[0048] Among them, data preprocessing of the data frame can be designed to include:

[0049] The data frame is processed with a designated bit as a unit, and multiple bits are clustered as one-dimensional features during feature clustering, wherein the designated bit is the number of bits corresponding to a single byte or multiple bytes;

[0050] The data frame length is aligned by padding each data frame with zeros and / or truncating the data frame.

[0051] During the transmission of network data frames, network congestion, retransmission mechanism or error detection mechanism may cause data frames to be retransmitted, resulting in a large number of duplicate data frames in the actual received network protocol data frames. If each frame is processed directly without deduplication, it will not only occupy unnecessary bandwidth, waste computing power, and affect the accuracy of performance evaluation, but also may deepen the uneven distribution of data sets and affect clustering performance. Therefore, it is necessary to deduplicate the data frames first.

[0052] For protocol data frames, bits are used as processing units and recorded as one-dimensional features. The dimensions of the data frames will usually range from hundreds to thousands of bits, and contain a lot of redundant information. In order to improve clustering performance and reduce the amount of calculation, it is necessary to cluster with multiple bits as one-dimensional features. Generally, 4 bits or 8 bits are considered, but when 4 bits are used as one-dimensional features, there are 16 possible values ​​for each dimension of the feature, from 0 to 15. At this time, there are fewer possible types of values ​​for each dimension of the feature. When the unknown protocol data contains a large number of protocols, some features that are not frequent items will be treated as frequent items, resulting in the inability to correctly extract protocol keywords and category keywords. Therefore, in the embodiment of this case, 8 bits, or 1 byte, can be selected as the one-dimensional feature.

[0053] In addition, for the same protocol, there will be multiple frame lengths due to different frame uses. Even for the same type of data frame, there will still be expandable fields, resulting in variable frame length. The inputs of byte-by-byte frequent item extraction and autoencoder are both fixed-length sequences, so the data frames need to be padded with zeros or truncated. Existing experiments have shown that almost all possible protocol features can be obtained when using the first 20 bytes for data mining. Therefore, in this embodiment, the first 32 bytes of each data frame can be intercepted to achieve length alignment.

[0054] Specifically, the preprocessed data frame is subjected to frequent item statistics byte by byte, which can be designed to include:

[0055] Count the frequency of occurrence of all elements contained in each byte of each data frame;

[0056] If the frequency of occurrence of an element is greater than the preset minimum support, the element is added to the frequent itemset to obtain the frequency of occurrence of each frequent item in the frequent itemset of each byte.

[0057] Among them, using the frequency of frequent items to filter out the key bytes of the protocol, and using the number of frequent items to filter out the key bytes of the protocol category, may include:

[0058] Sort the frequency and number of frequent items of each byte in the data frame;

[0059] According to the sorting results, the protocol keyword byte and the protocol category keyword byte are selected from the largest frequent item frequency and the largest frequent item quantity.

[0060] Although the traditional frequent item extraction based on Apriori and Eclat algorithms works well in association rule learning, they all have disadvantages such as large memory and computational overhead. Moreover, these algorithms can mine frequent items in different unknowns, but for protocol data frame clustering, only frequent items located on the same byte are meaningful, which leads to a lot of redundancy in traditional frequent item statistical algorithms; in addition, these frequent item extraction algorithms obtain variable-length sequences, while the autoencoder model generally requires input as a fixed-length sequence. Therefore, the traditional frequent item extraction algorithm has low applicability in unknown protocol clustering. Therefore, in the embodiment of this case, byte-by-byte frequent item extraction is used. Byte-by-byte frequent item extraction only performs frequent items for each byte, and does not consider the correlation between bytes. Compared with traditional algorithms, it has the characteristics of small computational complexity and is more suitable for unknown protocol clustering.

[0061] Byte-by-byte frequent item extraction counts the number of occurrences of all elements contained in each byte. If its occurrence frequency is greater than the set minimum support, it is included in the frequent item set. Through the above operation, the frequent item set of each byte and the frequency of occurrence of each frequent item can be obtained. The specific calculation formula can be expressed as follows:

[0062] IT={x∈A|(num(x) / N)>minsupport}

[0063] IT is the set of all frequent elements of the byte, A is the set of all elements of the byte, and minsupport is the minimum support set.

[0064] For any type of protocol, there are usually certain fixed bytes with corresponding rules, which can distinguish it from other protocols. These bytes can help us infer the type of protocol to a large extent, so they can be called protocol key bytes; similarly, for any category within the same protocol, there are usually certain fixed bytes with corresponding rules that can distinguish it from other categories. These bytes can also help infer the category, so they can be called category key bytes. For protocol key bytes, since they are common to a certain protocol, the frequency of their frequent item elements is usually large, and the protocol key bytes can be filtered out by the frequent item frequency. For category key bytes, since different protocols have multiple different categories, this will result in a large number of frequent items contained in the category key bytes, so the category key bytes can be filtered out according to the number of frequent items. The calculation formulas for the frequent item frequency and the number of frequent items can be expressed as follows:

[0065]

[0066] n it =len(IT)

[0067] where f it is the frequency of frequent items, n it is the number of frequent items, it i is the i-th frequent item in the byte, num(it i ) for it i The number of times this byte occurs.

[0068] Get each byte of f it and n it Then sort them by size, and f it The maximum 8 bytes are used as the protocol key byte. it The maximum 8 bytes are used as the category key byte.

[0069] Specifically, the frequency autoencoder and the quantity autoencoder both adopt a five-layer encoder structure, the first layer and the last layer of the five-layer encoder structure each include 8 neurons, the second layer and the penultimate layer each include 4 neurons, and the middle layer includes 2 neurons. The five-layer encoder structure is used to reduce the error between the reconstructed data and the input data, so that the middle layer completes the data dimensionality reduction processing by learning the low-dimensional representation of the input data.

[0070] After obtaining the protocol keyword byte and the category keyword byte, they are input into two autoencoders respectively. The two autoencoders have the same structure, and the only difference is the input data. The autoencoder that inputs the feature keyword byte is called the frequency autoencoder, and the autoencoder that inputs the category keyword byte is called the quantity autoencoder. Figure 2 As shown in the figure, the first and last layers of the autoencoder have 8 units, the second and penultimate layers have 4 units, and the middle embedding layer has 2 units. The autoencoder reduces the error between the reconstructed data and the input data, so that the embedding layer of the autoencoder can learn the low-dimensional representation of the input data and thus complete the dimensionality reduction of the data.

[0071] The training process of the frequency autoencoder and the quantity autoencoder is designed to include:

[0072] A weight matrix is ​​constructed according to the protocol data frame and using frequent item elements, and the mean square error is optimized using the weight matrix to obtain an autoencoder loss function;

[0073] The autoencoder is iteratively trained based on the autoencoder loss function and using sample data. The weight matrix is ​​initialized using the frequent item frequency before the first round of training so that the autoencoder can learn the features that focus on the key bytes of the protocol.

[0074] Autoencoders can map high-dimensional data to low-dimensional space while retaining the main features of the data. Therefore, they are widely used in data dimensionality reduction. Compared with traditional machine learning dimensionality reduction algorithms, autoencoder models can learn more complex data patterns.

[0075] Traditional autoencoder models usually use mean square error as their loss function. However, for autoencoders used for unknown protocol clustering, if the mean square error is directly used as the loss function, the autoencoder will indiscriminately learn all the features of the input data and fail to focus on frequent items that can distinguish unknown protocols. Therefore, it is necessary to optimize the loss function based on frequent items to make it more suitable for the task of unknown protocol clustering. Specifically, a weight matrix is ​​constructed for a batch of protocol data frames. If the element in the data frame is a frequent item element, the weight of the corresponding position is 1, otherwise it is 0. Figure 3 The weight matrix calculation is shown.

[0076] The new loss function is obtained by using weighted optimization of the mean square error. The specific calculation formula can be expressed as follows:

[0077]

[0078] Among them, w i is the i-th element of the weight vector of the data frame, x i is the i-th element of the frame data, is the i-th element of the reconstructed data, and N is the number of input data frames.

[0079] In addition, any neural network will have a weight initialization process before training, and the same is true for autoencoders. Weight initialization will directly affect the learning ability and training efficiency of the network. If the initialization is not good, it is easy to cause problems such as gradient disappearance, gradient explosion, and inaccurate learned features. The most commonly used initialization method today is He uniform initialization, and its weight matrix w obeys For unknown protocol clustering, it is hoped that the autoencoder can learn more frequent item features while avoiding gradient messages and gradient explosion. Therefore, in this embodiment, the frequent item frequency is used to optimize the first layer weight of the autoencoder, and its calculation formula can be expressed as follows:

[0080]

[0081] where w ij is the weight of the connection between the i-th neural network in the first layer of the autoencoder and the j-th neuron in the second layer, f it is the frequency set of frequent items, f iti is the first element in the set, n inis the number of neurons in the model. Through this weight initialization method, the autoencoder can be effectively guided to learn the features of frequent items and then perform subsequent clustering.

[0082] Specifically, when clustering features, the density-based clustering algorithm DBSCAN can be used to cluster the features. The clustering process can be designed to include:

[0083] Use grid search to set the two optimal parameters of radius and minimum number of points in the clustering algorithm DBSCAN based on the silhouette coefficient;

[0084] Randomly select an unvisited data point from the feature set, obtain all neighboring points of the unvisited data point within the radius parameter neighborhood, and classify the data point as a core point if the data point has at least one neighboring point that meets the minimum number of points parameter in the neighborhood;

[0085] Taking the core point as the cluster center, starting from the core point, recursively visit the core point's density-connected neighbor points, where the density-connected neighbor points are neighbor points within the radius parameter neighborhood;

[0086] The core points and their density-connected neighboring points are assigned to the same cluster, and the boundary points are assigned to the same cluster as the associated core points.

[0087] The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm clusters by defining core points (points with enough neighbors within a given radius), border points (points within the neighborhood of the core but not core points), and noise points. The algorithm first finds all core points, and then connects all adjacent core points to form clusters based on density reachability.

[0088] The features extracted by the frequency autoencoder are clustered using the DBSCAN algorithm to obtain coarse cluster labels, and then the coarse cluster labels are combined with the features extracted by the quantity autoencoder and a second DBSCAN clustering is performed to obtain fine cluster labels. The coarse cluster labels are obtained by clustering the protocol keywords, and they mainly contain protocol information, while the fine cluster labels are obtained by clustering the category keywords, and they mainly contain finer-grained category information. The specific two labels are as follows: Figure 4 shown.

[0089] Coarse clustering labels classify the same protocol data frames in unknown protocol data frames into one category, while fine clustering labels classify different categories of the same protocol into different categories. By combining fine clustering labels with coarse clustering labels, protocol clustering can be achieved while subdividing the categories within the protocol.

[0090] Further, based on the above method, an embodiment of the present invention also provides an unknown protocol clustering system based on frequent item extraction and a double-layer autoencoder, comprising: a data preprocessing module, a frequent item statistics module, a feature extraction module and a clustering analysis module, wherein:

[0091] A data preprocessing module, used for converting the received bit-form network protocol data frame into a byte representation and performing data preprocessing, wherein the data preprocessing includes: data frame deduplication processing and data frame length alignment processing;

[0092] A frequent item statistics module is used to perform frequent item statistics on the preprocessed data frame byte by byte to obtain the frequent item frequency and the number of frequent items of each byte of the data frame, wherein the frequent item is an element whose occurrence frequency in the byte is greater than a preset minimum support;

[0093] A feature extraction module is used to filter out protocol key bytes using the frequency of frequent items, filter out protocol category key bytes using the number of frequent items, and extract features of protocol key bytes and protocol category key bytes using a pre-trained frequency autoencoder and a pre-trained number autoencoder, respectively;

[0094] The clustering analysis module is used to coarsely cluster the features extracted by the frequency autoencoder to obtain a coarse protocol clustering label, merge the coarse protocol clustering label with the features extracted by the quantity autoencoder, and cluster the merged features to obtain a final output fine protocol clustering label, wherein the coarse protocol clustering label is a label containing protocol information obtained by clustering protocol keyword bytes, and the fine protocol clustering label is a label containing protocol category information obtained by clustering category keyword bytes.

[0095] In order to verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0096] The data set used in the experiment is shown in the following table:

[0097]

[0098] AIS comes from the data received in the actual communication network, and ARP, DNS, NBNS, and SMB come from the MACCDC 2012 dataset, which is the network traffic data collected during the National Network Watch Mid-Atlantic University Cyber ​​Defense Competition. The PCAP file of the dataset is deduplicated and preliminarily processed to obtain the above 8027 data frames. The traditional machine learning principal component analysis (PCA) algorithm, autoencoder, and the double-layer autoencoder based on frequent items in this case are used to extract features from the protocol data frames, and then the DBSCAN clustering algorithm is used for clustering. The ε and MinPts of the DBSCAN algorithm, and the F and purity commonly used in clustering are used as evaluation indicators.

[0099] The experimental results are as follows Figure 5 As shown in the figure, from the experimental results, it can be seen that compared with the other two algorithms, the F value and purity value of this scheme based on frequent item extraction autoencoder in unknown protocol clustering are higher, and the overall performance is better. Moreover, after clustering based on frequent item extraction autoencoder, the information at the protocol level can be retained, which is of great significance for the subsequent protocol reverse work.

[0100] In addition, due to the strong real-time nature of network communication data, the running time of the algorithm is also an important performance indicator. The above test set is also used to test the running time of the three algorithms. The specific results are shown in the following table:

[0101] Table running time statistics

[0102]

[0103] From the actual running data loss time, we can see that the traditional machine learning algorithm has certain advantages over deep learning in terms of running time. The running time of PCA is significantly shorter than that of the autoencoder, and compared with the autoencoder based on frequent item extraction in this case, although it still has certain advantages, the difference is not big. However, there is a significant gap between the clustering performance of PCA and the autoencoder based on frequent item extraction in this case. Therefore, in general, the unknown protocol clustering of the autoencoder based on frequent item extraction in this case has certain advantages over traditional algorithms.

[0104] In addition, the calculation time in the above table is calculated using the CPU. Nowadays, GPU is widely used in the model training process of deep learning due to its powerful parallel computing ability. If a GPU with better performance is used instead of the CPU for calculation, the calculation time of the frequent item extraction autoencoder in this case will also be shortened.

[0105] The above experimental data show that the solution in this case can retain protocol-level clustering while achieving category-level clustering by combining coarse clustering labels and fine clustering labels; in addition, compared with the traditional unknown protocol clustering invention, the computational complexity is small, and while ensuring the real-time clustering, it can effectively solve the problem of under-scoring of unknown protocol clustering, improve the clustering performance, and has good application prospects.

[0106] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0107] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0108] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0109] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0110] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for clustering unknown protocols based on frequent item extraction and double-layer autoencoder, characterized in that: Include: Convert the received bit-form network protocol data frame into a byte representation and perform data preprocessing, wherein the data preprocessing includes: data frame deduplication processing and data frame length alignment processing; Perform frequent item statistics on the preprocessed data frame byte by byte to obtain the frequency and number of frequent items of each byte of the data frame, wherein the frequent item is an element whose occurrence frequency in the byte is greater than the preset minimum support; The frequency of frequent items is used to filter out the key bytes of the protocol, the number of frequent items is used to filter out the key bytes of the protocol category, and the pre-trained frequency autoencoder and the pre-trained number autoencoder are used to extract features of the key bytes of the protocol and the key bytes of the protocol category respectively; The features extracted by the frequency autoencoder are coarsely clustered to obtain the coarse protocol clustering labels, the coarse protocol clustering labels are merged with the features extracted by the quantity autoencoder, and the merged features are clustered to obtain the final output fine protocol clustering labels, where the coarse protocol clustering labels contain protocol information obtained by clustering protocol keyword bytes, and the fine protocol clustering labels contain protocol category information obtained by clustering category keyword bytes.

2. The unknown protocol clustering method based on frequent item extraction and double-layer autoencoder according to claim 1 is characterized in that: Perform data preprocessing on the data frame, including: The data frame is processed with a designated bit as a unit, and multiple bits are clustered as one-dimensional features during feature clustering, wherein the designated bit is the number of bits corresponding to a single byte or multiple bytes; The data frame length is aligned by padding each data frame with zeros and / or truncating the data frame.

3. The unknown protocol clustering method based on frequent item extraction and double-layer autoencoder according to claim 1 is characterized in that: Perform frequent item statistics on the preprocessed data frame byte by byte, including: Count the frequency of occurrence of all elements contained in each byte of each data frame; If the frequency of occurrence of an element is greater than the preset minimum support, the element is added to the frequent itemset to obtain the frequency of occurrence of each frequent item in the frequent itemset of each byte.

4. The unknown protocol clustering method based on frequent item extraction and double-layer autoencoder according to claim 1 or 3, characterized in that: The frequent item frequencies are used to filter out the key bytes of the protocol, and the frequent item quantities are used to filter out the key bytes of the protocol category, including: sorting the frequent item frequencies and the frequent item quantities of each byte in the data frame; According to the sorting results, the protocol keyword byte and the protocol category keyword byte are selected from the largest frequent item frequency and the largest frequent item quantity.

5. The unknown protocol clustering method based on frequent item extraction and double-layer autoencoder according to claim 1 is characterized in that: The frequency autoencoder and the quantity autoencoder both adopt a five-layer encoder structure, the first layer and the last layer of the five-layer encoder structure each include 8 neurons, the second layer and the penultimate layer each include 4 neurons, and the middle layer includes 2 neurons. The five-layer encoder structure is used to reduce the error between the reconstructed data and the input data, so that the middle layer completes the data dimensionality reduction processing by learning the low-dimensional representation of the input data.

6. The unknown protocol clustering method based on frequent item extraction and double-layer autoencoder according to claim 1 or 5, characterized in that: The training process of the frequency autoencoder and the quantity autoencoder comprises: A weight matrix is ​​constructed according to the protocol data frame and using frequent item elements, and the mean square error is optimized using the weight matrix to obtain an autoencoder loss function; The autoencoder is iteratively trained based on the autoencoder loss function and using sample data. The weight matrix is ​​initialized using the frequent item frequency before the first round of training so that the autoencoder can learn the features that focus on the key bytes of the protocol.

7. The unknown protocol clustering method based on frequent item extraction and double-layer autoencoder according to claim 1 is characterized in that: When clustering features, the density-based clustering algorithm DBSCAN is used to cluster the features. The clustering process includes: Use grid search to set the two optimal parameters of radius and minimum number of points in the clustering algorithm DBSCAN based on the silhouette coefficient; Randomly select an unvisited data point from the feature set, obtain all neighboring points of the unvisited data point within the radius parameter neighborhood, and classify the data point as a core point if the data point has at least one neighboring point that meets the minimum number of points parameter in the neighborhood; Taking the core point as the cluster center, starting from the core point, recursively visit the core point's density-connected neighbor points, where the density-connected neighbor points are neighbor points within the radius parameter neighborhood; The core points and their density-connected neighboring points are assigned to the same cluster, and the boundary points are assigned to the same cluster as the associated core points.

8. An unknown protocol clustering system based on frequent item extraction and double-layer autoencoder, characterized in that: It includes: data preprocessing module, frequent item statistics module, feature extraction module and cluster analysis module, among which, A data preprocessing module, used for converting the received bit-form network protocol data frame into a byte representation and performing data preprocessing, wherein the data preprocessing includes: data frame deduplication processing and data frame length alignment processing; A frequent item statistics module is used to perform frequent item statistics on the preprocessed data frame byte by byte to obtain the frequent item frequency and the number of frequent items of each byte of the data frame, wherein the frequent item is an element whose occurrence frequency in the byte is greater than a preset minimum support; A feature extraction module is used to filter out protocol key bytes using the frequency of frequent items, filter out protocol category key bytes using the number of frequent items, and extract features of protocol key bytes and protocol category key bytes using a pre-trained frequency autoencoder and a pre-trained number autoencoder, respectively; The clustering analysis module is used to coarsely cluster the features extracted by the frequency autoencoder to obtain a coarse protocol clustering label, merge the coarse protocol clustering label with the features extracted by the quantity autoencoder, and cluster the merged features to obtain a final output fine protocol clustering label, wherein the coarse protocol clustering label is a label containing protocol information obtained by clustering protocol keyword bytes, and the fine protocol clustering label is a label containing protocol category information obtained by clustering category keyword bytes.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 can be implemented.

Images