A semi-supervised clustering based unknown protocol identification method and system

Abstract

The application discloses a kind of unknown protocol identification method and system based on semi-supervised clustering, the method includes: collecting different protocol labeled and unlabeled traffic data and extracting flow statistical feature vector and fingerprint feature vector;According to flow correlation, construct constraint information using labeled and unlabeled data, obtain must connect constraint set, do not connect constraint set, equivalence class set;Using must connect constraint set and do not connect constraint set, the Laplace score of each flow statistical feature is calculated to select features, and the flow statistical feature after feature selection is fused with fingerprint feature to obtain single flow feature vector;With equivalence class set, do not connect constraint information as guide, labeled and unlabeled single flow is mixed to carry out semi-supervised clustering;Using the traffic cluster after clustering constructs classifier.The application can more accurately represent the multi-dimensional characteristics of network traffic through feature selection and feature fusion.In the case of data scarcity, potential information in unlabeled data is mined by constructing constraint information, the identification effect of unknown protocol traffic is improved, and the dependence on labeled data is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network protocol identification, and particularly relates to an unknown protocol identification method and system based on semi-supervised clustering, which is suitable for network traffic analysis, network security protection, unknown protocol dynamic identification and the like scenes. BACKGROUND

[0002] In the scenes of industrial control, Internet of Things, military communication and the like, in order to meet the requirements of specific functions, transmission performance, privacy protection and the like, many network applications use customized private unknown protocols. However, due to the lack of prior knowledge, the network traffic generated thereby is difficult to be supervised by conventional methods, and these protocols often do not undergo sufficient security testing, and may have design defects, and are easy to become the target of hacker attacks, thereby posing a potential threat to the cyberspace.

[0003] Unknown protocol identification aims to determine the traffic belonging to the known protocol categories without prior knowledge or with only a small amount of prior knowledge, and to mine the private unknown protocol traffic, which has important value for network intrusion detection, malicious software detection, user behavior analysis and protocol reverse engineering and the like applications. However, the traditional identification methods based on ports and deep packet inspection technology are insufficient to cope with this challenge. The identification method based on ports mainly relies on detecting the transmission layer port of the traffic to identify the protocol type of the traffic, and the identification speed is very fast. However, many modern protocols use dynamic port allocation strategies, and the protocol traffic may be transmitted on any port, so the port information cannot accurately reflect the protocol type, and malicious traffic and some encrypted protocols may be disguised as common ports (such as 80, 443 and the like) to transmit data, thereby evading security detection and making the identification based on ports invalid. Deep packet detection needs to analyze the load content of each data packet, and the calculation overhead of this method is large, and it is difficult to process massive data in real time, especially in a dynamic network environment. With the continuous emergence of emerging protocols, the deep packet inspection technology usually relies on rule library or signature library for analysis, and the update lag of the rule library and the insufficient ability to analyze the contents of the unseen protocols limit the adaptability of the deep packet inspection technology to unknown protocols. Although machine learning and deep learning technologies have achieved remarkable results in the identification of known protocol traffic, they often rely on the need for a large amount of labeled data, and the cost of obtaining these data is high. In view of this, the unknown protocol identification problem has attracted widespread attention, and the current mainstream methods are divided into unsupervised methods of direct clustering, semi-supervised methods combined with known protocol information, and self-supervised learning based methods.

[0004] Unsupervised methods cluster by similarity between samples without label information or prior knowledge, but lack of label guidance, the results need to be analyzed to explain, increase the complexity of practical application. Some research combines self-supervised learning, supervised and unsupervised hybrid method for identification. The features generated by self-supervised learning already contain a lot of context information in the pre-training stage, which can improve the feature separation effect of unsupervised clustering and supervised classifier, and promote the identification accuracy of protocol type. However, its effect depends on whether the pre-training task design is reasonable. If the task constructed cannot capture the key features between protocols, it will directly affect the effect of the subsequent steps. Semi-supervised methods guide clustering through a small amount of labeled samples or constraint information, but existing research is difficult to construct the most representative features, and the prior knowledge contained in the labeled samples is not fully utilized and cannot achieve good results when the number of clustering categories is set small or the parameters are not selected properly. SUMMARY

[0005] The application provides an unknown protocol identification method and system based on semi-supervised clustering, which aims to solve the problems of unknown protocol identification difficulty, labeled sample scarcity and insufficient utilization of prior knowledge.

[0006] The application discloses an unknown protocol identification method based on semi-supervised clustering, comprising the following steps:

[0007] Step one: extracting data features of labeled and unlabeled protocol traffic samples;

[0008] Step two: constructing constraint information according to the data features;

[0009] Step three: filtering and fusing the data features of each traffic sample using the constraint information to obtain new data features;

[0010] Step four: using the new data features as input to perform semi-supervised clustering using the K-means clustering algorithm, and improving the K-means clustering algorithm using the constraint information in the clustering process to obtain the clustered traffic clusters;

[0011] Step five: constructing a classifier using the clustered traffic clusters as training data.

[0012] Preferably, the step one specifically comprises:

[0013] Collecting labeled and unlabeled traffic samples of different protocols;

[0014] Dividing the data packets with the same source IP address, target IP address, source port, target port and transmission layer protocol in the traffic sample into the same flow;

[0015] Extracting the data features of each flow.

[0016] Preferably, the data features of each flow include a statistical feature vector and a fingerprint feature vector.

[0017] Preferably, the statistical features include at least basic flow information (source IP, destination IP, protocol type, flow duration, source port, destination port), packet size (single flow and maximum, minimum, average, standard deviation of uplink and downlink packet size), packet number (packet number of single flow, packet number of uplink and downlink in single flow and their ratio), packet time interval (single flow and maximum, minimum, average, standard deviation of uplink and downlink packet arrival time interval), rate (flow rate of single flow, packet rate of uplink and downlink in single flow and their ratio, unit: byte / s), packet rate (flow rate of single flow, packet rate of uplink and downlink in single flow and their ratio, unit: number / s), byte (byte number of single flow, byte number of uplink and downlink in single flow and their ratio), packet fragmentation (maximum, minimum, average, standard deviation of uplink and downlink fragmentation number), which are combined into a statistical feature vector.

[0018] Preferably, the fingerprint features are a set of frequent item sets that can represent the communication mode of a specific protocol, including location information and content information of the frequent items. These frequent items reflect the commonality and regularity of data traffic generated under the same protocol in the communication process, for example, the repeated appearance of specific application layer protocol field values. Usually, these fields contain some semantic information, such as type field, Flag field, protocol identifier, version number, etc.

[0019] Preferably, the constraint information includes a set of must-link constraints, a set of cannot-link constraints, and a set of equivalence classes.

[0020] Preferably, the step two specifically includes:

[0021] For any two labeled flows with the same label in the traffic sample or any two related flows, add them as a must-link constraint to the set of must-link constraints;

[0022] For two labeled flows with different labels, add them as a cannot-link constraint to the set of cannot-link constraints;

[0023] Classify the flows with indirect association in the set of must-link constraints into the same equivalence class to obtain the set of equivalence classes.

[0024] Preferably, the correlation is that for two flows, if their destination IP, destination port, and transport layer protocol are the same and the time interval is within a certain range, then the two flows have correlation.

[0025] Preferably, the step three specifically includes:

[0026] constructing a neighbor graph using k-NN algorithm and calculating the weight matrix of the neighbor graph by using the statistical feature vector of the flow samples and the must-link constraint set;

[0027] constructing a negative constraint graph and calculating the weight matrix of the negative constraint graph by using the statistical feature vector of the flow samples and the cannot-link constraint set;

[0028] calculating the Laplacian matrix of the neighbor graph and the negative constraint graph respectively through the weight matrix of the neighbor graph and the negative constraint graph;

[0029] calculating the Laplacian score of each feature according to the Laplacian matrix and the statistical feature vector of the flow samples, and removing the flow statistical features lower than the threshold to obtain the statistical feature vector after feature selection;

[0030] fusing the statistical feature vector after feature selection and the fingerprint feature vector to obtain the new data features of each flow sample.

[0031] Preferably, the fusion of the statistical feature vector after feature selection and the fingerprint feature vector is performed by a serial fusion splicing method.

[0032] Preferably, the constructing of the neighbor graph using k-NN algorithm and the calculation of the weight matrix of the neighbor graph by using the statistical feature vector of the flow samples and the must-link constraint set specifically include:

[0033] For any two samples i and j, if the sample pair (i, j) belongs to the must-link constraint set or i is one of the k nearest neighbors of j, then i and j have a common edge in the neighbor graph;

[0034] For the samples i and j having a common edge in the neighbor graph, the value w ij in the weight matrix is calculated. The calculation formula is:

[0035]

[0036] where x i and x j are the statistical feature vectors of the samples i and j, ||x i -x j || 2 is the square of the Euclidean distance between the samples i and j, indicating their similarity. λ is a constant parameter for controlling the degree of similarity decay. Usually, λ is a hyperparameter selected by experiment or cross-validation;

[0037] For the samples i and j without a common edge in the neighbor graph, the value in the weight matrix is 0.

[0038] Preferably, the negative constraint graph is constructed and the weight matrix of the negative constraint graph is calculated using the traffic sample statistical feature vector and the disconnection constraint set, and the specific steps include:

[0039] For any two samples i and j, if the sample pair (i, j) belongs to the disconnection constraint set, i and j have a common edge in the negative constraint graph;

[0040] For samples i and j having a common edge in the negative constraint graph, the value of i and j in the weight matrix is 1;

[0041] For samples i and j not having a common edge in the negative constraint graph, the value of i and j in the weight matrix is 0.

[0042] Preferably, the step four specifically includes:

[0043] Randomly selecting one flow from each equivalence class as an initial cluster center;

[0044] Selecting the next cluster center according to the weighted probability distribution of the square of the distance of each flow in the traffic sample to the selected cluster center, and repeating this step until a predetermined number of initial cluster centers are selected;

[0045] Assigning each flow in the traffic sample to the respective cluster center;

[0046] Updating the cluster center, and then continuing to assign each flow in the traffic sample to the respective cluster center until convergence or the number of iterations reaches a predetermined amount.

[0047] Preferably, the next cluster center is selected according to the weighted probability distribution of the square of the distance of each data point to the selected cluster center, and the probability of each flow being selected as the next cluster center is:

[0048]

[0049] wherein v i is the new data feature of the traffic sample i, D(v i ) is the Euclidean distance of the traffic sample i to the nearest cluster center, and X is the set of new data features of all traffic samples.

[0050] Preferably, assigning each flow in the traffic sample to the respective cluster center specifically includes:

[0051] For the first single flow, selecting the traffic cluster in which the nearest cluster center to the first single flow is located as the traffic cluster to which the first single flow is to be assigned;

[0052] Judging whether the first single flow and all second single flows in the traffic cluster to be assigned violate the disconnection constraint, and if the disconnection constraint is violated, finding the next nearest cluster center until the disconnection constraint is not violated;

[0053] If the do not connect constraint is not violated, the first single flow is assigned to the corresponding flow cluster, and all third single flows in the equivalence class to which the first single flow belongs are directly assigned to the same flow cluster as the first single flow.

[0054] The present invention also provides a system corresponding to an unknown protocol identification method based on semi-supervised clustering. The system includes a traffic preprocessing module, a constraint information construction module, a multi-dimensional feature construction module, a semi-supervised clustering module, and a classifier construction module.

[0055] The traffic preprocessing module is used to split the collected labeled and unlabeled traffic samples and extract the statistical features and fingerprint features of each flow respectively.

[0056] The constraint information construction module is used to construct a set of mandatory connection constraints, a set of non-mandatory connection constraints, and a set of equivalence classes based on the labels of the marked single streams and the correlation between two different single streams.

[0057] The multidimensional feature construction module is used to calculate the constrained Laplace score of each statistical feature based on the constraint information, retain statistical features with scores higher than the threshold and fuse them with fingerprint features to obtain the feature vector of each traffic sample.

[0058] The semi-supervised clustering module uses constraint information to improve the K-means algorithm clustering process and clusters traffic samples to obtain multiple traffic clusters;

[0059] The classifier building module is used to build an NCC classifier based on traffic clusters.

[0060] The present invention also provides a computer device, characterized in that the device comprises: a storage medium storing a computer program for performing unknown protocol identification; a processor for executing the computer program; the computer program causing the processor to perform the steps of any of the methods described above.

[0061] Preferably, the storage medium includes a hard disk, a flash drive, or other non-volatile storage devices.

[0062] Beneficial effects:

[0063] This invention employs a semi-supervised clustering method, combining constraint information constructed from known protocol samples and traffic correlations. This allows for training using only a small number of labeled samples, fully utilizing unlabeled samples and effectively alleviating the problem of scarce data labeling. Compared to existing technologies, semi-supervised learning methods can leverage constraint information to enhance the clustering process, especially when dealing with unknown protocol traffic, resulting in more accurate and robust clustering results.

[0064] Statistical features can capture the basic data patterns of traffic, while fingerprint features can identify protocol-specific behavioral patterns. This invention extracts traffic statistical features and protocol fingerprint features, and constructs constraint information by combining a small number of known protocol samples with flow correlation. It uses Laplace scores to measure the "importance" of each feature to retain those with higher discriminative power, thus providing a more comprehensive characterization of network traffic behavior. Furthermore, this method considers the interrelationships and constraint information between features, refining and enhancing the representational power of features. It is more precise than methods that simply use metrics (such as information gain, variance, etc.) to remove redundant or irrelevant features, thereby improving the accuracy of clustering.

[0065] Regarding the initial cluster center selection strategy, if the initial cluster centers are not selected appropriately, the K-means algorithm may get stuck in a local optimum instead of finding the global optimum. This invention utilizes a constructed set of equivalence classes, changing the previous method of randomly selecting initial cluster centers in the K-means algorithm. It randomly selects samples from each equivalence class with labeled samples as initial cluster centers, and then selects the next cluster center based on a weighted probability distribution of the squared distance from each data point to the selected cluster center. This improves the representativeness of the initial cluster centers, enabling the clustering algorithm to find suitable cluster centers in fewer iterations, while reducing the algorithm's dependence on the initial center selection and enhancing the model's robustness. In the sample allocation process, a distance- and constraint-based allocation mechanism is adopted to ensure that each traffic sample does not violate the constraints during allocation, avoiding incorrect clustering. Simultaneously, the partitioning of equivalence classes ensures consistent clustering of similar traffic samples, effectively reducing the possibility of misclassification. This constraint-information-guided sample allocation method not only enhances the correlation of clustering results, but also better copes with complex network traffic, especially for the identification of unknown protocols and mutated traffic, and has stronger adaptability and robustness.

[0066] In summary, this invention effectively utilizes prior information from labeled samples, reducing reliance on labeled samples and improving the accuracy and efficiency of unknown protocol identification. This scheme achieves high accuracy and purity in identifying unknown protocols even with a small number of labeled samples and a low number of clusters. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the method described in this invention.

[0068] Figure 2 This is a framework diagram of the system corresponding to the semi-supervised clustering-based unknown protocol identification method.

[0069] Figure 3 This is a schematic diagram of the structure of a computer device according to the present invention.

[0070] Figure 4 This is a schematic diagram comparing the clustering method of this invention with other clustering methods.

[0071] Figure 5 This is a schematic diagram comparing the clustering purity of the clustering method of this invention with other semi-supervised clustering methods.

[0072] Figure 6 This is a schematic diagram comparing the accuracy of the classifier of this invention with other semi-supervised classifiers.

[0073] Figure 7 This diagram illustrates the impact of the number of labeled streams on classifier accuracy.

[0074] Specific implementation methods

[0075] The embodiments of the present invention will now be described with reference to the accompanying drawings. These examples are illustrative of the invention but are not intended to limit its scope.

[0076] Example: An unknown protocol identification method based on semi-supervised clustering. Figure 1 This is a flowchart of the unknown protocol identification method based on semi-supervised clustering of the present invention. The method includes the following steps:

[0077] S1: Extract data features from traffic samples of labeled and unlabeled protocols;

[0078] Step S1 specifically includes the following steps:

[0079] S11: Collect labeled and unlabeled traffic samples for different protocols. This step uses Wireshark traffic acquisition software to collect traffic data for labeled and unlabeled protocols of different protocols. Since the duration of data streams generated by most unknown protocols usually depends on the duration of user behavior, and the statistical characteristics of streams within the same time period are very similar, a timeout is set for all sample streams. That is, for a data stream, packets whose duration exceeds the timeout are removed. In this embodiment, the timeout is 120 seconds.

[0080] S11: Group data packets with the same source IP address, destination IP address, source port, destination port, and transport layer protocol from the traffic sample into the same flow. This step uses the Splitcap tool to divide the collected data according to the source IP address, destination IP address, source port, destination port, and transport layer protocol.

[0081] S12: Extract data features for each flow. The data features for each flow include a statistical feature vector and a fingerprint feature vector. These include at least basic flow information (source IP, destination IP, protocol type, flow duration, source port, destination port), packet size (maximum, minimum, average, and standard deviation of uplink and downlink packet sizes for a single flow), packet quantity (number of packets in a single flow, number of uplink and downlink packets in a single flow, and their ratio), packet time interval (maximum, minimum, average, and standard deviation of uplink and downlink packet arrival time intervals for a single flow), rate (flow rate of a single flow, uplink and downlink packet rates in a single flow, and their ratio, in bytes / second), packet rate (flow rate of a single flow, uplink and downlink packet rates in a single flow, and their ratio, in numbers / second), bytes (number of bytes in a single flow, number of uplink and downlink bytes in a single flow, and their ratio), and packet fragmentation (maximum, minimum, average, and standard deviation of the number of uplink and downlink fragments). These statistical features are combined to form a statistical feature vector. Table 1 below shows the flow statistics features of each single flow extracted in this embodiment.

[0082] Table 1 Flow Statistical Characteristics

[0083]

[0084] The fingerprint feature is a set of frequent items that represent a specific protocol communication pattern, including the location and content information of the frequent items, which are combined to form a fingerprint feature vector. These frequent items reflect the commonalities and regularities of data traffic generated under the same protocol during communication, such as the repeated occurrence of specific application layer protocol field values. These fields usually contain semantic information, such as type fields, flag fields, protocol identifiers, version numbers, etc. This step extracts 1-byte frequent items from the first 32 bytes of the application layer payload and records the position of the frequent items. Subsequently, the frequent items are sorted according to the number of times they appear simultaneously in the data stream, and representative frequent items and their positions are selected to construct the final fingerprint feature vector.

[0085] S2: Construct constraint information based on data characteristics;

[0086] Step S2 specifically includes the following steps:

[0087] S21: For any two labeled flows with the same label or any two related flows in the traffic sample, add them as a mandatory connection constraint to the mandatory connection constraint set. The two related single flows are defined as follows: if the destination IP, destination port, and transport layer protocol are the same, and the time interval between them is within a predetermined range, then these two flows are related. In this embodiment, the predetermined range is one minute.

[0088] S22: For two labeled flows with different labels, add them as a single non-connection constraint to the set of non-connection constraints;

[0089] S23: Group the flows with indirect associations in the set of mandatory connections into the same equivalence class to obtain the set of equivalence classes.

[0090] S3: Use constraint information to filter and fuse the data features of each traffic sample to obtain new data features.

[0091] Step S3 specifically includes the following steps:

[0092] S31: Construct a nearest neighbor graph using the k-nearest neighbor algorithm based on the statistical feature vectors of the traffic samples and the set of mandatory connections, and calculate the weight matrix of the nearest neighbor graph. In this step, for any two samples i and j, if the sample pair (i,j) belongs to the set of mandatory connections or i is one of the k neighbors with the smallest distance to j, then i and j have a common edge in the nearest neighbor graph; for samples i and j with a common edge in the nearest neighbor graph, calculate their value w in the weight matrix. ij The calculation formula is:

[0093]

[0094] Where x i and x j These are the statistical feature vectors of sample i and sample j, respectively, ||x i -x j || 2 The similarity is represented by the squared Euclidean distance between samples i and j. λ is a constant parameter that controls the degree of similarity decay. Typically, λ is chosen as a hyperparameter through experimentation or cross-validation. For samples i and j that have no common edge in the nearest neighbor graph, their value in the weight matrix is ​​0.

[0095] S32: Construct a negative constraint graph using the statistical feature vectors of the flow samples and the set of non-connection constraints, and calculate the weight matrix of the negative constraint graph. For any two samples i and j, if the sample pair (i, j) belongs to the set of non-connection constraints, then i and j have a common edge in the negative constraint graph; for samples i and j that have a common edge in the negative constraint graph, their value in the weight matrix is ​​1, and for samples i and j that do not have a common edge in the negative constraint graph, their value in the weight matrix is ​​0.

[0096] S33: Calculate the Laplacian matrix of the nearest neighbor graph and the negative constraint graph using the weight matrices of the nearest neighbor graph and the negative constraint graph, respectively. The calculation formula is as follows:

[0097] L kn =D kn -S kn

[0098] LCL =D CL -S CL

[0099] Among them, S kn S is the nearest neighbor graph weight matrix. CL Here, D is the weight matrix of the negative constraint graph, and D is a diagonal matrix. L kn and L CL These are the Laplacian matrices for the nearest neighbor graph and the negative constraint graph, respectively.

[0100] S34: Based on the Laplace matrix and the statistical feature vectors of the flow samples, calculate the Laplace score for each feature and remove flow statistical features below the threshold to obtain the selected statistical feature vector. The formula for calculating the constrained Laplace score in this step is:

[0101]

[0102] Among them, f r This is a vector composed of the values ​​of feature r in all samples. The threshold is selected based on the impact of the number of features on the experimental results. In this embodiment, the constrained Laplace scores of each feature are arranged from high to low, and the constrained Laplace score of the 40th feature is selected as the threshold.

[0103] S35: The selected statistical feature vector is fused with the fingerprint feature vector to obtain new data features for each traffic sample. Feature fusion is performed using a combined splicing method in this step.

[0104] S4: Using the new data features as input, perform semi-supervised clustering using the K-means clustering algorithm. At the same time, use constraint information to improve the K-means clustering algorithm during the clustering process to obtain the clustered traffic clusters.

[0105] Step S4 specifically includes the following steps:

[0106] S41: Randomly select one flow from each equivalence class as the initial cluster center.

[0107] S42: Select the next cluster center using a weighted probability distribution based on the squared distance of each flow in the flow sample to the already selected cluster center. The distance mentioned in this step is the Euclidean distance, and the probability of each single flow being selected as the next cluster center is:

[0108]

[0109] Among them, v i For traffic sample i, new data features, D(v) iLet be the Euclidean distance from traffic sample i to its nearest cluster center, and let X be the new set of data features for all traffic samples.

[0110] S43: Repeat step S42 until a predetermined number of initial cluster centers are selected.

[0111] S44: Assign each flow in the flow sample to its respective cluster center. In this step, for each single flow v i Select the flow cluster C containing the cluster center with the smallest distance. j The formula for assigning this single flow to the flow cluster is:

[0112]

[0113] Where μ j C j Cluster centers, μ g Let p be the cluster center of other traffic clusters, and p be the number of cluster centers. Before allocation, it is first determined whether the single flow violates the do-not-connect constraint with all single flows in the traffic cluster to be allocated. If the do-not-connect constraint is violated, the next nearest cluster center is found until the do-not-connect constraint is not violated. If the do-not-connect constraint is not violated, the single flow is allocated to the corresponding traffic cluster. At the same time, the equivalence class to which the currently successfully allocated single flow belongs is determined, and all single flows in the equivalence class are directly assigned to the same traffic cluster as the current sample.

[0114] S45: Update cluster centers. The specific formula for updating cluster centers μ is as follows:

[0115]

[0116] S46: Repeat steps S44-S45 until convergence or the number of iterations reaches the predetermined amount.

[0117] S5: Construct a classifier using the clustered traffic clusters as training data. Specifically, first obtain the data for each traffic cluster C. i The number of labeled single-stream feature vectors n i If n i A value of 0 indicates that the flow cluster does not contain any labeled single-flow feature vectors, and therefore the flow cluster C is considered... i Mapped to an unknown traffic cluster; if n i If the value is not 0, then the statistical label is y. j The number of single-stream feature vectors n ij After the statistics are completed, the traffic cluster C is calculated. i The posterior probability P(Y=y) for each label j |C i ) = n ij / n i Ultimately, the traffic cluster C will be...i The traffic cluster mapped to label y, where

[0118] Assume the flow categories are represented as Ω = {ω1, ..., ω} q These categories are generated through a mapping between clustering results and predefined categories. Each traffic category ω... i Use the set of cluster centroids M belonging to this category i To represent: M i ={m j :C j ∈ω i Among them, C j Describe a cluster, m j This is the centroid of the cluster. For a test traffic x, the classification rule is:

[0119]

[0120] Thus, the method provided by this invention abandons the traditional approach of identifying unknown protocols based on port and deep packet inspection. Building upon the K-means clustering algorithm, it offers a new approach to identifying unknown protocols using semi-supervised learning. Addressing the challenges of the complexity of unsupervised learning applications, the need for reasonable pre-training task design in self-supervised learning, and the insufficient utilization of prior knowledge in semi-supervised learning, this scheme extracts traffic statistical features and protocol fingerprint features, combines them with a small number of known protocol samples and flow correlations to construct constraint information, and calculates constraint Laplace scores for feature selection and fusion, refining and enhancing the representational power of the features. By optimizing the initial K-means cluster center selection and sample allocation process through constraint information, it effectively utilizes prior information in labeled samples, reduces dependence on labeled samples, and improves the accuracy and efficiency of unknown protocol identification.

[0121] like Figure 2 As shown, the present invention provides an unknown protocol identification system based on semi-supervised clustering, including: a traffic preprocessing module, a constraint information construction module, a multi-dimensional feature construction module, a semi-supervised clustering module, and a classifier construction module.

[0122] The traffic preprocessing module is used to split the collected labeled and unlabeled data into streams and extract the statistical features and fingerprint features of each stream.

[0123] The constraint information construction module is used to construct a set of mandatory connection constraints, a set of non-mandatory connection constraints, and a set of equivalence classes based on the labels of the marked single streams and the correlation between two different single streams.

[0124] The multidimensional feature construction module is used to calculate the constrained Laplacian score of each statistical feature based on the constraint information, retain features with scores higher than the threshold and fuse them with fingerprint features to obtain feature vectors of labeled and unlabeled single streams.

[0125] The semi-supervised clustering module is used to mix labeled and unlabeled single flows, improve the K-means algorithm clustering process using constraint information, and cluster the mixed single flows to obtain multiple flow clusters;

[0126] The classifier building module is used to build an NCC classifier based on traffic clusters.

[0127] Figure 3 A schematic diagram of the physical structure of an electronic device is provided, the device including a processor and a storage medium. The storage medium stores a computer program for performing unknown protocol identification, and the processor executes the computer program to perform the method described above.

[0128] like Figure 4 , Figure 5 , Figure 6 , Figure 7 The diagram illustrates some experimental data from embodiments of the present invention. The improved clustering method of the present invention is compared with four other traditional clustering methods, and the classifier obtained by the present invention is compared with classifiers from other semi-supervised methods.

[0129] Figure 4 The horizontal axis in the bar chart, from left to right, corresponds to the clustering purity, standard mutual information, and adjusted Land coefficient for each method. The methods, from left to right, are: the improved clustering method of this invention, DBSCAN, K-means, GMM, and BRICH.

[0130] Figure 5 This paper demonstrates a comparison of the clustering performance of this invention with other semi-supervised clustering methods under different unknown protocols. Even when some samples lack label information, this invention can not only correctly classify unlabeled traffic but also segment traffic belonging to unknown protocols.

[0131] Figure 6 The results show the performance comparison between the present invention and other semi-supervised classifier methods.

[0132] Figure 7The dependence of each classifier on labeled data was further analyzed. Labeling a small number of samples with known protocols can improve the clustering effect, enabling effective identification and differentiation of traffic from unknown protocols. As the number of labeled samples increases, the evaluation metrics of each classifier improve because more labeled samples allow for accurate identification of more known clusters. This invention outperforms the comparative classifiers in all sample labeling scenarios because it utilizes flow correlation to refine the constraint information during constraint establishment. This not only allows for the allocation of more samples with the same protocol to the same cluster during clustering but also facilitates cluster mapping.

[0133] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0134] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A method for identifying unknown protocols based on semi-supervised clustering, characterized in that, The method includes the following steps: S1: Extract data features from traffic samples of labeled and unlabeled protocols; S2: Construct constraint information based on data characteristics; S3: Use constraint information to filter and fuse the data features of each traffic sample to obtain new data features; S4: Using the new data features as input, perform semi-supervised clustering using the K-means clustering algorithm. At the same time, use constraint information to improve the K-means clustering algorithm during the clustering process to obtain the clustered traffic clusters. S5: Use the clustered traffic clusters as training data to construct a classifier. Specifically, step S1 includes: S11: Collect labeled and unlabeled traffic samples from different protocols; S12: Group data packets with the same source IP address, destination IP address, source port, destination port, and transport layer protocol into the same flow in the traffic sample; S13: Extract the data features of each stream, which include statistical feature vectors and fingerprint feature vectors; Among them, the constraint information includes the set of mandatory connections, the set of non-mandatory connections, and the set of equivalence classes; Step S2 specifically includes: S21: For any two labeled flows with the same label or any two related flows in the flow sample, add them as a mandatory connection constraint to the mandatory connection constraint set; S22: For two labeled flows with different labels, add them as a single non-connection constraint to the set of non-connection constraints; S23: Group the flows with indirect associations in the set of mandatory connections into the same equivalence class to obtain the set of equivalence classes; Correlation is defined as follows: for two flows, if their destination IP, destination port, and transport layer protocol are the same and the time interval between them is within a certain range, then these two flows are correlated.

2. The unknown protocol identification method based on semi-supervised clustering according to claim 1, characterized in that, Statistical features include at least basic flow information, packet size, number of packets, packet time interval, rate, packet rate, bytes, and packet fragmentation. These statistical features are combined to form a statistical feature vector. A fingerprint feature is a set of frequent items that can represent a specific protocol communication pattern, including the location information and content information of the frequent items, which are combined to form a fingerprint feature vector. The information includes: basic flow information such as source IP, destination IP, protocol type, flow duration, source port, and destination port; packet size including the maximum, minimum, average, and standard deviation of the uplink and downlink packet sizes for a single flow; packet quantity including the number of packets in a single flow, the number of uplink and downlink packets in a single flow, and their ratio; packet time interval including the maximum, minimum, average, and standard deviation of the arrival time interval of uplink and downlink packets for a single flow; rate including the flow rate of a single flow, the uplink and downlink packet rates in a single flow, and their ratio, in bytes per second; packet rate including the flow rate of a single flow, the uplink and downlink packet rates in a single flow, and their ratio, in numbers per second; bytes including the number of bytes in a single flow, the number of uplink and downlink bytes in a single flow, and their ratio; and packet fragmentation including the maximum, minimum, average, and standard deviation of the number of uplink and downlink fragments.

3. The method for identifying unknown protocols based on semi-supervised clustering according to claim 1, characterized in that, Step S3 specifically includes: S31: Using the statistical feature vector of the traffic sample and the set of mandatory connection constraints, construct the nearest neighbor graph using the k-nearest neighbor algorithm and calculate the weight matrix of the nearest neighbor graph; S32: Construct a negative constraint graph using the statistical feature vector of the flow sample and the set of non-connection constraints, and calculate the weight matrix of the negative constraint graph; S33: Calculate the Laplacian matrix of the nearest neighbor graph and the negative constraint graph using the weight matrices of the nearest neighbor graph and the negative constraint graph, respectively; S34: Based on the Laplace matrix and the statistical feature vector of the flow samples, calculate the Laplace score of each feature and remove the flow statistical features below the threshold to obtain the statistical feature vector after feature selection. S35: Fuse the statistical feature vector after feature selection with the fingerprint feature vector to obtain new data features for each traffic sample; In this process, the statistical feature vector after feature selection is fused with the fingerprint feature vector using a serial fusion and splicing method.

4. The method for identifying unknown protocols based on semi-supervised clustering according to claim 3, characterized in that, Using the statistical feature vectors of traffic samples and the set of mandatory connection constraints, the k-nearest neighbor algorithm is used to construct a nearest neighbor graph and calculate the weight matrix of the nearest neighbor graph. Specifically, this includes: For any two samples i and j, if the sample pair (i, j) belongs to the set of mandatory connection constraints or i is one of the k neighbors with the smallest distance from j, then i and j have a common edge in the nearest neighbor graph. For samples i and j that share a common edge in the nearest neighbor graph, calculate their value w in the weight matrix. ij The calculation formula is: Where x i and x j These are the statistical feature vectors of sample i and sample j, respectively, ||x|| i -x j || 2 λ is the square of the Euclidean distance between sample i and sample j, representing their similarity. λ is a constant parameter used to control the degree of similarity decay. λ is a hyperparameter chosen through experiments or cross-validation. For samples i and j that have no common edge in the nearest neighbor graph, their value in the weight matrix is ​​0.

5. The method for identifying unknown protocols based on semi-supervised clustering according to claim 4, characterized in that, The negative constraint graph is constructed using the statistical feature vectors of the traffic samples and the set of non-connection constraints, and the weight matrix of the negative constraint graph is calculated. Specifically, this includes: For any two samples i and j, if the sample pair (i, j) belongs to the set of non-connection constraints, then i and j have a common edge in the negative constraint graph. For samples i and j that share a common edge in the negative constraint graph, their value in the weight matrix is ​​1; For samples i and j that have no common edge in the negative constraint graph, their value in the weight matrix is ​​0.

6. The method for identifying unknown protocols based on semi-supervised clustering according to claim 1, characterized in that, Step S4 specifically includes: S41: Randomly select one flow from each equivalence class as the initial cluster center; S42: Select the next cluster center by weighting the square of the distance from each flow in the flow sample to the selected cluster center using a probability distribution; S43: Repeat step S42 until a predetermined number of initial cluster centers are selected; S44: Assign each flow in the flow sample to its respective cluster center; S45: Update cluster centers; S46: Repeat steps S44-S45 until convergence or the number of iterations reaches the predetermined amount.

7. The method for identifying unknown protocols based on semi-supervised clustering according to claim 6, characterized in that, The next cluster center is selected using a weighted probability distribution based on the squared distance of each flow to the already selected cluster center. The probability that each flow is selected as the next cluster center is: Among them, v i For traffic sample i, new data features, D(v) i Let be the Euclidean distance from traffic sample i to its nearest cluster center, and let X be the traffic sample. Each flow in the traffic sample is assigned to its respective cluster center, specifically including: For the first single flow, the flow cluster containing the nearest cluster center is selected as the flow cluster to which the first single flow will be assigned; Determine whether the first single flow violates the do-not-connect constraint with all second single flows in the traffic cluster to be allocated. If the do-not-connect constraint is violated, find the next nearest cluster center until the do-not-connect constraint is not violated. If the do not connect constraint is not violated, the first single flow is assigned to the corresponding flow cluster, and all third single flows in the equivalence class to which the first single flow belongs are directly assigned to the same flow cluster as the first single flow.

8. An unknown protocol identification system based on semi-supervised clustering, characterized in that, The system for implementing the unknown protocol identification method based on semi-supervised clustering as described in any one of claims 1-7 comprises: a traffic preprocessing module, a constraint information construction module, a multi-dimensional feature construction module, a semi-supervised clustering module, and a classifier construction module. The traffic preprocessing module is used to split the collected labeled and unlabeled data into streams and extract the statistical features and fingerprint features of each stream. The constraint information construction module is used to construct a set of mandatory connection constraints, a set of non-mandatory connection constraints, and a set of equivalence classes based on the labels of the marked single streams and the correlation between two different single streams. The multidimensional feature construction module is used to calculate the constrained Laplacian score of each statistical feature based on the constraint information, retain features with scores higher than the threshold and fuse them with fingerprint features to obtain feature vectors of labeled and unlabeled single streams. The semi-supervised clustering module is used to mix labeled and unlabeled single flows, improve the K-means algorithm clustering process using constraint information, and cluster the mixed single flows to obtain multiple flow clusters.

9. A computer device, characterized in that, The device includes: Storage medium containing a computer program for performing unknown protocol identification; A processor for executing the computer program; The computer program causes the processor to implement the steps of the method as described in any one of claims 1 to 7. The storage medium includes hard disks, flash drives, or other non-volatile storage devices.

Images