Shape-based anti-noise encrypted traffic clustering method

Abstract

The invention discloses an anti-noise encrypted traffic clustering method based on shapes. The method comprises the following steps: 1) acquiring and analyzing original flow data to obtain a plurality of data streams; dividing uplink data and downlink data of each data stream according to a set time interval to obtain an uplink sequence U and a downlink sequence D corresponding to each data stream; 2) splicing the uplink sequence U and the downlink sequence D of each stream to form an integral sequence F which represents the behavior characteristics of the corresponding data stream; shape lines S and statistical features corresponding to the data streams are extracted from the behavior features of the data streams; 3) clustering the data streams based on the normalized shape lines S of the data streams, and clustering the data streams with the same behavior pattern into a cluster; 4) calculating clustering center features of each cluster based on statistical features; and then calculating the Euclidean distance between the statistical features of the data streams which are not clustered into any cluster and each clustering center feature, and determining the category of the corresponding data stream or whether the data stream is an abnormal point or not.

Description

Technical Field

[0001] The invention belongs to the technical field of network communication, in particular to the technical field of encrypted traffic analysis and traffic classification, and relates to a shape-based anti-noise encrypted traffic clustering method. Background Art

[0002] Network traffic classification technology is a basic technology in the field of network communications. Traditional methods mainly rely on port analysis and deep packet inspection (DPI), but these technologies are limited in effectiveness when facing encrypted traffic because the contents of encrypted data packets cannot be read directly. With the popularization of encryption technologies such as SSL / TLS, researchers have turned to methods based on machine learning and deep learning. These methods achieve classification by analyzing the statistical characteristics of traffic rather than direct content, thus showing advantages in processing encrypted traffic. However, these methods usually require a large amount of labeled data for model training, which is challenging in practical applications. The shape-based noise-resistant encrypted traffic clustering method proposed in this application proposal aims to solve the problem of dependence on a large amount of labeled data in the existing technology and achieve efficient classification of encrypted traffic through unsupervised learning.

[0003] Deep learning technology is a branch of machine learning that uses multi-layer neural networks to simulate the way the human brain processes information. In the field of network traffic classification, deep learning methods are used to automatically extract traffic features and classify them. Deep learning models are able to process high-dimensional data and learn complex patterns in the data, which is particularly useful for classifying encrypted traffic. However, the training of deep learning models usually requires a large amount of labeled data and consumes a lot of computing resources.

[0004] Unsupervised learning is a method in machine learning that does not require labeled data to train the model. Unsupervised learning algorithms discover natural clustering of data by exploring the intrinsic structure and patterns of the data. In network traffic classification, unsupervised learning can be used to identify unknown traffic types and can adapt to changes in the network environment. Unsupervised learning methods have potential when processing encrypted traffic because they do not rely on direct inspection of packet content, but rather on pattern recognition of traffic behavior.

[0005] Time series analysis is a method in statistics that analyzes data points that change over time. In network traffic classification, time series analysis can be used to capture the dynamic characteristics of traffic, such as changes in packet arrival rates. By analyzing these time series data, traffic can be more accurately identified and classified, especially when facing encrypted traffic, where these dynamic characteristics may become the key to distinguishing different applications.

[0006] Shape similarity metric is a method to evaluate the similarity of two time series data. In network traffic classification, shape similarity metric can be used to compare the time series of different flows to identify flows with similar behavior patterns. This method is particularly useful for identifying similar application patterns in encrypted traffic, because even if the data content is encrypted, the shape characteristics of the traffic can still be used for classification.

[0007] The following are encrypted traffic classification methods that mainly rely on supervised learning:

[0008] (1) Encrypted traffic classification methods based on traditional machine learning. These methods usually use support vector machines (SVM), neural networks, or incremental support vector machines (SVM) to classify traffic. When processing encrypted traffic, these methods rely on statistical features extracted from traffic data, such as packet size, transmission time interval, etc. However, these methods require a large amount of labeled data for model training, which may be difficult to obtain in practical applications, and it is difficult to effectively classify unknown traffic patterns.

[0009] (2) Encrypted traffic classification methods based on deep learning. With the development of deep learning technology, some researchers have begun to try to use deep learning models, such as convolutional neural networks (CNN) and recurrent neural networks (RNN), to deal with encrypted traffic classification problems. These methods can automatically learn features from data, reducing the dependence on large amounts of labeled data. However, deep learning models usually require a lot of computing resources, and for newly emerging traffic patterns, the model may need to be retrained to adapt.

[0010] The disadvantages of the prior art are as follows:

[0011] (1) It relies on a large amount of labeled data, which is difficult to obtain in real environments and cannot handle unknown traffic.

[0012] Existing encrypted traffic classification methods mainly rely on supervised learning, which requires a large amount of labeled data. For example, some methods use support vector machines (SVM) or neural networks to classify traffic. However, these methods perform poorly in dynamic and complex network environments because they require a predefined number of clusters and have difficulty adapting to new unknown traffic patterns.

[0013] (2) It ignores the temporal dynamic characteristics of traffic, such as packet arrival time and rate variation, which are crucial for accurately classifying encrypted traffic.

[0014] Although machine learning methods can solve many problems that port- and payload-based methods cannot, there are still some limitations: they cannot automatically extract and select features and need to rely on the experience of domain experts, resulting in great uncertainty when applying machine learning to encrypted traffic classification; features are prone to failure and need to be constantly updated.

[0015] (3) Unable to adapt to the changes in the behavior of different applications of the same service under different network conditions.

[0016] In real network environments, class imbalance is also an important issue in encrypted traffic classification, which directly affects classification accuracy. Moreover, since privacy protection and traffic annotation tools such as deep packet analysis tools cannot process encrypted traffic, it is difficult to legally collect and accurately annotate encrypted traffic datasets in a short time and at a low cost. Summary of the invention

[0017] In view of the problems existing in the prior art, the purpose of the present invention is to provide a shape-based noise-resistant encrypted traffic clustering method. The present invention does not rely on a large amount of labeled data, but captures traffic behavior patterns by analyzing the changes in the byte rate of the traffic. This method can effectively adapt to complex and dynamic network environments and can handle unknown traffic patterns, which is difficult to achieve with existing technical solutions. In addition, the method proposed in this application can resist noise interference such as network jitter and packet loss when processing encrypted traffic, thereby improving the accuracy and robustness of classification.

[0018] First, the method analyzes the byte rate changes of network traffic and captures traffic behavior patterns, thereby deeply exploring the characteristics of encrypted traffic. This method goes beyond the limitations of traditional supervised learning methods that rely on large amounts of labeled data and can automatically identify and classify network traffic without human intervention.

[0019] Secondly, by adopting the noise-resistant shape line extraction technology, this method can effectively resist noise interference such as network jitter and packet loss while retaining the core behavior characteristics of the traffic. This technology enables accurate analysis of traffic time series and improves the accuracy and robustness of classifying encrypted traffic in complex network environments.

[0020] In addition, this method designs a multi-dimensional feature extraction strategy that not only analyzes the uplink and downlink traffic characteristics, but also combines statistical and shape-based features to adapt to the changes in the behavior of different applications of the same service under different network conditions. This approach ensures that the traffic patterns of different applications can be accurately identified and clustered in a changing network environment.

[0021] Finally, this method proposes an unsupervised classification algorithm that combines shape-based density clustering and feature allocation strategy, overcomes the limitation of the number of predefined clusters in traditional methods, and can handle unknown traffic types. The proposal of this algorithm provides a new solution for processing large-scale, high-dimensional encrypted traffic data with strong scalability and adaptability.

[0022] The present invention mainly includes the following contents:

[0023] 1) Noise-resistant shape line extraction method

[0024] This application proposes a traffic feature extraction method that is effectively resistant to noise by designing a unique shape line extraction algorithm. This method can still accurately retain the core characteristics of traffic behavior when facing noise interference such as network jitter and packet loss. By removing outliers and applying smoothing, the shape line extraction method can filter out sudden noise and abnormal traffic fluctuations, so that the main shape characteristics of the traffic can be effectively restored. This noise-resistant design ensures the accuracy of the system's traffic feature extraction in complex network environments, greatly improves the robustness of classification, and maintains accuracy and stability even in high-noise or unstable network conditions.

[0025] 2) Multidimensional feature extraction strategy

[0026] In order to further improve the accuracy of traffic classification, this application proposes a multi-dimensional feature extraction strategy that combines statistical features and shape features. This method not only extracts the overall statistical information of the traffic from statistical features such as byte rate, but also combines the shape features of the traffic to obtain the temporal dynamic pattern. The multi-dimensional feature extraction strategy not only adapts to different network conditions, but also can accurately cluster different applications of the same service, ensuring that even in complex service scenarios, the system can accurately capture the unique characteristics of the traffic from multiple dimensions and classify it.

[0027] 3) Unsupervised clustering algorithm

[0028] This application designs an unsupervised clustering algorithm that combines shape similarity density clustering and feature allocation strategy to achieve an efficient clustering method without pre-defining the number of clusters. The introduced shape-based density clustering method can automatically identify different traffic types and process newly emerging unknown traffic, adapting to the needs of a changing network traffic environment. At the same time, this application further enhances the system's adaptability in the face of high-noise environments and complex traffic patterns through a feature allocation strategy. The method's automatic clustering mechanism effectively reduces manual intervention, allowing the system to still have powerful classification and recognition capabilities in ever-changing traffic patterns.

[0029] The technical solution of the present invention is:

[0030] A shape-based noise-resistant encrypted traffic clustering method, the steps of which include:

[0031] 1) Acquire and parse the original traffic data to obtain multiple data streams; divide the uplink data and downlink data of each data stream according to the set time interval to obtain the uplink sequence U and downlink sequence D corresponding to each data stream;

[0032] 2) Splice the uplink sequence U and the downlink sequence D of each data stream to form an overall sequence F, which represents the behavior characteristics of the corresponding data stream; extract the shape line S and statistical characteristics of the corresponding data stream from the behavior characteristics of each data stream;

[0033] Normalize the data in each shape line S;

[0034] 3) Clustering each data stream based on the normalized shape line S corresponding to each data stream, and clustering data streams with the same behavior pattern into one cluster;

[0035] 4) For each cluster obtained by clustering, the average value of the statistical characteristics of each data stream in the cluster is calculated as the cluster center feature of the corresponding cluster; then the Euclidean distance between the statistical characteristics of the data stream that is not clustered into any cluster in step 2) and each of the cluster center features is calculated. If the Euclidean distance between the statistical characteristics of the data stream i that is not clustered into any cluster and the cluster center feature of the jth cluster is the smallest and less than the set threshold δ, the data stream i is divided into the jth cluster; otherwise, the data stream i is marked as an abnormal point.

[0036] Furthermore, the method of extracting the shape line S of the corresponding data stream from the behavior characteristics of each data stream is: using a moving average algorithm with a sliding window to calculate the shape line S of the entire sequence F = (f1, f2, ..., f q ) for smoothing; if the zero value ratio of the uplink sequence U is greater than 50%, the sliding window size Otherwise the sliding window size For each point f in the overall sequence F t , and its corresponding point s on the shape line S t is a vector (f t-w+1 ,…,f t ), shape line S=(s w ,s w+1 ,…,s q ),

[0037] Furthermore, the statistical features include the average value of the downstream sequence D, the ratio of the average value of the upstream sequence U to the average value of the downstream sequence D, the proportion of zero values ​​in the upstream sequence U, and the total length of the overall sequence F.

[0038] Furthermore, the method for clustering each data stream based on the normalized shape line S corresponding to each data stream is:

[0039] 31) For two shape lines of unequal length and shape lines First make exist Slide relatively on the surface and calculate its inner product at each phase shift s x m Shape Line The mth data segment, y n Shape Line The nth data segment in;

[0040] 32) According to the inner product Calculate shape lines With shape line The normalized cross-correlation of

[0041] 33) According to the formula Calculate shape lines With shape line Similarity if If the shape line is smaller than the set threshold, Shape Line The corresponding data streams are grouped into the same cluster.

[0042] Furthermore, Among them, s∈[1-m,n-1].

[0043] Furthermore, the uplink sequence U={U1,U2,…,U K}, downlink sequence D = {D1, D2, ..., D K}; Among them, U k =∑ i payload i ·I[k×Δt≤time i <(k+1)×Δt], D k =∑ i payload i ·I[k×Δt≤time i <(k+1)×Δt],payload i is the byte size of the i-th data packet, time i is the arrival time of the ith data packet, Δt is the set time interval, k=1~K, K=T / Δt.

[0044] Furthermore, raw traffic data is captured and parsed in real time from network interfaces or traffic capture files.

[0045] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the above method.

[0046] A computer-readable storage medium stores a computer program, wherein the computer program implements the above method when executed by a processor.

[0047] The advantages of the present invention are as follows:

[0048] 1) No need for a large amount of labeled data

[0049] In response to the reliance of existing technologies on large amounts of labeled data, this application uses an unsupervised shape-based density clustering algorithm and an automatic clustering strategy based on shape similarity to perform efficient traffic classification without relying on labeled data. The algorithm does not require predefined categories, and automatically identifies clusters and classifies traffic behaviors by calculating the shape similarity of time series. This feature not only reduces the difficulty of obtaining labeled data, but also enables the system to better handle new and unknown traffic and has strong generalization capabilities. Compared with existing technologies that rely on labeled data, this solution can adaptively and accurately classify traffic in traffic environments where data changes frequently.

[0050] 2) Capturing the temporal dynamics of traffic

[0051] Existing static feature extraction methods have difficulty capturing the dynamic characteristics of traffic changing over time, which limits the accuracy of traffic classification. The present application solution can capture the temporal dynamic characteristics of traffic in detail by analyzing changes in traffic byte rate. Combined with the shape line extraction algorithm, the present application accurately reflects traffic behavior characteristics in the temporal dimension of traffic. Unlike traditional methods that rely solely on static features, temporal dynamic analysis can better adapt to complex traffic patterns and network fluctuations, especially in environments with more noise or dynamic changes, significantly improving classification accuracy and ensuring the accuracy of the system in complex scenarios.

[0052] 3) High robustness to changing network conditions

[0053] In view of the shortcomings of the prior art in that it has poor adaptability when network conditions change, the multi-dimensional feature extraction strategy proposed in this application combines statistical features and shape features, which can achieve accurate classification of traffic under different network conditions. By extracting multi-dimensional data of byte rate and shape features, this solution effectively responds to the impact of different network environments on traffic characteristics, especially when parameters such as network bandwidth and latency change, it can still maintain classification accuracy. In addition, whether it is the feature differences of the same service under different applications, or the behavioral changes of the same application under different network conditions, the system can accurately identify and classify, effectively enhancing the overall robustness and practicality of the solution. This high robustness enables the present application to adapt to the dynamic changes of the adaptive network environment in practical applications, and provide stable and accurate traffic classification effects.

[0054] The proposed method can resist noise interference such as network jitter and packet loss when processing encrypted traffic, and improves the accuracy and robustness of classification. This method can effectively classify encrypted traffic, improve the efficiency and security of network management, and has important practical significance for network operators and service providers. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0057] The shape-based noise-resistant encrypted traffic clustering method provides an efficient and reliable technical solution for the encryption classification of network traffic. By analyzing the byte rate changes of the uplink byte rate sequence U and the downlink byte rate sequence D, the scheme constructs a noise-resistant shape line F, which can effectively resist noise interference such as network jitter and packet loss, ensuring that the core characteristics of traffic behavior are preserved. At the same time, the scheme designs a multi-dimensional feature extraction strategy that combines the shape characteristics and statistical characteristics of the byte rate sequence, which can accurately identify and cluster different applications of the same service under different network conditions, thereby significantly improving the accuracy and adaptability of traffic classification. In addition, the scheme proposes an unsupervised algorithm that combines shape-based density clustering and feature allocation, which not only avoids the limitation of the traditional method on the predefined number of clusters, but also has the ability to handle unknown traffic types. This solution is particularly suitable for scenarios such as communication companies, terminal manufacturers, and mobile Internet companies that need to accurately classify traffic in a changing network environment, which helps traffic optimization, network security, and service quality assurance.

[0058] The shape-based anti-noise encrypted traffic clustering technology solution process is as follows Figure 1 As shown, it mainly includes the following steps:

[0059] 1. Data collection and preprocessing

[0060] In the data collection phase, the system uses a real-time stream processing framework, such as NFStream and Zeek, to capture and parse raw traffic data from network interfaces or traffic capture files in real time. NFStream writes customized traffic analysis scripts to parse all traffic packets one by one, extract basic attribute information (such as five-tuple: source IP, destination IP, source port, destination port, protocol type) to identify independent data flows, and record basic information such as timestamp, transmission direction (upstream or downstream), byte size, etc.

[0061] Specifically, the traffic data is divided into fixed time intervals Δt (e.g., 50 milliseconds) for segment statistics to generate the uplink and downlink byte rate sequences for each flow. For each time interval, the system accumulates the number of bytes of all data packets in the time period and stores them in the uplink sequence U = {U1, U2, ..., U K} and the downlink sequence D={D1,D2,…,D K Specifically, K is determined by the duration of the flow and the duration of each time interval. Assuming that the duration of the flow is T and the duration of each time interval is Δt, K can be expressed as K = T / Δt, which is the ratio of the flow duration to the time interval. At each time point k, the element U in the sequence k and D k Respectively represent the number of bytes in the uplink and downlink directions in the time interval [k×Δt, (k+1)×Δt):

[0062]

[0063] Among them, payload i Indicates the byte size of the i-th data packet, time i is the arrival time of the data packet. The indicator function I takes the value of 1 when the condition is met, otherwise it takes the value of 0. In this way, the data of each stream is represented as two time series in the time dimension, and the parsed data is directly recorded in the message queue Kafka, providing a reliable data stream channel for subsequent real-time processing.

[0064] (II) Behavioral feature extraction

[0065] a) Shape line feature extraction

[0066] In this stage, a noise-resistant shape line extraction method is used to obtain the shape line S of each flow from the upstream and downstream byte rate sequences of the flow, ensuring that the core characteristics of the traffic behavior are preserved in the presence of noise interference. This stage includes data cleaning and standardization. First, extreme values ​​(such as the top 0.1% peaks and valleys) are filtered out by setting a threshold, and then missing value interpolation is performed to ensure data integrity. In terms of data smoothing, the solution uses a sliding window-based moving average algorithm to eliminate random jitter. This smoothing process avoids frequent changes caused by network fluctuations by using an adaptive window size, while retaining the core shape characteristics of the traffic to the greatest extent. Finally, the generated shape lines are normalized to ensure consistency of different traffic data.

[0067] 1) Merge upstream and downstream sequences: The upstream and downstream byte sequences of each stream are concatenated to form an overall sequence F, which represents the behavioral characteristics of the stream.

[0068] 2) Noise processing: Filter out extreme values ​​that deviate from the mean (such as the top 0.1% of peaks or valleys in the data sequence) and perform linear interpolation on missing values. In this way, outliers that may be caused by factors such as network jitter are removed.

[0069] 3) Smoothing: Use a moving average algorithm with a small sliding window to smooth F. Set the sliding window size according to the length of the traffic sequence and the proportion of zero values ​​to eliminate random fluctuations. Specifically, if the proportion of zero values ​​in the upstream sequence U is greater than 50%, the window size is Otherwise, the window size is doubled, that is, the window size is Assume the window length is w, the step size is 1, for each point f in the sequence F of length q t , the corresponding point s on the shape line t is a vector (f t-w+1 ,…,f t ), the shape line S can be expressed as:

[0070] F=(f1,f2,…,f q )

[0071]

[0072] S=(s w ,s w+1 ,…,s q )

[0073] In this way, a smoothed shape line S can be generated.

[0074] 4) Normalization: Normalize the data in the shape line S corresponding to each data flow to make its mean 0 and variance 1, so as to ensure the consistency of different flow data.

[0075] b) Statistical feature extraction

[0076] In order to preserve the intrinsic statistical properties of the original byte rate sequences U and D, this scheme proposes a multi-dimensional feature extraction strategy. In addition to the shape line features, several key statistical features are extracted from each pair of sequences [U, D]. These features include the mean of the downstream sequence, the ratio of the mean of the upstream to the downstream sequence, the proportion of zero values ​​in the upstream sequence, and the total length of the entire sequence F. Table 1 describes the definition and calculation method of these features in detail. In order to ensure data consistency and improve the performance of the subsequent clustering algorithm, the extracted features are standardized to have a mean of 0 and a standard deviation of 1.

[0077] Table 1 shows the defined characteristics and calculation methods

[0078] Feature Name Calculation formula The average of the descending series mean(D) Ratio of the mean of the upward and downward series mean(U) / mean(D) The proportion of zero values ​​in the ascending sequence count(U[i]=0foralli) / len(U) The total length of the entire sequence len(U)+len(D)

[0079] (III) Shape-based clustering

[0080] The unsupervised clustering algorithm proposed in this application includes two stages, namely shape-based clustering and statistical feature-based allocation. In the shape clustering stage, in order to identify the shape similarity of traffic data, the scheme performs a similarity measure based on sliding cross-correlation. In this step, the scheme slides and matches shape line sequences of different lengths, and calculates the correlation coefficient at each time offset by dot product. The maximum cross-correlation value finally obtained is used to judge the shape similarity of the two sequences, and the similarity is further quantified by normalized cross correlation (NCC) to ensure the accuracy of the classification results. The scheme uses density clustering (such as DBSCAN) for clustering. This method defines cluster boundaries based on shape distance (Shape-based Distance, SBD), automatically identifies high-density areas and assigns traffic of similar shapes to the same cluster. This shape-based density clustering does not require the number of categories to be defined in advance when allocating boundaries, and is suitable for processing variable unknown encrypted traffic.

[0081] 1) Similarity calculation: For each shape line S, calculate its sliding cross correlation with other shape lines as a shape similarity measure. The dot product of each position is calculated by sliding and multiplying, so as to achieve shape matching between time series of different lengths. The calculated similarity result is represented by the normalized cross correlation (NCC), and its value range is [-1,1], where 1 means complete similarity and -1 means complete dissimilarity.

[0082] For two shape lines of unequal length and shape lines where x i and j Represents shape lines and The values ​​at positions i and j. The subscripts m and n are the shape lines. and The length of m≠n, that is, the lengths of the two shape lines are not equal. First, exist Slide relatively upward and calculate its inner product at each phase offset s, which is defined as follows:

[0083]

[0084] For all possible offsets s∈[1-m,n-1], calculate the inner product of the two shape lines at phase offset s This shows the similarity:

[0085]

[0086] Normalized cross correlation (NCC) is often used to limit the value to the range of [-1,1]. The shape distance (SBD) based on NCC is defined between 0 and 2. The smaller the SBD value, the higher the similarity of the two shape lines. The specific formula is as follows:

[0087]

[0088] 2) Density clustering: Density clustering is used to identify high-density areas and to separate them with low-density areas to determine the boundaries of clusters. Density clustering algorithms, such as DBSCAN, usually use Euclidean distance for similarity measurement, but this distance is not suitable for similarity measurement of time series data. To this end, this scheme introduces a measure based on shape similarity as the basis for distance calculation. There are two reasons for choosing density clustering: first, this scheme requires unsupervised clustering of different types of traffic services, and the number of clusters cannot be determined in advance. Density clustering can meet this requirement; second, density clustering can identify dense areas of arbitrary shapes and sizes, and the shape similarity between shape lines helps to automatically expand the boundaries of clusters.

[0089] Specifically, density clustering based on shape similarity detects high-density areas in the data and gradually expands these areas based on the shape distance between data points (i.e., shape line sequences) to form clusters. Core points are defined as points that contain at least minPts objects within a distance range ∈, and only objects within the range ∈ can be classified into the same cluster. By using the density clustering method based on shape similarity, a preliminary clustering analysis can be performed on the traffic data to reveal the potential clustering structure.

[0090] 4. Statistical feature allocation

[0091] In addition, in the statistical feature assignment stage, since density clustering may produce some unclassified samples in low-density areas, the extracted multidimensional features are used to supplement the clustering results. For each cluster, the four-dimensional feature mean of its members is calculated as the cluster center feature. Then, these unclassified samples are compared with the feature values ​​of each cluster center and assigned by the minimum Euclidean distance. For samples whose distance from the cluster center exceeds the threshold, the server marks them as outliers, thereby ensuring the accuracy and robustness of clustering. This process helps identify abnormal traffic and potential threats, while supporting efficient processing of complex encrypted traffic.

[0092] 1) Calculate the cluster center feature: For each formed cluster, calculate the mean of the four-dimensional feature vectors of its members to obtain the cluster center feature vector. Specifically, suppose a cluster contains l shape lines, each of which has a four-dimensional feature vector f i , then the central eigenvector f of the cluster centric Calculation formula:

[0093]

[0094] 2) For unclassified shape lines, the Euclidean distance between its feature vector and the feature vectors of all cluster centers is calculated. If the distance is less than the specified threshold δ, it is assigned to the nearest cluster, otherwise it is marked as an outlier. Let the feature vector of the unclassified point be f u , cluster C j The central eigenvector of is f j , by calculating the Euclidean distance from the unclassified point to all cluster centers, assign it to the cluster with the closest distance:

[0095]

[0096] Among them, C u represents the cluster to which the unclassified point u is assigned. In addition, it should be noted that some unclassified shape lines may not be similar to any cluster, so they should be marked as outliers. To this end, we set a specific distance threshold δ. If ∥f u -f j ∥ is less than the predetermined threshold δ, then the unclassified points are assigned to cluster C i , otherwise it is marked as an outlier.

[0097] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A shape-based noise-resistant encrypted traffic clustering method, the steps of which include: 1) Obtain and parse the original traffic data to obtain multiple data streams; The uplink data and downlink data of each data stream are divided according to the set time interval to obtain the uplink sequence U and downlink sequence D corresponding to each data stream; 2) The upstream sequence U and downstream sequence D of each data stream are concatenated to form an overall sequence F, which represents the behavioral characteristics of the corresponding data stream; Extract the shape line S and statistical features of the corresponding data flow from the behavioral features of each data flow; Normalize the data in each shape line S; 3) Clustering each data stream based on the normalized shape line S corresponding to each data stream, and clustering data streams with the same behavior pattern into one cluster; 4) For each cluster obtained by clustering, the average value of the statistical characteristics of each data stream in the cluster is calculated as the cluster center feature of the corresponding cluster; Then, the Euclidean distance between the statistical features of the data stream that is not clustered into any cluster in step 2) and the features of each cluster center is calculated. If the Euclidean distance between the statistical features of the data stream i that is not clustered into any cluster and the features of the cluster center of the j-th cluster is the smallest and less than the set threshold δ, the data stream i is divided into the j-th cluster; otherwise, the data stream i is marked as an outlier.

2. The method according to claim 1, characterized in that: The method of extracting the shape line S of the corresponding data stream from the behavioral characteristics of each data stream is: using a moving average algorithm with a sliding window to calculate the overall sequence F = (f1, f2, ..., f q ) for smoothing; if the zero value ratio of the uplink sequence U is greater than 50%, the sliding window size Otherwise the sliding window size For each point f in the overall sequence F t , and its corresponding point s on the shape line S t is a vector (f t-w+1 ,…,f t ), shape line S=(s w ,s w+1 ,…,s q ), 3. The method according to claim 1, characterized in that: The statistical features include the average value of the downlink sequence D, the ratio of the average value of the uplink sequence U to the average value of the downlink sequence D, the proportion of zero values ​​in the uplink sequence U, and the total length of the overall sequence F.

4. The method according to claim 1, 2 or 3, characterized in that: The method for clustering each data stream based on the normalized shape line S corresponding to each data stream is: 31) For two shape lines of unequal length and shape lines First make exist Slide relatively on the surface and calculate its inner product at each phase shift s x m Shape Line The mth data segment, y n Shape Line The nth data segment in; 32) According to the inner product Calculate shape lines With shape line The normalized cross-correlation of 33) According to the formula Calculate shape lines With shape line Similarity if If the shape line is smaller than the set threshold, Shape Line The corresponding data streams are grouped into the same cluster.

5. The method according to claim 4, characterized in that Among them, s∈[1-m,n-1].

6. The method according to claim 1, 2 or 3, characterized in that: Uplink sequence U = {U1, U2, ..., U K }, downlink sequence D = {D1, D2, ..., D K }; Among them, U k =∑ i payload i ·I[k×Δt≤time i <(k+1)×Δt], D k =∑ i payload i ·I[k×Δt≤time i <(k+1)×Δt],payload i is the byte size of the i-th data packet, time i is the arrival time of the ith data packet, Δt is the set time interval, k=1~K, K=T / Δt.

7. The method according to claim 1, characterized in that Capture and parse raw traffic data in real time from network interfaces or traffic capture files.

8. A server, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images