A decentralized application encryption traffic classification method and device based on metric space

Abstract

The present invention discloses a decentralized application encrypted traffic classification method and device based on metric space, including collecting the encrypted traffic of decentralized applications, marking the application of each encrypted traffic; clustering using the characteristic vector of each encrypted traffic to divide simple samples and difficult samples; taking any difficult sample in the clustering result as a positive sample F, and the cluster center point of the category to which the positive sample F belongs as the positive sample cluster center C F , any difficult sample in other categories is regarded as a negative sample F′ i , the negative sample F′ i The cluster center of the category is used as #imgabs0# to construct several quadruplets #imgabs1#. The quadruplet network is trained using the quadruplet S to obtain a classification model. The samples in the test set are input into the classification model, and the similarity is calculated in the metric space to obtain the classification result of the target encrypted traffic. The present invention provides high-quality samples containing more information, effectively screens simple data sets, and automatically learns effective features through the network, making DApps classification more efficient and accurate.

Description

Technical Field

[0001] The present invention belongs to the field of network security and relates to the field of computer software, and specifically to a method and device for classifying encrypted traffic of decentralized applications (DApps) based on a metric space. Background Art

[0002] Since the birth of blockchain represented by Bitcoin in 2008, blockchain technology has received extensive attention and research due to its characteristics of decentralization, anonymity, immutability, and liquidity. There are more than 9,800 digital currencies based on blockchain. The functions carried by blockchain have also evolved from cryptocurrencies represented by Bitcoin to smart contracts represented by Ethereum, thus becoming Turing-complete programmable cryptocurrencies.

[0003] Blockchain technology is essentially an underlying ledger that serves real-world scenarios. The top application layer of the blockchain includes electronic wallets, various blockchain applications, etc. Most of them can be classified as DApps, which are the main landing projects of blockchain technology. DApps are managed autonomously without the control of a single entity. Blockchain technology can naturally provide anonymity for each user, which is a unique advantage that traditional applications cannot provide. So far, more than 3,700 DApps have been deployed on different blockchain platforms, such as Ethereum (81.91%), EOS (8.97%), Steem (1.62%), etc. However, DApps are in the early stages of development and face two key issues: how to better manage the DApps network and how to ensure a secure network environment. Network traffic classification has emerged. For the former, DApps services are classified according to different priority strategies to better manage the network; for the latter, malicious DApps traffic is detected for anomalies to ensure network security. Traffic classification has attracted the attention of many researchers. Many methods have been proposed for website classification, mobile application classification, and user behavior classification, but there are few studies on DApps encrypted traffic classification.

[0004] By passively collecting traffic, the encrypted traffic of decentralized application networks can be classified and analyzed. The existing traditional classification methods can be divided into three types: ① Traffic classification method based on port matching, ② Traffic classification method based on data packet payload matching (deep packet inspection), and ③ Traffic classification method based on machine learning. However, with the development of technology, ① is no longer applicable due to port randomization and obfuscation; because the decentralized application network communication is encrypted and the communication content is randomized, the ② technology, such as matching plaintext feature fields and behavioral feature libraries, is also invalid. That is, the rule-based matching method (① and ②) is not suitable for encrypted decentralized applications; and due to the revival of artificial intelligence technology and the development of computer hardware technology, the method based on machine learning has become the mainstream method in the field of encrypted traffic classification.

[0005] 80% of DApps are deployed on Ethereum and use the same communication interface. In addition, unlike traditional applications, different SSL / TLS protocol versions are not used due to differences in companies. Therefore, DApps deployed on the same blockchain platform use similar encrypted traffic settings, which increases the similarity of DApps traffic.

[0006] Although previous work has achieved good accuracy results in traditional application classification scenarios, these methods designed very complex network structures and combined manually designed extracted features based on professional knowledge, manpower and time costs. Some studies use fewer or simpler features to achieve higher accuracy, but the results in DApps encrypted traffic classification scenarios are poor. Some studies also use large-scale datasets to train for a long time to improve performance (for example, the dataset contains 956,000 flows), but the dataset contains many simple samples that can be directly distinguished by the model, which leads to dataset redundancy. In general, the DApps encrypted traffic classification task can be decomposed into two subtasks: how to automatically extract features and accurately classify DApps traffic; how to improve model training efficiency.

[0007] Representative DApps classification methods include FFP and RF+LT. Both methods are based on expert prior knowledge, and manually analyze the characteristics of DApps and classification scenarios to design feature sets suitable for corresponding scenarios. The FFP method extracts time series, packet length, and burst sequence features, and fuses the three features through kernel functions to generate high-dimensional features. The accuracy of DApps traffic classification is 90%, but due to the large input feature vector, the training and testing time of this method is relatively high. The authors who studied the RF+LT method found that about 60% of the DApps encrypted traffic in the dataset is short flow, resulting in poor effect of burst features, so only the time and packet length sequences are extracted, and a classifier is constructed using random forests to distinguish different DApps. However, these methods rely too much on experts, and the manually designed features often contain redundant features, which increase the training and testing costs and affect the test results. Summary of the invention

[0008] In order to solve the above problems, the present invention provides a method and device for classifying encrypted traffic of decentralized applications based on metric space. The technical solution of the present invention is called CQNet architecture. Without decrypting or reversing the encrypted traffic of decentralized applications, the decentralized application traffic is passively detected, and an embedding space is obtained through deep metric learning, so that each encrypted stream is mapped to this space to form an embedding vector. This invention includes two mechanisms: a simple sample screening algorithm (FE-set algorithm) and a quadruple network. The FE-set algorithm is based on Mini-Batch KMeans, Kuhn-Munkres algorithm and exploration cluster center algorithm to filter out simple samples from the data set, and finally divide the data set into a simple data set and a difficult data set, where semi-hard and hard data are uniformly classified as difficult data sets. Afterwards, a quadruple is constructed on the difficult data set as the input of the quadruple network, and the restriction relationship of the training is increased to ultimately improve the training efficiency and classification accuracy. Experiments show that the present invention can remove some easily distinguishable traffic samples and obtain a high-quality trainable sample set containing more information. By increasing the restrictive relationship between samples, the categories of samples can be distinguished more accurately. The model of the present invention performs better than the latest encrypted traffic classification model on the self-collected real decentralized application data set.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A decentralized application encryption traffic classification method based on metric space, the steps of which include:

[0011] 1) Collect the encrypted traffic of decentralized applications and mark the application for each encrypted traffic;

[0012] 2) Using the feature vectors of each encrypted traffic, clustering is performed to divide simple samples and difficult samples;

[0013] 3) Take any difficult sample in the clustering result as a positive sample F, and the cluster center point of the category to which the positive sample F belongs as the positive sample cluster center C F , any difficult sample in other categories is regarded as a negative sample F′ i , the negative sample F′ i The cluster center of the category is To construct several quadruples

[0014] 4) Using the quaternary group S to train the quadruple network to obtain a classification model;

[0015] 5) Input the samples in the test set into the classification model, calculate the similarity in the metric space, and obtain the classification results of the target encrypted traffic.

[0016] Furthermore, the categories of decentralized applications include: market, social, financial, high-risk and game; the market category includes: Ocean Market, OpenSea, Superrare and District0x; the social category includes: Livepeer, Loom Network and 2key; the financial category includes: Tether, MakerDAO and Nexo; the high-risk category includes: DoubleWay, E2X and Gandhiji; the game category includes: Axie Infinity, BFH and EvolutionLand.

[0017] Furthermore, before extracting the feature vector, the encrypted traffic is preprocessed.

[0018] Furthermore, the preprocessing includes: filtering encrypted traffic without Client Hello packets, filtering encrypted traffic without SNI fields, filtering ACK packets in the encrypted traffic, and filtering retransmitted data packets in the encrypted traffic.

[0019] Furthermore, the characteristic vector of each encrypted flow is obtained through the following steps:

[0020] 1) Extract the first n bytes of each encrypted traffic as the original features;

[0021] 2) Remove all zero rows and normalize the original features;

[0022] 3) Map the normalized results into the same embedding space to obtain the feature vector of each encrypted flow.

[0023] Furthermore, the clustering method includes: using a distance-based Mini-Batch K-Means clustering algorithm.

[0024] Furthermore, the distance includes: Euclidean distance.

[0025] Furthermore, difficult samples are obtained through the following steps:

[0026] 1) Using the K-Means method, obtain the cluster center and cluster radius of each cluster;

[0027] 2) According to the true label and cluster label, the mapping relationship between the true label and the cluster label is obtained through the Kuhn-Munkres algorithm. For the categories without cluster labels, the sample point with the largest distance to the existing cluster center in the samples of each class is selected as the class center;

[0028] 3) By setting the threshold of the distance between each feature vector and the corresponding cluster center, simple samples can be screened and difficult samples can be obtained.

[0029] Furthermore, other types of selection methods include: selecting the positive sample cluster center C F The nearest categories.

[0030] Furthermore, when training the quadruple network, the similarity information between categories is incorporated into the threshold of the loss function.

[0031] A storage medium stores a computer program, wherein the computer program is configured to execute the above method when running.

[0032] An electronic device includes a memory and a processor, wherein the memory stores a program for executing the above method.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] 1) This invention is the first to address the problem of redundancy in decentralized application datasets. The FE-set module can effectively filter simple datasets and provide the network with high-quality samples containing more information.

[0035] 2) The present invention increases the number of sampled samples and the restrictive relationships between samples to better train the deep learning model, thereby effectively identifying the category to which the traffic samples belong. Combined with the FE-set module, the DApps classification can be made more efficient and accurate.

[0036] 3) The existing DApps traffic classification model uses manually designed features and requires strong expert knowledge, while the present invention can automatically learn effective features through the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The basic block diagram of the decentralized application encryption traffic classification method based on metric space. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below through specific embodiments and drawings. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0039] The decentralized application encrypted traffic classification method of the present invention develops a new specific instance CQNet of the decentralized application encrypted traffic classification algorithm through clustering algorithm and deep learning, which includes two mechanisms FE-set and quadruple network; based on FE-set, the redundancy problem of DApps encrypted traffic data set is handled, and simple data sets and difficult data sets are divided based on clustering and other algorithms, thereby improving training efficiency and obtaining similarity information between data set samples; the quadruple network samples and constructs quadruple inputs, and integrates the data set characteristics learned by FE-set into the model to generate more restriction relationships, and maps the original traffic features to the same embedding space through a neural network. Samples of the same type are classified into the same cluster due to similar embedding vectors. Experiments have proved that CQNet can effectively handle the DApps encrypted traffic classification problem, and the specific steps include:

[0040] 1) Collect the encrypted traffic of decentralized applications and mark the application category of each encrypted traffic;

[0041] 2) Divide the simple data set and the difficult data set according to the simple sample screening algorithm;

[0042] 3) According to the classification target and annotation label, the raw data corresponding to the data in the difficult dataset is input into the final network, the traffic features are automatically extracted and trained, and finally the sample categories are distinguished by similarity in the same metric space.

[0043] Furthermore, before screening simple samples, the encrypted traffic is preprocessed; the preprocessing includes: filtering encrypted traffic without Client Hello packets or SNI fields, filtering ACK packets and retransmitted data packets in each encrypted traffic, so as to remove the impact of network fluctuations on the training model.

[0044] Furthermore, the FE-set algorithm is used to screen the simple samples and difficult samples in the original data set, including:

[0045] Extract the first n bytes of features of each stream as the original features of each stream, preprocess the original features, including removing all zero rows and normalizing, map the original features into the same embedding space, and obtain the feature vector of the decentralized application;

[0046] The feature vectors are clustered into clusters using the distance-based clustering algorithm Mini-Batch K-Means, and then the cluster center and radius are obtained by clustering the samples of a single cluster using K-Means.

[0047] According to the real label and cluster label, the mapping relationship between the real label and the cluster label is obtained through the Kuhn-Munkres algorithm;

[0048] For categories without cluster labels, select the sample point with the largest distance from the existing cluster center in each class sample as the class center;

[0049] By setting a threshold based on the distance from the center C, simple samples are screened out, and the remaining samples are classified as difficult samples for subsequent neural network training.

[0050] Furthermore, the input of the quadruple network is constructed, a sample is randomly selected as a positive sample, and the cluster center point of the category to which it belongs is used as an anchor point. Negative samples are randomly sampled from the 20 categories closest to this center point obtained by the FE-set algorithm, and the corresponding cluster center point is used as another anchor point to form a quadruple.

[0051] Furthermore, the corresponding original traffic features are input into the neural network to automatically extract important features;

[0052] Furthermore, the similarity information between categories learned by the FE-set algorithm is integrated into the threshold of the original loss function to integrate the characteristics of the data set into the training of the network.

[0053] Furthermore, the known decentralized applications include 14 categories: market, social, financial, high-risk, and game; the market category includes Ocean Market, OpenSea, Superrare, District0x, etc.; the social category includes Livepeer, Loom Network, 2key, etc.; the financial category includes Tether, MakerDAO, Nexo, etc.; the high-risk category includes DoubleWay, E2X, Gandhiji; the game category includes Axie Infinity, BFH, Evolution Land, etc.

[0054] A decentralized application encryption traffic classification system based on metric space, comprising:

[0055] 1) DApps encrypted traffic collection module, which is used to collect encrypted traffic of decentralized applications in real environments and mark the application categories of each traffic;

[0056] 2) DApps sample screening module, which is used to divide the data set into simple data sets and difficult data sets, and learn the characteristics between data set samples;

[0057] 3) DApps encrypted traffic classification module, which is used for sampling and construction of quadruple network input, automatically extracts effective feature features through neural networks, and maps them into the same embedding space to calculate similarity to identify DApps encrypted traffic.

[0058] Specifically, Figure 1The basic block diagram of the method of the present invention is as follows. The DApps classification method is divided into a training phase and a classification phase. In the training phase, the data set is first processed by the FE-set module, and the remaining samples are used as the training data set to obtain the final deep learning model.

[0059] In the classification stage, the original features of the traffic data set to be classified are sent into the classifier through the deep learning model classifier obtained in the training stage to obtain the corresponding classification results.

[0060] The scheme of the present invention comprises the following steps:

[0061] Training phase:

[0062] 1. Data collection:

[0063] Two Dell desktop computers were used, and 14 categories and 61 representative decentralized applications were selected, including market (Ocean Market, OpenSea, Superrare, District0x), social (Livepeer, Loom Network, 2key), financial (Tether, MakerDAO, Nexo), high-risk (DoubleWay, E2X, Gandhiji), and game (Axie Infinity, BFH, Evolution Land). Two Win 7 virtual machines were installed on each computer to collect real decentralized application encrypted traffic. For each decentralized application, a script was used to simulate the real user operation of the application in the virtual machine, and the traffic collection tool was used to collect the encrypted traffic of the pure decentralized application, and the encrypted traffic was labeled by the process ID. The traffic collection of each application was repeated 500 times, and the original encrypted traffic data set was finally obtained.

[0064] 2. Data preprocessing:

[0065] Since the unavoidable network traffic from system applications, browsers, or third-party libraries affects the accuracy of the data set, the Server Name Indication extension (SNI) field in the extended part of the Client Hello packet in each data stream is used to filter out the clutter, leaving only the data stream of decentralized applications. However, for data streams without Client Hello packets or SNI fields, this data stream is filtered out; for the remaining data streams, due to the network environment or other reasons, the number of ACK packets in the same application may be different, and there may be retransmitted data packets, so the ACK packets and retransmitted data packets in each stream are filtered out. Since the subsequent processing is to encrypt the first n bytes of the traffic, the first n bytes in the data set are padded with 0 or truncated. The training set and test set are divided into 4:1.

[0066] 3. Sample screening:

[0067] Randomly sampling positive and negative sample pairs contains a large number of easily distinguishable sample pairs. The model cannot learn enough information and is prone to overfitting and falling into local optimality. In order to provide the network with samples containing more information, the data set is preliminarily screened through the FE-set module.

[0068] Since Euclidean distance is used as the similarity calculation between sample pairs, the distance-based clustering algorithm Mini-Batch K-Means is used to cluster them into K clusters, and then the samples of a single cluster are clustered by K-Means to obtain the cluster center C and radius R; according to the true label and cluster label, the maximum matching Kuhn-Munkres algorithm is used to obtain the mapping relationship between the true label and the cluster label; for categories without cluster labels, the sample points with the largest distance to the existing cluster center in each class sample are selected as the class center; the threshold ρ*R is set by the distance from the center C to divide the Easy and Hard datasets (including semi-hard samples).

[0069] 4. Model structure:

[0070] Common network structures used in metric learning are twin networks and triple networks. Although additional information has been added compared to traditional methods, the restriction relationship is still not enough to train the network better. Compared with the triple network, the anchor point and negative sample point are replaced with the center of their respective category clusters, and a negative sample point is added. This negative sample point is not randomly selected, but belongs to the 20 categories closest to this center point obtained by the FE-set algorithm. Then the mined data features are integrated into the network training, and the network obtains more additional information based on the restriction relationship. Finally, the category of the flow is judged by the similarity of each flow in the embedding space, and the network is trained based on the characteristics of the data set and hinge loss to obtain the final classification model.

[0071] Classification stage:

[0072] The test set is processed according to the data preprocessing process in the second step above, and the input of the network is constructed according to the four-tuple construction method, and input into the trained model for classification.

[0073] Examples

[0074] We selected decentralized applications from multiple categories, including: Market (Ocean Market, OpenSea, Superrare, District0x, etc.), Social (Livepeer, Loom Network, 2key, etc.), Finance (Tether, MakerDAO, Nexo, etc.), High-risk (DoubleWay, E2X, Gandhiji, etc.), and Game (Axie Infinity, BFH, Evolution Land, etc.), a total of 14 categories and 61 representative decentralized applications. Starting from October 12, 2020, we accessed decentralized application behaviors through Python scripts in virtual machines, and captured the traffic generated by the access process through Wireshark. Each application access behavior was repeated 500 times to obtain the final data set (a total of 195,000 flows, millions of data packets). After data preprocessing, we obtained the original traffic characteristics of the data set, screened high-quality samples, constructed network inputs, trained the classifier model, and used a ten-fold cross-validation method to verify the classification effect of the model. The accuracy of DApps classification was 98.37%.

[0075] The above-mentioned embodiments only express the implementation methods of the present invention, and the description is relatively specific, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A decentralized application encryption traffic classification method based on metric space, the steps of which include: 1) Collect the encrypted traffic of decentralized applications and mark the application for each encrypted traffic; 2) clustering using the characteristic vectors of each encrypted flow to divide the simple samples into the difficult samples; wherein clustering using the characteristic vectors of each encrypted flow to divide the simple samples into the difficult samples includes: Use the distance-based clustering algorithm Mini-Batch K-Means to cluster the feature vectors into clusters, and then use K-Means to cluster the samples of a single cluster to obtain the cluster center point and radius; According to the real label and cluster label, the mapping relationship between the real label and the cluster label is obtained through the Kuhn-Munkres algorithm; For categories without cluster labels, select the sample point with the largest distance from the existing cluster center point in the samples of each class as the class center; Simple samples are filtered out by setting a threshold value based on the distance from the cluster center point, and the remaining samples are classified as difficult samples; 3) Take any difficult sample in the clustering result as a positive sample F, and the cluster center point of the category to which the positive sample F belongs as the positive sample cluster center C F , any difficult sample in other categories is regarded as a negative sample F′ i , the negative sample F′ i The cluster center of the category is To construct several quadruples 4) Use the quaternary network S to train and obtain a classification model; where: The quadruple network replaces the anchor points and negative sample points of the triple network with the cluster centers of their respective categories, and adds a negative sample point; the negative sample point is the class center point of the 20 categories closest to the cluster center point obtained by the FE-set algorithm; the FE-set algorithm uses Euclidean distance as the similarity calculation between sample pairs, uses the distance-based clustering algorithm Mini-Batch K-Means to cluster them into K clusters, and then uses K-Means to cluster the samples of a single cluster to obtain the cluster center C and radius R; according to the true label and the cluster label, the mapping relationship between the true label and the cluster label is obtained by the maximum matching Kuhn-Munkres algorithm; for categories without cluster labels, the sample point with the largest distance from the existing cluster center in each class sample is selected as the class center; the threshold ρ*radius R is set by the distance from the center C to divide the simple sample and the difficult sample; The training of the quadruple network using the quadruple set S refers to integrating the similarity information between the categories learned by the FE-set algorithm into the threshold of the loss function; 5) Input the samples in the test set into the classification model, calculate the similarity in the same metric space, and obtain the classification results of the target encrypted traffic.

2. The method according to claim 1, characterized in that The categories of decentralized applications include: marketplace, social, finance, high-risk and games; the marketplace includes: Ocean Market, OpenSea, Superrare and District0x; the social category includes: Livepeer, Loom Network and 2key; the financial category includes: Tether, MakerDAO and Nexo; the high-risk category includes: DoubleWay, E2X and Gandhiji; the game category includes: AxieInfinity, BFH and Evolution Land.

3. The method according to claim 1, characterized in that Before extracting the feature vector, the encrypted traffic is preprocessed; the preprocessing includes: filtering the encrypted traffic without Client Hello packets, filtering the encrypted traffic without SNI fields, filtering ACK packets in the encrypted traffic, and filtering retransmitted data packets in the encrypted traffic.

4. The method according to claim 1, characterized in that Obtain the feature vector of each encrypted flow through the following steps: 1) Extract the first n bytes of each encrypted traffic as the original features; 2) Remove all zero rows and normalize the original features; 3) Map the normalized results into the same embedding space to obtain the feature vector of each encrypted flow.

5. The method according to claim 1, characterized in that The clustering method includes: using a distance-based Mini-Batch K-Means clustering algorithm; the distance includes: Euclidean distance.

6. The method according to claim 1, characterized in that Other categories of selection methods include: selecting the positive sample cluster center C F The nearest categories.

7. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

8. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images