A Tor hidden service traffic identification method based on feature selection

Abstract

The present invention designs a Tor hidden service traffic identification method based on feature selection. The method completes Tor version 3 hidden service traffic collection and data preprocessing by constructing a hidden service traffic collection system based on a cloud server, and constructs an optimal feature subset for different recognition models by designing a feature selection algorithm. Hidden service traffic collection and data preprocessing mainly include constructing a traffic collection system based on a cloud server, Tor hidden service traffic collection, and Tor hidden service traffic data preprocessing. The feature selection algorithm is based on the mRMR algorithm to construct the optimal feature subset for different recognition models by improving the estimation error of relevant redundancy and reducing the negative impact of irrelevant redundancy, thereby reducing the actual deployment overhead of the recognition model while ensuring the recognition accuracy and false alarm rate.

Description

Technical Field

[0001] The present invention relates to a Tor hidden service traffic identification method based on feature selection, belongs to the field of cyberspace security, and is specifically applied to network traffic identification for version 3 hidden services in a Tor network. Background Art

[0002] An anonymous network is a communication network that uses measures such as data forwarding, content encryption, and traffic obfuscation to hide the content and relationships of communications. Tor is the most popular anonymous communication network in actual deployment, and it provides a foundation for the development of hidden services. Tor not only uses a three-hop encryption proxy mechanism to protect the anonymity of users, but also combines distributed hash tables and hidden service directory server technology to protect the anonymity of hidden service sites. However, while hidden services provide anonymity for users, they also provide space for network crimes. On the one hand, hidden services are used by criminals to build anonymous markets for anonymous transactions and anonymous forums for pirated content distribution; on the other hand, hidden services are used by hackers to deploy servers to sell malware or launch network attacks such as DDoS.

[0003] The industry and academia are very concerned about the security issues brought by the above hidden services, and have proposed solutions from different angles to deal with this problem. Some work attempts to establish a correlation between the behavior of surface network users and the behavior of hidden service users to analyze hidden services, and some work implements the deanonymization of hidden services from the perspective of hidden service protocol vulnerability analysis. However, since these methods usually require the deployment of onion routing as a prerequisite, there is a certain degree of implementation difficulty. Due to the advancement of obfuscation technology in Tor, the methods in some work are no longer applicable to the latest version 3 of Tor scenarios. In addition, using artificial intelligence methods at the backbone nodes or gateways of the network to identify and intercept traffic accessing Tor hidden services has become an effective way to ensure network security and deal with Tor hidden services.

[0004] Network traffic identification is an effective technology for enhancing network controllability. It takes network flows as objects and identifies each flow as traffic that accesses a certain website, belongs to a certain application, or belongs to a certain protocol type. Network administrators can use network traffic identification to analyze traffic types with high bandwidth requirements to better control service quality; they can also identify malicious traffic to protect network security; they can also identify traffic that accesses malicious sites such as phishing websites to protect user property safety. The variety of application scenarios makes network traffic identification show great potential in the fields of network security, network optimization, and network management.

[0005] Some scholars have conducted relevant research on traffic identification in the Tor anonymous network. Some works design new classification models for traffic identification; some works propose new features to help the classification model obtain more effective information. However, there are still two shortcomings in the current traffic identification for Tor hidden services. First, it is difficult to collect hidden service traffic in Tor version 3. Since the domain name of the hidden service in Tor is not public, it is difficult to obtain the domain name of the hidden service. Since the hidden service does not set restrictions on domain name application, a large number of hidden services have a short survival cycle and frequent changes, which increases the difficulty of collecting hidden service traffic. Since the hidden service completed the upgrade to version 3 in 2021, the hidden service domain name of version 2 was completely abandoned, resulting in a lack of the latest version 3 hidden service traffic data set. Second, for the task of Tor hidden service traffic identification, how to construct the optimal feature subset is a challenge. Previous work has proposed a large number of meaningful features for Tor traffic identification. The increase in the number of features leads to an increase in the computational overhead of the model in training and use. In addition, the effectiveness of features varies for different identification tasks and classifiers. Studying the construction of the optimal feature subset can select effective features and provide guidance for identification work.

[0006] In view of the above problems, the present invention proposes a Tor hidden service traffic identification method based on feature selection, which can effectively reduce the number of features and reduce the overhead of the actual application of the model under the premise of ensuring the recognition accuracy and false alarm rate. The present invention includes two steps: Tor hidden service traffic collection and data preprocessing, and Tor hidden service traffic identification. In the Tor hidden service traffic collection and data preprocessing steps, a Tor hidden service traffic collection method based on a cloud server is included, which can simulate users accessing Tor hidden services and automatically capture traffic and save it in the cloud server. In the Tor hidden service traffic identification step, a two-stage heuristic feature selection algorithm is included, which reduces the error of redundant estimation between features by combining the minimum redundancy maximum correlation (mRMR) and information gain (IG) methods to achieve effective feature selection. Summary of the invention

[0007] This paper takes the Tor anonymous network as the research scenario and studies the problem of traffic identification when users access hidden services through Tor clients. Based on the collection and analysis of the traffic of Tor version 3 hidden services, a Tor hidden service traffic identification method based on feature selection is proposed. In Tor hidden service traffic identification, the number of features can be effectively reduced while ensuring the accuracy and false alarm rate, and the cost of model deployment can be reduced.

[0008] In order to achieve the above objectives, the technical solution of the present invention is as follows: a Tor hidden service traffic identification method based on feature selection, the method comprising the following two main steps:

[0009] S1: Tor hidden service traffic collection and data preprocessing;

[0010] S2: Identification of Tor hidden service traffic.

[0011] Among them, S1: Tor hidden service traffic collection and data preprocessing. The process is divided into three steps: building a traffic collection system based on cloud servers, Tor hidden service traffic collection, and Tor hidden service traffic data preprocessing.

[0012] S11: Construct a traffic collection system based on a cloud server. The hidden service traffic collection system constructed by this method is as follows: Figure 1 As shown, the specific process is divided into 4 steps:

[0013] (1) Build the traffic generation end. The specific steps of this process are as follows:

[0014] a) Install scripts to control the browser to automatically and sequentially access hidden service domain names;

[0015] b) According to the access status of the browser, the access status and operation code are sent to the traffic collection end.

[0016] (2) Deploy the traffic collection terminal. The specific steps of this process are as follows:

[0017] a) Select a cloud server located in a specific geographical location as a Tor proxy; later, you can replace cloud servers located in different geographical locations according to traffic collection needs;

[0018] b) Install the Tor proxy program to provide access to the Tor network;

[0019] c) The installation script receives the access status and operation code of the traffic generation end and performs the operation.

[0020] (3) Select the website access terminal. The specific steps of this process are as follows:

[0021] a) Install a script to collect a large number of hidden service domain names from domain name aggregation sites;

[0022] b) Filter the version 3 domain names with 56 characters based on the domain name length from the search results in the previous step;

[0023] c) Check the validity and content of domain names through the traffic collection system and remove duplicate domain names;

[0024] d) All hidden service domain names after deduplication are used as access targets.

[0025] (4) Connectivity test: After completing the construction of the traffic generation end, deploying the traffic collection end, and selecting the website access end according to steps (1) to (3), test the connectivity of the hidden service traffic collection system. If the browser of the traffic generation end can connect to the Tor proxy of the traffic collection end and successfully access the hidden service, it means that the traffic collection system is built.

[0026] S12: Tor hidden service traffic collection. This process is divided into three steps: running the script to generate access traffic, running the script to automatically collect traffic, and capturing and storing hidden service traffic:

[0027] (1) Run the script to generate access traffic. Algorithm 1 shows the flow of the traffic generation script. The specific steps are as follows:

[0028] a) Simulate various behaviors of real users when using browsers, such as entering domain names, clicking, searching, etc. to access hidden services and realize automatic generation of traffic;

[0029] b) According to the access status (domain name, access time), send the access status and operation code (START, STOP, EXIT) to the traffic collection end.

[0030]

[0031] (2) Run the script to automatically collect traffic. Algorithm 2 shows the flow of the traffic collection script. The specific steps are as follows:

[0032] a) receiving the access status and operation code of the traffic generating end;

[0033] b) Select the three operations of start acquisition, stop acquisition and storage, and exit program according to the operation code;

[0034] c) The stored file name is named in the format of "domain name + access time" in the access status.

[0035] (3) Capture and store hidden service traffic. To ensure the validity of the collected traffic, record the port number used by the Tor proxy software on the traffic collection end and use this port number as the target for traffic collection. To facilitate subsequent processing, record the HTTP status code generated when accessing each hidden service for data preprocessing in step S13.

[0036]

[0037] S13: Tor hidden service traffic data preprocessing. In order to ensure the validity of the traffic, the original traffic obtained in step S12 needs to go through a preprocessing step before it can be used as a valid data set for identification. This process is specifically divided into three steps: traffic filtering, label setting, and undersampling.

[0038] (1) Traffic filtering. There may be background traffic or invalid traffic in the original traffic that affects model training, so the traffic needs to be filtered. It is divided into three steps:

[0039] a) Divide the original traffic data into different flows according to the five-tuple (source IP address, source port number, destination IP address, destination port number, and transport layer protocol). In the present invention, the source IP address and destination IP address, source port number and destination port number of the five-tuple that are the same after being interchanged are still considered to belong to the same five-tuple;

[0040] b) Based on the IP address of the five-tuple to which the flow belongs, the traffic generated by other applications on the server is marked as irrelevant background traffic and filtered out;

[0041] c) Based on the HTTP status code records returned when each traffic sample is accessed, all traffic that returns 4xx, 5xx status codes and traffic that times out and fails to successfully load the page are marked as invalid traffic and filtered out.

[0042] (2) Set labels. Supervised machine learning requires samples to have correct labels to complete training. Therefore, it is necessary to set labels for the collected traffic data. This is divided into three steps:

[0043] a) According to the number of domain names obtained in step (3) of S11, set the number n of hidden service domain names used in the experiment;

[0044] b) Establish a correspondence between the labels from 0 to n-1 and each hidden service in the experiment in alphabetical order;

[0045] c) Obtain the domain name of the traffic sample according to the file name of the file stored in S12 (2), and set the correct label for each sample obtained in step S13 (1) according to the domain name of the traffic sample and the corresponding relationship.

[0046] (3) Undersampling. Since Tor hidden services are unstable, the number of samples collected from each domain name varies. Therefore, undersampling is needed to balance the number of samples for each domain name. This is divided into three steps:

[0047] a) Count the number of samples of each of the n hidden service domain names, and set C to be the minimum number of samples required for each domain name;

[0048] b) Select all hidden service categories with a sample size greater than C, and calculate the number of hidden service categories with the smallest sample size C min ;

[0049] c) For samples with a size greater than C min The hidden service categories of C are randomly undersampled and only C min samples.

[0050] S2: Identification of Tor hidden service traffic. This process is divided into three steps: constructing a complete set of Tor hidden service traffic features, designing a feature selection algorithm, and identifying Tor hidden service traffic.

[0051] S21: Construct the full set of Tor hidden service traffic features. This process is divided into the following 9 steps:

[0052] (1) Extract the arrival time information from the original traffic, including the four quarter points in the three directions of incoming, outgoing, and bidirectional, and the sum of these 16 values;

[0053] (2) Extract the transmission rate information from the original traffic, including the statistical values ​​of the number of packets transmitted per second (maximum, minimum, average, standard deviation, median), and the number of packets transmitted in each segment after the total duration is evenly divided into 20 segments;

[0054] (3) Extract the inter-arrival time (IAT) information from the original traffic, including the statistical values ​​(maximum value, average value, standard deviation, and 3 / 4 equal points) of the incoming, outgoing, and bidirectional directions;

[0055] (4) Extract basic information about the transmission direction in the original traffic, including the total number of packets in the three directions of incoming, outgoing, and bidirectional, the total number of packets in the first 30 packets and the total number of packets in the last 30 packets, and the ratio of the number of packets in the incoming and outgoing directions to the total number;

[0056] (5) Extract the statistical information of the transmission direction in the original traffic, including the maximum, average, standard deviation and median of the number of outgoing packets in each segment after every 20 bidirectional data packets are divided into a segment, the average and standard deviation of the position of the data packets in the complete flow in both the incoming and outgoing directions, and the number of outgoing packets in each segment after the total number of packets is evenly divided into 70 segments;

[0057] (6) Extract the size and direction information of the data packets in the original traffic, and obtain the signed cumulative sum sequence and the absolute value cumulative sum sequence of the size and direction of the data packets. The cumulative sum sequence is a set of values, each of which represents the sum of the sizes of all the data packets before it;

[0058] (7) extracting interaction pattern information related to the number of data packets, and selecting 100 values ​​from the signed cumulative sum and the absolute value cumulative sum sequences at 100 points divided by the number of data packets;

[0059] (8) extracting the interaction pattern information related to the packet arrival time, and selecting 100 values ​​from the signed cumulative sum and the absolute value cumulative sum sequences at 100 points divided according to the packet arrival time;

[0060] (9) All the information extracted in steps (1)-(8) constitutes the complete feature set of Tor hidden service traffic in the present invention.

[0061] S22: Design a feature selection algorithm. In the identification of Tor hidden service traffic, if the model is directly trained using the full set of features, not only may the best identification result not be obtained, but also a large amount of computational overhead may be incurred. For the selected identification task and classifier, a feature selection algorithm is designed to select the optimal feature subset from the full set of features, which can reduce the classifier overhead while ensuring the identification result. Based on the above ideas, the present invention designs a two-stage heuristic feature selection algorithm (TH), which can construct an optimal feature subset for the Tor hidden service traffic identification task for a specific classifier under the premise of limiting the size of the optimal feature subset. As shown in Algorithm 3, TH is specifically divided into five steps: constructing mmRMR feature importance sequence, constructing IG feature importance sequence, fusing feature importance sequence, constructing the optimal feature subset based on the forward selection method, and setting the end condition. The first two steps belong to the first stage of using the filtering method, and the last three steps belong to the second stage of using the packaging method. The specific five steps are as follows:

[0062]

[0063] (1) Construct the mmRMR feature importance sequence. The modified-mRMR (mmRMR) evaluates the correlation between features and labels and the redundancy between different features. mmRMR takes the candidate feature set Ω S and the classification target variable h as input, select the most important features in the candidate feature set according to formula (1) in each iteration, and output the mmRMR feature importance sequence in the order of selection. In formula (1), S is the optimal feature subset, Ω S is the candidate feature set, i is the feature being evaluated, j is a selected feature in S, h is the categorical target variable, F(i,h) is the F statistical test value calculated between i and h, F(k,h) is the F statistical test value calculated between k and h, and c(i,j) is the Pearson correlation coefficient between i and j;

[0064]

[0065] (2) Construct the IG feature importance sequence. Information gain (IG) converts the candidate feature set Ω Sand the categorical target variable h as input, and output the IG feature importance sequence by evaluating the correlation between the features and the labels individually;

[0066] (3) Fusion feature importance sequence. The two feature importance sequences obtained in steps (1) and (2) are merged using a fusion method to compensate for their shortcomings. The specific implementation of the fusion method is shown in lines 20-28 of Algorithm 3. The mmRMR and IG feature importance sequences are taken as inputs, and the two sequences are traversed synchronously from front to back. The features that do not appear in the fusion sequence are added in order to obtain the fusion method feature importance sequence.

[0067] (4) Construct the optimal feature subset based on the forward selection method. Take the feature importance sequence of the fusion method as input, use the packaging method to test each feature, and output the optimal feature subset. The improved sequence forward selection is an iterative process. In each round, for the next candidate feature, test whether adding the optimal feature subset will improve the recognition effect of the classifier. If it can improve the recognition effect, then add the optimal feature subset;

[0068] (5) Set the end condition. The optimal feature subset is continuously expanded according to the iterative process until the following two situations occur: 1) there are no remaining features in the candidate feature set; 2) the number of features in the optimal feature subset reaches the maximum number m.

[0069] S23: Tor hidden service traffic identification. This process selects different classifiers to identify Tor hidden service traffic to verify the effectiveness of the method of the present invention. It is specifically divided into the following 6 steps:

[0070] (1) Select a machine learning classifier as the classifier used in the recognition process;

[0071] (2) dividing the data set preprocessed in step S13 into a training data set and a test data set;

[0072] (3) Use five-fold cross validation and grid search methods on the training data set to tune the classifier hyperparameters and determine the optimal hyperparameters in this recognition process;

[0073] (4) Obtain the optimal feature subset under the current data set and classifier according to step S22;

[0074] (5) Train the classifier using the training data set;

[0075] (6) Use the test dataset to evaluate the recognition effect of the classifier.

[0076] Compared with the prior art, the advantages of the present invention are as follows:

[0077] The present invention proposes a Tor hidden service traffic identification method based on feature selection. Compared with other Tor hidden service traffic identification methods, the advantages of this method are:

[0078] (1) This solution builds a Tor hidden service traffic collection system based on a cloud server. 1) The traffic collection system is connected to the real Tor anonymous network to ensure that the captured interactive traffic between the hidden service and the client is real and valid. 2) The traffic collection end is deployed in the cloud server. By selecting cloud servers in different geographical locations, it can effectively approximate various network conditions in real user scenarios. 3) The traffic generation end uses a program to realize the automated hidden service access process, which can quickly and efficiently complete the traffic collection task. 4) The entire system does not need to control any Tor routing nodes, which reduces time and money costs and has good feasibility and practicality;

[0079] (2) This scheme designs an efficient recognition method based on feature selection. First, step S21 constructs the complete set of Tor hidden service traffic features, summarizes and extracts the hidden service traffic characteristics from multiple perspectives, and provides support for the feature selection algorithm. Secondly, step S22 designs a two-stage heuristic feature selection algorithm (TH) to construct the optimal feature subset for different classifiers. The advantages of the TH algorithm are as follows: 1) By taking the correlation between the features and the classification labels as weights to perform weighted averaging on the calculation of the redundant parts of the mRMR method, an effective estimation of the relevant redundant parts is achieved, which reduces the evaluation error of the mRMR method; 2) The feature importance sequences of mmRMR and IG methods are merged through the fusion method, which reduces the error of the irrelevant redundant parts in the mRMR formula; 3) The number of attempts of the second-stage packaging method is reduced through the improved sequence forward selection method, which greatly reduces the computational complexity of the TH algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 : Structure diagram of hidden service traffic collection system based on cloud server;

[0081] Figure 2 : Flow chart of the flow generation end of the flow collection system in the embodiment;

[0082] Figure 3 : Flow chart of flow collection end of flow collection system in the embodiment;

[0083] Figure 4 : Flow chart of flow preprocessing in the embodiment;

[0084] Figure 5 : Flowchart of feature selection algorithm in the embodiment. DETAILED DESCRIPTION

[0085] The technical scheme in the embodiment is described in detail below in conjunction with the accompanying drawings of the present invention. Obviously, the embodiment described in the following text is only one embodiment of the method of the present invention, rather than all embodiments. Based on the following embodiments, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0086] Example 1: See Figure 1 to Figure 5 , a Tor hidden service traffic identification method based on feature selection. The implementation process of the present invention is divided into two main steps: Tor hidden service traffic collection and data preprocessing, and Tor hidden service traffic identification.

[0087] S1: Tor hidden service traffic collection and data preprocessing. The implementation process is divided into three steps: building a traffic collection system based on a cloud server, Tor hidden service traffic collection, and Tor hidden service traffic data preprocessing.

[0088] S11: Build a traffic collection system based on a cloud server. The implementation process includes three steps: equipment preparation, environment configuration, Tor proxy service installation and testing;

[0089] (1) Equipment preparation. Figure 1 As shown, in order to collect hidden service traffic in various network environments, this embodiment sets up 3 local hosts and 3 cloud servers. The local host is configured as the traffic generation end, and the cloud server is configured as the traffic collection end. The operating system of the local host is Ubuntu 18.04 64Bit, the memory is 4GB, and there is no requirement for the geographical location. The cloud servers are rented from the RackNerd platform, have their own independent public IP addresses, the operating system is Ubuntu18.0464Bit, the memory is 2.5GB, and the geographical locations are Los Angeles, New York, and San Francisco respectively;

[0090] (2) Environment configuration. The traffic generation and collection scripts need to configure a complete operating environment. This process includes three steps:

[0091] a) Configure a static public IP address for each cloud server and install the traffic capture tool tcpdump;

[0092] b) Install Firefox browser on each local host;

[0093] c) All local hosts and cloud servers must have Python 3 runtime installed, as well as the required Python modules, including selenium, requests, json, and logging.

[0094] (3) Tor proxy service installation and testing. To access hidden services, you need Tor's official proxy to access the Tor network. The specific installation process is divided into four steps:

[0095] a) Use SSH to remotely log in to the overseas cloud server, set the hosts.allow and hosts.deny files, and only allow the IP address of the corresponding local host to access the cloud server to prevent the cloud server from being interfered with by malicious attacks;

[0096] b) Download the source code of Tor proxy service from Tor's official website https: / / www.torproject.org / download / tor / , upload it to the cloud server and install it;

[0097] c) Set the configuration of Tor proxy service, including: listening IP and port, connection type, and allowed access IP range;

[0098] d) Test network connectivity: Set the remote proxy address of the local host Firefox browser to the listening IP and port of the Tor proxy; if the local host can access any hidden service domain name, it means that the Tor proxy service is installed successfully.

[0099] S12: Tor hidden service traffic collection. The implementation process includes 4 steps: obtaining the version 3 hidden service domain name, writing the traffic generation end script, writing the traffic collection end script, capturing the hidden service traffic and storing it.

[0100] (1) Obtain the domain name of version 3 hidden service. Collecting the traffic of hidden services requires a set of domain names of hidden services as collection targets, which is divided into four steps:

[0101] a) Use selenium and pyautoGUI crawler technologies to access hidden service domain name aggregation sites to obtain domain names, including: hidden service wikis and search engines in hidden services;

[0102] b) Filter version 3 domain names based on domain name length, and only retain domain names with a length of 56 characters. After screening, this embodiment obtains a total of 15,000 version 3 hidden service domain names;

[0103] c) Use the traffic collection system to access each domain name in turn, check the validity of the domain name, remove invalid domain names, and record the title of the domain name page;

[0104] d) Only one domain name with the same title is retained, and all the filtered domain names are saved in the file hsurl.txt. In this embodiment, a total of 343 version 3 hidden service domain names with different contents are obtained.

[0105] (2) Write a traffic generation script. The script process is as follows: Figure 2 As shown. In order to automatically generate traffic, use selenium to operate the browser to access all hidden service domain names in hsurl.txt. Use requests to send status information and operation codes to the traffic collection end. It is divided into the following three steps:

[0106] a) Download and install the geckodriver driver that matches the Firefox browser version on the local host. Selenium provides a simple API to operate the browser by calling geckodriver. This API can be used to simulate the various behaviors of real users using the browser;

[0107] b) Write a Python script, call the Selenium API to initialize the Firefox browser, set the remote proxy server to the IP and port of the Tor proxy service; call requests to establish a TCP connection with the traffic collection end;

[0108] c) For each domain name in hsurl.txt, perform the following 4 steps: notify the traffic collection end to use tcpdump to start collecting traffic, status code START; use selenium to control the browser to access the hidden service; wait for the access to complete or reach the timeout time of 60 seconds; notify the traffic collection end to stop capturing traffic and save it, status code STOP.

[0109] (3) Write a traffic collection script. The script process is as follows: Figure 3 As shown, the process includes the following 5 steps:

[0110] a) Initialize the Tor proxy service, record the IP address and port number of the Tor entry route, and wait for the traffic generation end to initiate a TCP connection;

[0111] b) Perform operations according to the operation code state received from the traffic generation end;

[0112] c) When state is START, open tcpdump to start capturing traffic; the filter expression of tcpdump uses the entry route IP address and port number recorded in a) as conditions;

[0113] d) When the state is STOP, close tcpdump to stop capturing traffic and save it as a .pcap file, naming it according to "domain name + collection time";

[0114] e) In order to solve the storage space limitation of long-term traffic collection, purchase Microsoft OneDrive service and mount it to the file system of the cloud server to expand the storage space of the cloud server.

[0115] (4) Capture hidden service traffic and store it. This process includes the following four steps:

[0116] a) Copy the file hsurl.txt to the script directory of each local host;

[0117] b) Set the number of traffic collection cycles to 500, the maximum access time of the hidden service domain name to 60 seconds, and the access interval between different hidden services to 5 seconds;

[0118] c) Run the scripts in steps (2) and (3) to start acquisition;

[0119] d) Record the IP address and port number of the ingress route for the preprocessing step of the traffic data.

[0120] S13: Tor hidden service traffic data preprocessing. This step preprocesses the original traffic obtained in S12. The specific process is as follows: Figure 4 As shown, it is divided into the following 7 steps:

[0121] (1) Traffic splitting: Use the splitcap tool to divide the original traffic into different flows based on the five-tuple;

[0122] (2) Filter background traffic and filter traffic whose IP addresses do not belong to the cloud server or the ingress router;

[0123] (3) Filter invalid traffic, including traffic with access timeout or HTTP status code 4xx or 5xx;

[0124] (4) According to the total number of domain names n, which is 343 in this embodiment, an integer from 0 to n-1 is assigned to each domain name as a label in alphabetical order;

[0125] (5) Set a label for each sample based on the domain name in the .pcap file name;

[0126] (6) Under-sampling all marked flows. According to the description of step S13 in the content of the invention, this embodiment under-samples the number of samples of all domain names to 350;

[0127] (7) This embodiment constructs a data set containing 343 hidden service sites, each of which contains 350 traffic samples.

[0128] S2: Identification of Tor hidden service traffic. The implementation process specifically includes the following three steps: constructing a complete set of Tor hidden service traffic features, designing a feature selection algorithm, and identifying Tor hidden service traffic.

[0129] S21: Construct the full feature set of Tor hidden service traffic. After step S13, a traffic data set in flow units is obtained. For each flow, features are extracted from multiple angles to construct a full feature set. This process is specifically divided into the following three steps:

[0130] (1) Use the scapy module in Python to extract the size, direction, and arrival time of each data packet for each flow, and arrange them in order of arrival time to form a size and direction sequence;

[0131] (2) extracting relevant features such as arrival time, transmission rate, inter-arrival time, transmission direction, interaction mode, etc. in sequence according to the contents of step S21 in the invention content;

[0132] (3) A total of 436 features were constructed, as shown in Table 1.

[0133] Table 1 Specific names of the full set of features

[0134]

[0135]

[0136] S22: Implement feature selection algorithm. This step uses the two-stage heuristic feature selection algorithm TH to process the full set of features extracted in S21 to obtain the optimal feature subset suitable for a specific classifier. Figure 5 As shown, it is divided into the following 4 steps:

[0137] (1) Setting the input parameters of the TH algorithm. The recognition model clf selected in this embodiment is the SVM classifier, and the maximum number of features m in the optimal feature subset is 95;

[0138] (2) Write a Python program to implement the calculation process of the mmRMR feature importance sequence according to formula (1) and lines 3-15 of Algorithm 3. Take the candidate feature set and the classification target variable as input to calculate the mmRMR feature importance sequence seqM;

[0139] (3) The information gain calculation method was proposed by Ross BC. This embodiment uses the sklearn library in Python to complete the calculation. The candidate features and the classification target variable are used as inputs, and the sklearn library is used to calculate the IG feature importance sequence seqI;

[0140] (4) Write a Python program to implement the calculation process of the fusion method according to lines 20-28 of Algorithm 3. Use seqM and seqI as input to calculate the feature importance sequence seqF of the fusion method;

[0141] (5) Write a Python program to implement the sequence forward selection process in lines 30-42 of Algorithm 3. Take seqF, clf, and m as inputs and add the features that can improve the classification accuracy to the optimal feature subset feats opt Until the number of features in the optimal feature subset reaches the maximum number m or there are no unprocessed features in seqF, the optimal feature subset feats is finally output opt .

[0142] S23: Tor hidden service traffic identification. In this embodiment, five typical machine learning classifiers are selected to identify Tor hidden service traffic, including: decision tree classifier (CART), K nearest neighbor classifier (KNN), support vector machine classifier (SVM), Adaboost classifier, multi-layer perceptron classifier (MLP). The implementation process specifically includes four steps:

[0143] (1) After step S22, the feature data set DS is obtained;

[0144] (2) Call the train_test_split() function in the sklearn module to divide the feature data set into the training set DS in a ratio of 4:1 train and the test set DS test ;

[0145] (3) Hyperparameter tuning: This embodiment calls the GridSearchCV() function in the sklearn module and selects the training set DS train , five-fold cross validation and grid search methods were used to tune the classifier's hyperparameters and determine the optimal hyperparameters in this recognition process;

[0146] (4) Using the training dataset DS train Train the classifier;

[0147] (5) Using the test dataset DS test Evaluate the recognition effect of the classifier and calculate the macro average f1 of acc and f1 value macro , the results are shown in Table 2.

[0148] Table 2 Experimental results of different classification models

[0149]

[0150] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Equivalent replacements or substitutions made on the basis of the above technical solutions all fall within the protection scope of the present invention.

Claims

1. A Tor hidden service traffic identification method based on feature selection, characterized in that: The method comprises the following two steps: S1: Tor hidden service traffic collection and data preprocessing; S2: Identification of Tor hidden service traffic; Among them, step S1 is divided into 3 steps: S11: Build a traffic collection system based on cloud servers, S12: Tor hidden service traffic collection, S13: Tor hidden service traffic data preprocessing; Step S2 is divided into 3 steps: S21: Constructing a complete set of features for Tor hidden service traffic, S22: Design feature selection algorithm, S23: Tor hidden service traffic identification; Among them, S11: Building a traffic collection system based on a cloud server, the specific process is divided into 4 steps: (1) Build the traffic generation end. The specific steps are as follows: a) Install scripts to control the browser to automatically and sequentially access hidden service domain names; b) Send access status and operation code to the traffic collection end according to the browser's access status; (2) Deploy the traffic collection terminal. The specific steps are as follows: a) Select a cloud server located in a specific geographical location as a Tor proxy; subsequently, according to the traffic collection requirements, Changing cloud servers located in different geographical locations; b) Install the Tor proxy program to provide access to the Tor network; c) The installation script receives the access status and operation code of the traffic generation end and performs the operation; (3) Select the website access terminal. The specific steps are as follows: a) Install a script to collect a large number of hidden service domain names from domain name aggregation sites; b) Filter the version 3 domain names with 56 characters based on the domain name length from the search results in the previous step; c) Check the validity and content of domain names through the traffic collection system and remove duplicate domain names; d) All hidden service domain names after deduplication are used as access targets; (4) Connectivity test: After completing the construction of the traffic generation end, deploying the traffic collection end, and selecting the website access end according to steps (1) to (3), test the connectivity of the hidden service traffic collection system. If the browser of the traffic generation end can connect to the Tor proxy of the traffic collection end and successfully access the hidden service, it means that the traffic collection system is built successfully; S12: Tor hidden service traffic collection, which is divided into three steps: running the script to generate access traffic, running the script to automatically collect traffic, and capturing and storing hidden service traffic; (1) Run the script to generate access traffic. Algorithm 1 shows the process of the traffic generation script. The specific steps are as follows: a) Simulate various behaviors of real users when using browsers, access hidden services by entering domain names, clicking, and searching, and realize automatic generation of traffic; b) according to the access status, wherein the access status includes the domain name and the access time, the access status and the operation code are sent to the traffic collection end, wherein the operation code includes START, STOP, and EXIT; (2) Run the script to automatically collect traffic. Algorithm 2 shows the flow of the traffic collection script. The specific steps are as follows: a) receiving the access status and operation code of the traffic generating end; b) Select the three operations of start acquisition, stop acquisition and storage, and exit program according to the operation code; c) The stored file name is named in the format of "domain name + access time" in the access status; (3) Capture and store hidden service traffic. To ensure the validity of the collected traffic, record the port number used by the Tor proxy software on the traffic collection end and use this port number as the target for traffic collection. To facilitate subsequent processing, record the HTTP status code generated when accessing each hidden service for data preprocessing in step S13. S21: Construct the full set of features of Tor hidden service traffic, which is divided into the following 9 steps: (1) Extract the arrival time information from the original traffic, including the four quarter points in the three directions of incoming, outgoing, and bidirectional. And the sum of these 16 values; (2) Extracting transmission rate information from the original traffic, including the statistical value of the number of packets transmitted per second, where the statistical values ​​include the maximum value, minimum value, average value, standard deviation, median value, and the number of packets transmitted in each segment after the total duration is evenly divided into 20 segments; (3) Extract the inter-arrival time (IAT) information from the original traffic, including the statistical values ​​of the incoming, outgoing, and bidirectional directions. The statistical values ​​include the maximum value, average value, standard deviation, and 3 / 4 equal points. (4) Extract basic information about the transmission direction in the original traffic, including the total number of packets in the incoming, outgoing, and bidirectional directions, the total number of packets in the incoming and outgoing directions in the first 30 packets, the total number of packets in the incoming and outgoing directions in the last 30 packets, and the ratio of the total number of packets in the incoming and outgoing directions to the total number of packets in the incoming, outgoing, and bidirectional directions; (5) Extract the statistical information of the transmission direction in the original traffic, including the maximum, average, standard deviation and median of the number of outgoing packets in each segment after every 20 bidirectional data packets are divided into a segment, the average and standard deviation of the position of the data packets in the complete flow in both the incoming and outgoing directions, and the number of outgoing packets in each segment after the total number of packets is evenly divided into 70 segments; (6) extracting the size and direction information of the data packets in the original traffic, and obtaining the signed cumulative sum sequence and the absolute value cumulative sum sequence of the size and direction of the data packets. The cumulative sum sequence is a set of values, each of which represents the sum of the sizes of all the data packets before it; (7) extracting interaction pattern information related to the number of data packets, and selecting 100 values ​​from the signed cumulative sum and the absolute value cumulative sum sequences at 100 points divided by the number of data packets; (8) extracting the interaction pattern information related to the packet arrival time, and selecting 100 values ​​from the signed cumulative sum and the absolute value cumulative sum sequences at 100 points divided according to the packet arrival time; (9) All the information extracted in steps (1)-(8) constitutes the complete set of features of Tor hidden service traffic; S22: Design a feature selection algorithm. As shown in Algorithm 3, the two-stage heuristic feature selection algorithm (TH) is specifically divided into five steps: constructing mmRMR feature importance sequence, constructing IG feature importance sequence, fusing feature importance sequence, constructing optimal feature subset based on forward selection method, and setting end conditions. The first two steps belong to the first stage of using filtering method, and the last three steps belong to the second stage of using packaging method. The specific five steps are as follows: (1) Constructing the mmRMR feature importance sequence, the modified-mRMR (mmRMR) evaluates the correlation between features and labels and the redundancy between different features. The mmRMR sets the candidate feature set Ω S and the classification target variable h as input, select the most important features in the candidate feature set in each iteration according to formula (1), and output the mmRMR feature importance sequence in the order of selection. In formula (1), S is the optimal feature subset, Ω S is the candidate feature set, i is the feature being evaluated, j is a selected feature in S, h is the categorical target variable, F(i,h) is the F statistical test value calculated between i and h, F(k,h) is the F statistical test value calculated between k and h, and c(i,k) is the Pearson correlation coefficient between i and k; (2) Construct the IG feature importance sequence, and the information gain (IG) converts the candidate feature set Ω S and the categorical target variable h as input, and output the IG feature importance sequence by evaluating the correlation between the features and the labels individually; (3) Fusion of feature importance sequences: The two feature importance sequences obtained in steps (1) and (2) are merged using a fusion method to compensate for their shortcomings. The specific implementation of the fusion method is shown in lines 20-28 of Algorithm 3. The mmRMR and IG feature importance sequences are used as input, and the two sequences are traversed synchronously from front to back, and the features that do not appear in the fusion sequence are added in order to obtain the feature importance sequence of the fusion method; (4) Based on the forward selection method, the optimal feature subset is constructed. The feature importance sequence of the fusion method is used as input, each feature is tested using a packaging method, and the optimal feature subset is output. The improved sequence forward selection is an iterative process. In each round, for the next candidate feature, test whether adding the optimal feature subset will improve the recognition effect of the classifier. If it can be improved, add the optimal feature subset; (5) Set the end condition. The optimal feature subset is continuously expanded according to the iterative process until the following two situations occur: 1) There are no remaining features in the candidate feature set; 2) The number of features in the optimal feature subset reaches the maximum number m.

2. The Tor hidden service traffic identification method based on feature selection according to claim 1 is characterized in that: S13: Tor hidden service traffic data preprocessing. In order to ensure the validity of the traffic, the original traffic obtained in step S12 needs to go through a preprocessing step before it can be used as a valid data set for identification. It is specifically divided into three steps: traffic filtering, label setting, and undersampling. (1) Traffic filtering: There may be background traffic or invalid traffic in the original traffic that affects model training. Therefore, the traffic needs to be filtered. The specific steps are as follows: a) Divide the original traffic data into different flows according to the five-tuple, where the five-tuple includes the source IP address, source port number, destination IP address, destination port number, and transport layer protocol. If the source IP address and destination IP address, source port number and destination port number of the five-tuple are the same after being swapped, they are still considered to belong to the same five-tuple; b) Based on the IP address of the five-tuple to which the flow belongs, the traffic generated by other applications on the server is marked as irrelevant background traffic and filtered out; c) Based on the HTTP status code records returned when each traffic sample is accessed, all traffic that returns 4xx, 5xx status codes and traffic that times out and fails to load the page successfully are marked as invalid traffic and filtered out; (2) Setting labels. Supervised machine learning requires samples to have correct labels to complete training. Therefore, it is necessary to set labels for the collected traffic data. This is divided into three steps: a) According to the number of domain names obtained in step (3) of S11, set the number n of hidden service domain names used in the experiment; b) Establish a correspondence between the labels from 0 to n-1 and each hidden service in the experiment in alphabetical order; c) Obtain the domain name of the traffic sample according to the file name of the file stored in step (2) of S12, and set the correct label for each sample obtained in step (1) of S13 according to the domain name of the traffic sample and the corresponding relationship; (3) Undersampling. Since Tor hidden services are unstable, the number of samples collected from each domain name varies. Therefore, undersampling is needed to balance the number of samples for each domain name. This is divided into three steps: a) Count the number of samples of each of the n hidden service domain names, and set C to be the minimum number of samples required for each domain name; b) Select all hidden service categories with a sample size greater than C, and calculate the number of hidden service categories with the smallest sample size C min ; c) For samples with a size greater than C min The hidden service categories of C are randomly undersampled and only C min samples.

3. The Tor hidden service traffic identification method based on feature selection according to claim 1 is characterized in that: S23: Tor hidden service traffic identification. The traffic identification process selects different classifiers to identify Tor hidden service traffic to verify the effect. It is divided into the following 6 steps: (1) Select a machine learning classifier as the classifier used in the recognition process; (2) dividing the data set preprocessed in step S13 into a training data set and a test data set; (3) Use five-fold cross validation and grid search methods on the training data set to tune the classifier hyperparameters and determine the optimal hyperparameters in this recognition process; (4) Obtain the optimal feature subset under the current data set and classifier according to step S22; (5) Train the classifier using the training data set; (6) Use the test dataset to evaluate the recognition effect of the classifier.

Images