A method for identifying open source proxy software traffic based on feature optimization

Abstract

The present invention designs a method for traffic identification of open source proxy software based on feature optimization. The method constructs a cloud-based large-scale traffic collection and processing platform to generate a complete high-quality data set. According to machine learning theory and feature optimization ideas, an efficient identification scheme with three independent tasks is established, and the scheme is applied to complete flow identification. The cloud-based large-scale traffic collection platform mainly includes the construction of open source proxy services, the writing of user behavior simulation scripts, the collection and storage of server-side traffic, and the preprocessing of traffic data. The traffic identification scheme based on feature optimization defines three classification tasks on the one hand to achieve supervision from different angles and granularities, and on the other hand proposes a two-stage feature selection algorithm (TT) based on two dimensions to construct the optimal feature subsets under different classifiers for different identification tasks, which meets the requirements of rapid on-site detection while ensuring classification performance.

Description

Technical Field

[0001] The present invention relates to the fields of cyberspace security governance and supervision, and network traffic detection in a big data environment. It is specifically used to detect and identify open source proxy software traffic on network infrastructure equipment, making it possible to discover network anomalies or perform traffic scheduling as quickly as possible. Background Art

[0002] As users' demands for privacy and security continue to rise, specialized network services such as proxy servers are gaining popularity. Proxy services offer a variety of functions, such as increasing access speeds, controlling access to internal resources, concealing real IP addresses, and bypassing regional website restrictions. While this indirect connection between one terminal and another helps protect the privacy of network terminals, it also creates an anonymous environment for network sabotage and attacks. This has become a breeding ground for various cybercrimes, including dark web transactions, extortion, data theft, and pornography distribution, posing a significant challenge to cybersecurity regulation.

[0003] In recent years, a number of well-designed open-source proxy software, such as Shadowsocks, V2Ray, and Trojan, have emerged. Unlike traditional SOCKS and HTTP-based proxies, these utilize proprietary secure communication protocols and support multiple protocol configurations. This flexible, customized solution meets the diverse needs of building anonymous and secure private networks and has gained widespread adoption. Furthermore, by releasing these projects into the open-source community, these open-source proxy software programs have not only undergone technological iterations but also achieved widespread dissemination within a few years. Due to their powerful functionality and ease of deployment, open-source proxy software has also become a crucial tool for criminals to engage in black market transactions, conduct cyber extortion, and spread online rumors. A reasonable solution to these problems is to identify open-source proxy software traffic on network infrastructure. Only by identifying open-source proxy software traffic on the network infrastructure can we subsequently detect network anomalies and implement traffic scheduling.

[0004] The rise of open-source proxy software and the research on its traffic identification technology are two sides of the same coin in cyberspace attack and defense. Compared to the rapid evolution of open-source proxy software, the development of traffic identification technology has been relatively slow. While researchers have conducted some research on open-source proxy software traffic, two pressing challenges remain. First, there is a lack of high-quality open-source proxy software traffic datasets. Different types of open-source proxy software support a variety of protocol configurations, resulting in an increasing number of open-source proxy software traffic types. Existing research has relied on self-collected datasets covering only one or two traffic types. There is no publicly available open-source proxy software traffic dataset for subsequent researchers to utilize, hindering technical communication and dissemination. Second, there is a lack of efficient and well-defined traffic identification solutions. Previous research on open-source proxy software traffic identification has focused on different areas. Some have focused on identifying open-source proxy software traffic from ordinary encrypted traffic or on performing website fingerprinting attacks on open-source proxy software traffic. These studies have addressed significant differences in their research, with some preferring to improve machine learning methods rather than focusing on the traffic identification problem itself. Therefore, it is necessary to propose a traffic identification solution for open-source proxy software that clearly defines the key tasks in open-source proxy software traffic identification research. As most identification models utilize machine learning algorithms, this article will also utilize them when discussing traffic identification. However, the key to this research lies in leveraging traffic characteristics to improve classification performance based on machine learning theory and feature optimization. Related research has proposed numerous beneficial features from various perspectives for training machine learning models and achieving excellent performance. A universal method is needed to evaluate these features and construct an optimal feature set for specific classifiers and identification tasks to make the traffic identification process more efficient. In summary, obtaining high-quality datasets and developing efficient traffic identification solutions are pressing challenges for open source proxy software traffic identification.

[0005] Based on the above problems, a method for identifying open source proxy software traffic based on feature optimization is proposed for typical open source proxy software such as Shadowsocks, Trojan, and V2Ray. Compared with previous work, the innovation of this invention lies in: (1) building an automatic collection platform for open source proxy software traffic and constructing a high-quality data set covering different open source proxy software and different protocol configurations; (2) proposing an efficient traffic identification solution based on feature optimization. On the one hand, this solution focuses on three different classification tasks and performs multi-faceted identification of open source proxy software traffic; on the other hand, it proposes a two-dimensional two-stage feature selection algorithm (TT), which not only achieves the improvement of classification performance but also meets the requirements of rapid on-site detection. Summary of the Invention

[0006] This paper examines network traffic detection in a big data environment. Addressing the challenges of identifying open-source proxy software traffic on network infrastructure, and based on an in-depth analysis of current open-source proxy software, it proposes a feature-optimized method for identifying and classifying open-source proxy software traffic. This method addresses the challenges of building high-quality datasets, multi-task classification, and constructing optimal feature subsets. The method comprises two main steps: traffic collection and data preprocessing for open-source proxy software, and identification of open-source proxy software traffic.

[0007] S1: Traffic collection and data preprocessing for open source proxy software. This process is divided into two steps: traffic collection for open source proxy software and traffic preprocessing for open source proxy software.

[0008] S11: Traffic collection for open source proxy software. This process is divided into three steps: building a cloud-based traffic collection platform, writing scripts to simulate user access traffic, and capturing and storing open source proxy software traffic.

[0009] (1) Build a cloud-based traffic collection platform. Figure 1 As shown in the figure, the entire traffic collection platform is divided into the client, server and application sides. The specific steps are as follows:

[0010] a) Client deployment. Open source proxy software is installed not only on the local host but also on rented cloud servers. Different deployment methods are expected to increase the diversity of traffic sources;

[0011] b) Server-side deployment: Rent several cloud servers and deploy open-source proxy software on them to provide proxy services. The same open-source proxy software has multiple configuration options when deployed, and different options will result in different characteristics of the generated network traffic.

[0012] c) Application deployment. Select several websites as access targets;

[0013] d) Connectivity test. After setting up the client, server, and application, test the connectivity of the traffic collection platform. If the client can access the application through the server, the platform is complete.

[0014] (2) Write a script to simulate user access traffic. Use crawler technology to manipulate the browser to access all websites on the application side to achieve automatic generation of network traffic. The specific steps are as follows:

[0015] a) Write a script to generate open source proxy software traffic. First, write a script to simulate the various behaviors of human users visiting the target website, such as daily clicks, searches, and page turning. Write corresponding access scripts for all websites on the application side to achieve automatic generation of traffic. Second, some websites can identify the access requests generated by the script and continuously pop up human-machine verification windows to interrupt the traffic collection process. In this case, control the mouse and keyboard to automatically interact with the browser to achieve automatic control of the mouse and keyboard to slide down, turn pages, and other operations. Algorithm 1 shows the specific steps for generating open source proxy software traffic;

[0016] b) Write a script to generate traffic from the non-open-source proxy software. Similar to the method for generating traffic from the open-source proxy software, this script still simulates daily user clicks, searches, and page flips when accessing the target website. However, the access requests to the target application website are sent directly from the server host rather than the client host. Algorithm 2 shows the specific steps for generating traffic from the non-open-source proxy software.

[0017] (3) Capture and store the traffic of the open source proxy software. To ensure that the collected traffic is sufficiently reliable, listen to the corresponding port number of the deployed open source proxy software and use the traffic capture tool to complete the data capture of the corresponding port. In addition, the IP addresses and port numbers of the communicating parties are recorded as logs to facilitate subsequent data processing. A traffic storage method that cooperates with the cloud server and cloud disk is used. After the traffic collection for a day is completed, the captured traffic is synchronized to the cloud.

[0018]

[0019]

[0020] S12: Traffic preprocessing for open-source proxy software. To ensure that the traffic contains as much valid information as possible, the raw traffic obtained in step S11 must be preprocessed before it can be used as a valid dataset for experiments. This process is divided into three steps: traffic diversion and filtering, traffic labeling, and traffic undersampling.

[0021] (1) Divide and filter. First, the original traffic data is divided into five-tuples. A set of data packets with the same five-tuple is defined as a flow. Second, the original traffic content is preliminarily analyzed using traffic analysis tools. The flow is filtered according to its geographical location and flows that do not meet the conditions are eliminated.

[0022] (2) Traffic labeling. Based on the collected logs in step S11(3), each flow can be accurately labeled with three types of labels. The first type of label is used to distinguish whether the flow belongs to open source proxy software traffic. Once a flow is labeled as open source proxy software traffic, it is necessary to further determine which protocol configuration is used and which application website is accessed. For multi-classification tasks, these two types of labels contain several different values, which are the same as the type of protocol configuration and the number of application websites.

[0023] (3) Traffic undersampling. In order to provide a balanced dataset for the traffic identification model, all labeled flows need to be undersampled. For each website visited under each open source proxy software protocol configuration, several flows are randomly selected from a set of qualified samples. The transport layer protocols used by these flows are different. TCP and UDP flows need to be evenly distributed in all selected samples in proportion. In addition, the same number of non-open source proxy software flows as the open source proxy software flows need to be sampled, where the proportion of TCP flows and UDP flows remains unchanged.

[0024] S2: Identification of open source proxy software traffic. This part is divided into four steps: building a formal identification framework, constructing a complete feature set of open source proxy software traffic, designing a feature optimization selection algorithm, training the model, and identifying open source proxy software traffic.

[0025] S21: Construct a formal identification framework. After processing in step S12, all the raw data of network traffic is organized into flows as the basic unit and a formal method is used to define the open source proxy software traffic identification work. It is divided into 7 steps:

[0026] (1) Divide the traffic data captured in S1 into a training set With the test set

[0027] (2) Each flow is considered as a sample. Assume that the training set contains m flows, denoted as The test set contains n flows, denoted as Set 3 different types of label sets for each flow. Mark whether the current flow is open source proxy software traffic; Corresponding to K types of open source proxy software protocol configurations, Corresponding to R typical access applications;

[0028] (3) Extract features of different dimensions for a stream and combine all computable features into a complete feature set The specific feature extraction process is shown in step S22;

[0029] (4) From the full set of features Select k features from the dataset to construct a feature subset for a specific task.

[0030] (5)Use exist The feature matrix C constructed on X ,as well as exist and The value on Train a given machine learning classifier;

[0031] (6) According to exist The characteristic matrix C on Y , using the trained classifier to predict exist and The results on Among them, C X 、 The definition of is shown in formula (1) and (2), C Y 、 The definition of C is similar. X It is an m×k matrix, each row represents a flow, each column represents a feature of the flow, v ij represents the jth feature of the i-th stream. are column vectors containing m elements, which together constitute The label matrix L of all samples X ;

[0032]

[0033]

[0034] (7) According to the different levels of traffic perception, the identification of open source proxy software traffic is divided into three tasks: T1, T2, and T3. Figure 2 As shown, each classification task is completed using a classifier. In T1, the training set With the test set Both contain non-open source proxy software flow samples, and in T2 and T3, the training set With the test set Only samples of open source proxy software flows are included. The definitions of T1, T2, and T3 are as follows:

[0035] a) T1: C X and As input, use the given classifier to train the model, and then C Y Identify the test set as input Whether each flow in is open source proxy software traffic, and then obtain

[0036] b) T2: and As input, the model is trained with the given classifier, and then Identify the test set as input The open source proxy software protocol configuration used by each flow in

[0037] c) T3: and As input, the model is trained with the given classifier, and then Identify the test set as input The network application accessed by each flow in the

[0038] S22: Construct the complete feature set of open source proxy software traffic. This process is the expansion of S21(3) and is divided into the following 8 steps:

[0039] (1) All captured traffic is divided into TCP-based flows (referred to as TCP flows) and UDP-based flows (referred to as UDP flows). Since there are huge differences in features between the two, for example, the header of a TCP message contains multiple fields and flags, while the UDP message is much simpler. In order to facilitate the subsequent modeling using machine learning methods, all traffic captured in step S1 is first divided into TCP flows and UDP flows;

[0040] (2) Extract basic information of TCP and UDP flows, including the direction, duration, number of sent / received packets, number of sent / received bytes, and transmission rate of a flow;

[0041] (3) Extract statistical information about packets in TCP and UDP flows, including the distribution characteristics of packet lengths and the distribution characteristics of packet arrival time intervals within a flow. Both distribution characteristics include maximum, minimum, mean, lower quartile, median, upper quartile, variance, and standard deviation.

[0042] (4) Extract relevant statistical information of the TCP and UDP stream network layers, including the IP header information of a stream, such as minimum TTL, maximum TTL, number of IP TTL changes, IP service type, etc.;

[0043] (5) Extracting protocol-related statistical information of TCP flows, including TCP header information of a flow during its duration, such as initial sequence number, number of sequence number failures, number of bytes received without defective ACKs, number of ACK number failures, initial effective window size, average effective window size, TCP option count, etc.;

[0044] (6) Extract statistical information related to the SSL protocol in the TCP stream, including the status of the SSL protocol, the decoded application layer protocol type and category, the SSL protocol version number, etc.

[0045] (7) Two unreasonable features were deleted through manual analysis: the first was features that were repeated in statistics; the second was features that interfered with machine learning modeling, such as the source and destination IP addresses of the flow;

[0046] (8) Constructing a complete feature set for TCP and UDP flows respectively and

[0047] S23: Design a feature optimization selection algorithm. Whether in the field of machine learning or traffic identification research, using the full set of features to train the model not only cannot guarantee the optimal results but also introduces a lot of overhead. For specific identification tasks and selected classifiers, extracting the optimal feature subset based on the full set of features is the key to building a high-quality model. In addition, as more and more traffic features are proposed, how to construct the optimal feature subset under the premise of limiting the size of the feature set is a key issue in helping the model to quickly improve its performance in practice. To address the above problems, this framework proposes a two-dimensional two-phase feature selection algorithm (TT), which can construct the optimal feature subset for different tasks of open source proxy software traffic identification for different classifiers under the premise of limiting the set size. As shown in Algorithm 3, TT is divided into two stages, specifically the following five steps:

[0048]

[0049]

[0050] (1) In the first stage, a filtering method is used to evaluate the features in the feature set constructed by S22. One dimension considers the correlation between features and labels, using the chi-square (CS) method; the other dimension considers the correlation between features, using the mRMR method. After calling the above method for sorting, two different lists are obtained;

[0051] (2) Divide the above two lists into several sublists in order and construct two sets and

[0052] (3) In the second stage, the packaging method is used to construct the optimal feature subset. The TT algorithm sets multiple rounds. In the first round, select and The corresponding sublists in the dataset are used as feature sets to train the classifier and perform test comparisons. The sublists with higher accuracy are selected as the first candidate list, and the lower ones as the second candidate list. After the first round of comparison, the sublists with better performance will be directly added to the optimal feature subset.

[0053] (4) The sublists of subsequent rounds are arranged in the same order as the first round. After completing the comparison of the sublists, try to add the features in the two sublists to the current optimal feature subset in sequence. If the expanded set can improve the classifier performance, then it will be used as the new optimal feature subset;

[0054] (5) The optimal feature subset will be continuously expanded according to the above method until the following three situations occur: 1) The number of features in the optimal feature subset has reached the set number N; 2) and The features contained in all sublists in have been tried to expand the optimal feature subset; 3) After a round of execution, the optimal feature subset has not been expanded.

[0055] S24: Train the model and identify open source proxy software traffic. This process selects different classification algorithms to classify open source proxy software traffic. It is divided into the following seven steps:

[0056] (1) Select a machine learning classification algorithm as the classifier in the open source proxy software traffic identification framework;

[0057] (2) Divide the dataset into training dataset and test dataset;

[0058] (3) Use cross-validation to adjust the parameters of the classification model on the training data set to confirm the optimal parameters of the classification model;

[0059] (4) Determine the classification label and classification target according to the specific classification task;

[0060] (5) Obtaining the optimal feature subset under the specific classification model according to step S23;

[0061] (6) Use the trained model to classify the test data set and calculate the classification performance;

[0062] (7) Repeat steps (2)-(6) several times and take the mean of the evaluation index as the final result under the specific classification algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Schematic diagram of the structure of the open source proxy software traffic collection platform;

[0064] Figure 2 Schematic diagram of the three classification tasks of the open source proxy software traffic identification framework;

[0065] Figure 3 Schematic diagram of the overall framework of open source proxy software traffic identification in the embodiment;

[0066] Figure 4 Flowchart of the automated script for accessing Amazon Japan Mall in the embodiment;

[0067] Figure 5 Flowchart of the automated script for controlling the keyboard and mouse using PyAutoGUI in the embodiment;

[0068] Figure 6 Flowchart of pre-processing of original traffic in the embodiment;

[0069] Figure 7 Specific flow chart of feature selection algorithm in the embodiment;

[0070] Figure 8 Flowchart of traffic classification using six machine learning algorithms in the embodiment.

[0071] Advantages of the present invention

[0072] This paper proposes a method for identifying open source proxy software traffic based on feature optimization. Compared with other traffic identification methods for open source proxy software, this method has the following advantages:

[0073] (1) This solution builds a large-scale cloud-based traffic collection and processing platform. First, in step S11, a traffic collection platform construction plan, user behavior simulation script, and traffic capture and storage plan are designed to ensure the automated generation and collection of open source proxy software traffic. Secondly, in step S12, the preprocessing process of open source proxy software traffic is described. Through traffic filtering, labeling, and undersampling steps, the traffic contains as much valid information as possible, providing a basis for subsequent classification and identification as an effective data set. After step S1, a high-quality traffic data set covering a variety of open source proxy software with different configurations is obtained. This data set includes two parts: open source proxy software traffic and non-open source proxy software traffic. In addition to being suitable for binary classification, it can also be used in scenarios where multiple classifications are performed on specific protocols or access content, providing a research foundation for the field of open source proxy software traffic analysis.

[0074] (2) The scheme designs an efficient identification method with three independent tasks based on feature optimization. First, in step S21, a traffic identification framework is defined in a formal way, making the entire traffic identification process clear and straightforward. Secondly, three different classification tasks are defined, first exploring whether the proxy server traffic can be classified into two categories. In the case of a positive result, on the one hand, the protocol type used by the proxy service is explored, and on the other hand, the applications or services hidden in it are inferred, achieving supervision from different angles and depths. In addition, an open source proxy software traffic feature extraction scheme is designed. While completing the construction of a full set of multi-faceted features for TCP traffic and UDP traffic, it continues to provide value as input to subsequent feature selection algorithms. Finally, a two-stage feature selection algorithm (TT) based on dual dimensions is proposed to construct the optimal feature subsets for different tasks under different classifiers, which not only ensures the classification performance but also meets the requirements of rapid on-site detection. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments are described in detail below in conjunction with the accompanying drawings of the present invention. Obviously, the embodiment described below is only one embodiment of the method of the present invention, and not all embodiments. Based on the following embodiment, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0076] Example 1: A method for identifying open source proxy software traffic based on feature optimization, Figure 3 The overall framework of open source proxy software traffic identification in this embodiment is presented. The implementation process of the present invention is divided into two main steps: collection and data preprocessing of open source proxy software traffic and identification of open source proxy software traffic.

[0077] S1: Traffic collection and data preprocessing for open source proxy software. This process is divided into two steps: traffic collection for open source proxy software and traffic preprocessing for open source proxy software.

[0078] S11: Traffic collection for open source proxy software. This implementation process includes three steps: building a cloud-based traffic collection platform, writing a script to generate open source proxy software traffic, and capturing and storing open source proxy software traffic.

[0079] (1) Build a cloud-based traffic collection platform. The implementation process includes three steps: equipment preparation, protocol configuration and target application selection, and open source proxy service establishment;

[0080] a) Equipment Preparation. The equipment required for this embodiment of the present invention includes: four local hosts, two Alibaba Cloud servers, and several overseas servers rented from GoCloud and Vultr. Overseas servers are billed based on usage time and traffic.

[0081] b) Protocol configuration and target application selection. First, this embodiment selects Shadowsocks, V2ray and Trojan as the basic software for generating open source proxy traffic. Secondly, 8 different mainstream configuration schemes are selected from the three open source proxy software Shadowsocks, V2ray and Trojan. The specific protocol configurations are shown in Table 1. Finally, this embodiment mainly considers access to web applications. In order to make the generated traffic as consistent as possible with the daily needs of the public, 8 mainstream websites were selected from the top 150 Alexa rankings, covering as many categories as possible, such as portals, video websites, social networking sites, encyclopedias, shopping, publications, etc., as shown in Table 2.

[0082] c) Set up an open source proxy service. Install the CentOS7 X64 system on a rented overseas server. Use SSH to remotely connect to the server, turn off the server's firewall, and install Shadowsocks, V2Ray, and Trojan services. According to the 8 different protocol configurations, select different transmission protocols, port numbers, encryption protocols, user ID / UUIDs, camouflage types, etc. when configuring the proxy service configuration file. If the open source proxy software traffic is disguised as HTTPS website traffic, you will also need to register a domain name for the website separately. Install Shadowsocks, V2Ray, and Trojan clients on the local host and the rented Alibaba Cloud server, and configure the server. After the configuration is complete, enable PAC mode. Test network connectivity: If the local host or Alibaba Cloud server can access Google, the proxy service is set up successfully.

[0083] Table 1 8 specific protocol configurations of three proxy service software

[0084]

[0085] Table 2 covers 8 target websites of 6 categories of applications

[0086]

[0087] (2) Write a traffic generation script for the open source proxy software. To automatically generate network traffic, use two crawler technologies, Selenium and PyAutoGUI, to manipulate the browser to access all target websites on the application side. This is divided into the following three steps:

[0088]

[0089] a) First, download the Chrome driver for the Chrome browser version on your local host and the rented Alibaba Cloud server. Selenium provides a simple API to access all web browser features. Use this API to write Python scripts to simulate the various behaviors of human users visiting the aforementioned websites.

[0090] b) For Amazon, New York Times, Reddit, Wikipedia, and Youtube, use Selenium API to write corresponding access scripts to achieve automatic traffic generation. Taking access to Amazon Japan as an example, the script process is as follows Figure 4 As shown in the figure, this script randomly selects a word from a random vocabulary, enters it into the search box, and submits it. For each product result on the results page, it clicks and jumps to the bottom of the subpage, closes the subpage, returns to the parent page, and then accesses the next product. This entire process mimics the user's search, click, and browse process for a product, generating traffic.

[0091] c) Google, Facebook, and Twitter websites can identify access requests generated by scripts and constantly pop up human-machine verification windows to interrupt the traffic collection process. In this case, the second crawler technology is used, that is, using PyAutoGUI to control the mouse and keyboard to automatically interact with the browser, and realize automatic control of the mouse and keyboard to perform operations such as scrolling down and turning pages. Algorithm 4 shows the steps of writing access scripts using PyAutoGUI. The script flow is as follows: Figure 5 shown.

[0092] (3) Capture and store open source proxy software traffic. In this embodiment, after traffic is generated, it is collected and stored according to the following five steps:

[0093] a) Listen to the corresponding port number of the open source proxy software and use Tcpdump to capture data from the corresponding port on the rented overseas server;

[0094] b) A .pcap file will be generated for every 100M traffic collected. The .pcap file will be named according to "protocol configuration + website name + system time";

[0095] c) Record the IP addresses and port numbers of both communicating parties as logs to facilitate subsequent data processing;

[0096] d) Using a traffic conservation method that combines cloud servers and cloud disks, we purchased Microsoft OneDrive services, virtualized the cloud disk into a disk using rclone technology, and mounted it on the file system of the cloud server (CentOS7);

[0097] e) Synchronize the collected raw traffic to OneDrive regularly on a daily basis.

[0098] S12: Traffic preprocessing for open source proxy software. This step preprocesses the original traffic obtained in S11, such as Figure 6 As shown, it is divided into the following 7 steps:

[0099] (1) Use Wireshark to conduct preliminary analysis of the original traffic content;

[0100] (2) Use the Tranalyzer2 tool to extract the flow of the original .pcap file according to the five-tuple;

[0101] (3) Eliminate background traffic. Filter out traffic that does not meet the requirements for communication between the client and the server based on the geographic location represented by the "src / dstIP" of a flow. For traffic accessing specific applications, eliminate flows that do not meet the requirements based on the dstIPOrg or sslServerName of a flow.

[0102] (4) Eliminate streams with a send byte count or a receive byte count of 0. There is not enough interactive information between the client and the server, and analysis of such invalid streams should be avoided;

[0103] (5) Accurately label each flow with three types of labels based on the experimental logs. The first type is used to distinguish whether the flow belongs to open source proxy software traffic and only involves two values. Once a flow is marked as open source proxy software traffic, it is necessary to further determine which protocol configuration is used and which application website is accessed. As shown in Tables 2 and 3, the last two types of labels each contain 8 different values;

[0104] (6) Undersample all labeled flows. For each website visited under each open source proxy software protocol configuration, 1,000 flows are randomly selected from a set of qualified samples. The dataset contains a total of 64,000 open source proxy software flows. In Table 2, four protocol configurations use TCP, while the other four use UDP, so TCP and UDP flows are evenly distributed in all selected samples. In addition, 64,000 non-open source proxy software flows are sampled, half of which are TCP flows and the other half are UDP flows;

[0105] (7) A high-quality dataset containing 128,000 flows is obtained.

[0106] S2: Identification of open source proxy software traffic. The implementation process specifically includes the following three steps: constructing a complete feature set of open source proxy software traffic, using a feature optimization selection algorithm to construct an optimal feature subset, and training a model to identify open source proxy software traffic.

[0107] S21: Construct a complete feature set of open source proxy software traffic. After the above step S12, a data set based on flows is obtained. For each flow, features of different dimensions are extracted. This process is divided into the following four steps.

[0108] (1) All captured traffic is divided into TCP streams and UDP streams;

[0109] (2) Use basic plug-ins such as protoStats, basicFlow, basicStats, tcpFlags, tcpStates, descriptiveStats, and sslDecode in Tranalyzer2 to extract features for the flow;

[0110] (3) Delete two unreasonable features. The first is repeated statistical features; the second is features that interfere with machine learning modeling, such as the source and destination IP addresses of the flow;

[0111] (4) A total of 101 features are constructed for TCP flows and 51 features are constructed for UDP flows. The specific feature names are shown in Tables 3 and 4.

[0112] Table 3 Complete set of TCP traffic characteristics

[0113]

[0114] Table 4 Complete set of UDP traffic characteristics

[0115]

[0116] S22: Use the feature optimization selection algorithm to construct the optimal feature subset. This step uses the two-stage feature selection algorithm TT based on dual dimensions to select the optimal feature subset from the full set of features obtained in S21 as the input of the classification model. Figure 7 As shown, it is divided into the following 4 steps:

[0117] (1) Set the input parameter s of the TT algorithm to 10, that is, the size of the divided sublist is 10, and the number of optimal feature subsets is set to 20;

[0118] (2) In the first stage of TT, the CS and mRMR algorithms are used to sort the entire feature set and divide it into several sublists in order;

[0119] (3) In the second stage of TT, the first 10 features after sorting are used as the feature set to train the classifier and perform tests and comparisons. The sublist with higher accuracy is directly added to the optimal feature subset;

[0120] (4) In subsequent rounds, the features in the two sublists that can improve the classifier performance are continuously added to the optimal feature subset in sequence until the number of optimal feature subsets is reached or all the features contained in the sublists have been tried.

[0121] S23: Train the model and identify the open source proxy software traffic. In this embodiment, six common machine learning algorithms are selected to identify the open source proxy software traffic, including K-nearest neighbor algorithm (KNN), support vector machine algorithm (SVM), naive Bayes algorithm (NB), random forest algorithm (RF), extreme gradient boosting algorithm (XGBoost) and multi-layer perceptron algorithm (MLP). These algorithms represent simple nonlinear, linear, probabilistic, tree-type, integrated and artificial neural network machine learning models, respectively. In this embodiment, the experiment was conducted using a five-fold cross-validation method of ten times, and the accuracy (ACC) and F1-Measure (F1) were selected as indicators to measure the recognition performance. As Figure 8 As shown in the figure, the classification process of this part is divided into the following 7 steps:

[0122] (1) After processing in step S22, a feature data set D is obtained;

[0123] (2) Obtain corresponding data labels according to the classification task;

[0124] (3) Perform maximum and minimum normalization on the feature data set D to obtain the data set

[0125] (4) Split into training set in a ratio of 4:1 and test set

[0126] (5) Determine the classifier algorithm ε, Perform five-fold cross validation to obtain the optimal model parameters of algorithm ε;

[0127] (6) Using the test set Test the effect of the trained model and calculate the ACC and F1 values;

[0128] (7) Repeat the process (4)-(6) 10 times and calculate the average value and standard deviation as the classification effect of the classifier algorithm ε on this task.

[0129] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Equivalent replacements or substitutions made on the basis of the above technical solutions all fall within the scope of protection of the present invention.

Claims

1. A method for identifying open source proxy software traffic based on feature optimization, characterized in that: The method comprises the following steps: S1: Traffic collection and data preprocessing for open source proxy software; S2: Identification of open source proxy software traffic; S1: Traffic collection and data preprocessing for open source proxy software, specifically as follows: S11: Traffic collection for open source proxy software; S12: Traffic preprocessing for open source proxy software; Among them, S11: Traffic collection for open source proxy software is divided into three steps: building a cloud-based traffic collection platform, writing scripts to simulate user access traffic, and capturing and storing open source proxy software traffic. S12: Traffic preprocessing for open source proxy software, which is divided into three steps: traffic diversion and filtering, traffic labeling, and traffic undersampling. Among them, S2: Identification of open source proxy software traffic is divided into 4 steps: S21: Constructing a formal identification framework, S22: Constructing a complete feature set of open source proxy software traffic, S23: Designing a feature optimization selection algorithm, and S24: Training the model and identifying open source proxy software traffic. Among them, S21: Construct a formal identification framework. After processing in step S12, all the original data of network traffic are organized into flows as basic units and the open source proxy software traffic identification work is defined using a formal method. It is specifically divided into 7 steps: (1) Divide the traffic data captured in S1 into a training set With the test set (2) Each flow is regarded as a sample. Suppose the training set contains m flows, which are recorded as The test set contains n flows, denoted as Set 3 different types of label sets for each flow. Mark whether the current flow is open source proxy software traffic; Corresponding to K types of open source proxy software protocol configurations, Corresponding to R typical access applications; (3) Extract features of different dimensions for a stream and combine all computable features into a complete feature set (4) From the full set of features Select k features from the dataset to construct a feature subset for a specific task. (5)Use exist The feature matrix C constructed on X ,as well as exist and The value on Train a given machine learning classifier; (6) According to exist The characteristic matrix C on Y , using the trained classifier to predict exist and The results on Among them, C X 、 The definition of is shown in formula (1) and (2), C Y 、 The definition of C is similar to this. X It is an m×k matrix, each row represents a flow, each column represents a feature of the flow, v ij represents the jth feature of the i-th stream, are column vectors containing m elements, which together constitute The label matrix L of all samples X ; (7) According to the different angles and granularities of traffic identification, the identification of open source proxy software traffic is divided into three tasks: T1, T2, and T3. Each task is completed using a classifier. In T1, the training set With the test set Both contain non-open source proxy software flow samples, and in T2 and T3, the training set With the test set This sample only contains open source proxy software flows. The definitions of T1, T2, and T3 are as follows: a) T1: C X and As input, use the given classifier to train the model, and then C Y Identify the test set as input Whether each flow in is open source proxy software traffic, and then obtain b) T2: and As input, the model is trained with the given classifier, and then Identify the test set as input The open source proxy software protocol configuration used by each flow in c) T3: and As input, the model is trained with the given classifier, and then Identify the test set as input The network application accessed by each flow in 2. The open source proxy software traffic identification method based on feature optimization according to claim 1 is characterized in that: S22: Construct a complete feature set of open source proxy software traffic, which is divided into the following 8 steps: (1) All captured traffic is divided into TCP-based flows and UDP-based flows. Due to the huge feature differences between the two, in order to facilitate the subsequent modeling using machine learning methods, all traffic captured in step S1 is first divided into TCP flows and UDP flows; (2) Extract basic information of TCP and UDP flows, including the direction, duration, number of sent / received packets, number of sent / received bytes, and transmission rate of a flow; (3) Extract statistical information of packets in TCP and UDP flows, including the distribution characteristics of packet lengths and the distribution characteristics of packet arrival time in a flow. Both distribution characteristics include maximum, minimum, mean, lower quartile, median, upper quartile, variance, and standard deviation. (4) Extract relevant statistical information of the TCP and UDP stream network layers, including the IP header information of a stream, minimum TTL, maximum TTL, number of IP TTL changes, and IP service type; (5) Extract protocol-related statistical information of TCP flows, including TCP header information, initial sequence number, number of sequence number failures, number of bytes received without defects in ACKs, number of ACK number failures, initial effective window size, average effective window size, and TCP option count for a flow during its duration; (6) Extracting statistical information related to the SSL protocol in the TCP stream, including the state of the SSL protocol, the decoded application layer protocol type and category, and the version number of the SSL protocol; (7) Two unreasonable features were deleted through manual analysis. The first one was the feature of repeated statistics. The second type is features that interfere with machine learning modeling; (8) Constructing a complete feature set for TCP and UDP flows respectively and 3. The open source proxy software traffic identification method based on feature optimization according to claim 2 is characterized in that: S23: Design a feature optimization selection algorithm, which is divided into the following five steps: (1) In the first stage, a filtering method is used to evaluate the features in the feature set constructed by S22. One dimension considers the correlation between features and labels, using the chi-square (CS) method; the other dimension considers the correlation between features, using the mRMR method. After calling the above methods for sorting, two different lists are obtained; (2) Divide the above two lists into sublists in order and construct two sets and (3) In the second stage, the packaging method is used to construct the optimal feature subset. The TT algorithm sets multiple rounds. In the first round, and The corresponding sublists in the training set are used as feature sets to train the classifier and perform test comparisons. The sublists with high accuracy are used as the first candidate list, and the sublists with low accuracy are used as the second candidate list. After the first round of comparison, the sublists with better performance will be directly added to the optimal feature subset. (4) The sublists of subsequent rounds are arranged in the same order as the first round. After the sublists are compared, try to add the features in the two sublists to the current optimal feature subset in sequence. If the expanded set can improve the classifier performance, then it will be used as the new optimal feature subset. (5) The optimal feature subset will be continuously expanded according to the above method until the following three situations occur: 1) The number of features in the optimal feature subset has reached the set number N; 2) and All features contained in the sublists in have been tried to be used to expand the optimal feature subset; 3) After a round of execution, the optimal feature subset is not expanded because the classifier performance is not improved.

4. The open source proxy software traffic identification method based on feature optimization according to claim 3 is characterized in that: S24: Train the model and identify open source proxy software traffic. This process selects different classification algorithms to classify open source proxy software traffic. It is divided into the following seven steps: (1) Select a machine learning classification algorithm as the classifier in the open source proxy software traffic identification framework; (2) Divide the dataset into training dataset and test dataset; (3) Use cross-validation to adjust the parameters of the classification model on the training data set to confirm the optimal parameters of the classification model; (4) Determine the classification label and classification target according to the specific classification task; (5) Obtaining the optimal feature subset under the specific classification model according to step S23; (6) Use the trained model to classify the test data set and calculate the classification performance; (7) Repeat steps (2)-(6) and take the mean of the evaluation index as the final result under the specific classification algorithm.

Images