Cloud website identification method and device based on HTTP version inference

Abstract

The invention discloses a cloud website identification method and device based on HTTP version inference, and belongs to the technical field of encrypted traffic analysis. The method comprises the following steps: dividing network traffic into network flows; extracting flow characteristics of the network flow, and encoding the flow characteristics to obtain a characteristic matrix of the network flow; wherein the flow characteristics comprise HTTP version information used by the network flow and the number of borne network resources; and obtaining a website identification result based on the feature matrix. According to the method, a robust and accurate lightweight website identification technical solution can be provided for real applications such as network supervision and tracking of network criminal activities.

Description

Technical Field

[0001] The present invention relates to the technical field of encrypted traffic analysis, and in particular to a cloud website identification method and device based on HTTP version inference. Background Art

[0002] With the widespread use of cloud computing, more and more websites and services are deployed in cloud environments. However, cloud environments have some unique characteristics that make it particularly complex and challenging to identify the websites visited by users from encrypted traffic. First, the number of hosts in cloud environments is huge, and a single cloud server often hosts multiple websites or application services. At the same time, due to limited public IP resources, many cloud hosts do not directly have independent public IP addresses, but communicate through shared egress IPs. In other words, multiple websites may share the same egress IP, which increases the difficulty of accurately identifying the specific target website visited.

[0003] In this context, how to effectively identify the specific websites on the cloud that users visit from encrypted traffic has become a key technical requirement. This is crucial for network supervision and tracking of cybercrime activities, especially when facing an increasing number of cloud services using encrypted protocols (such as HTTPS), traditional identification methods based on plaintext information are no longer applicable. Therefore, developing technology that can accurately identify the websites visited by users in an encrypted traffic environment can not only improve network security and management capabilities, but also effectively deal with malicious use of cloud platforms to cover up criminal activities.

[0004] Existing encrypted traffic website identification technologies are mainly divided into two categories: identification methods based on side channel information and identification methods based on traffic characteristics and classification models.

[0005] First, the identification method based on side-channel information relies on metadata information exposed outside of HTTPS encryption to identify websites. Although the HTTPS protocol encrypts the transmitted content, unlike the Tor anonymous network, there is still some side-channel information in the modern Internet architecture that can be exploited. For example, by looking up the resolved domain name through the IP address at the network layer, it is possible to infer the website visited by the user; at the security layer, the SNI (Server Name Indication) field in the TLS handshake directly indicates the target website visited; the plaintext DNS request at the application layer also explicitly exposes the website information visited by the user.

[0006] Secondly, the identification method based on traffic characteristics and classification models mainly analyzes the transmission characteristics of encrypted traffic, such as packet size, direction sequence, and time interval between packets. This method usually actively collects traffic samples from different websites, extracts their transport layer characteristics, and then uses machine learning or deep learning models for classification and identification under the framework of supervised learning. This type of method is widely used in existing encrypted traffic classification work, especially for identifying encrypted HTTPS traffic.

[0007] However, with the evolution of network technology, existing technologies are facing more and more challenges (summarized as Figure 1 ). First, the identification method based on side channel information is gradually becoming ineffective. The Internet Engineering Task Force (IETF) is actively launching a series of new technologies to solve the anonymity problem of HTTPS. For example, encrypted DNS protocols (such as DoH, DoT, DoQ) effectively solve the problem of privacy leakage of plaintext DNS; TLS ECH technology hides the real SNI information in the traffic, further improving user privacy; in addition, with the widespread application of cloud environments and the popularity of CDN, more and more websites share the same IP address, forming an IP co-location phenomenon, making the reverse query method based on IP address no longer reliable.

[0008] At the same time, classification models based on traffic features also face many limitations in practical applications. These methods have poor robustness and low processing efficiency in actual production environments, which affects their availability and application prospects. Specifically, existing methods rely on traffic characteristics at the transport layer, which are closely related to network transmission conditions and are therefore easily affected by different network environments. In actual production environments, it is difficult for users to exhaust all possible network environments, which leads to inconsistencies between training data and test data, thereby reducing the robustness of the model. In addition, existing methods usually require a large number of training samples to support the accuracy of the model, and once the performance of the model deteriorates, it is necessary to re-collect new data and retrain the model, which is not only time-consuming, but also has high complexity and cost in actual deployment and maintenance.

[0009] The above problems indicate that the existing encrypted traffic website identification technology is gradually losing its effectiveness in the face of the evolving network environment, especially the cloud environment, and a more robust and efficient new method is urgently needed. Summary of the invention

[0010] In view of the fact that existing encrypted traffic identification solutions are gradually failing in cloud website identification scenarios and lack robustness and availability, the present invention proposes a cloud website identification method and device based on HTTP version inference, providing a robust and accurate lightweight website identification technology solution for real-world applications such as network supervision and tracking of cybercrime activities.

[0011] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0012] A method for identifying a website on the cloud based on HTTP version inference, the method comprising:

[0013] Divide network traffic into network flows;

[0014] Extracting the traffic features of the network flow, and encoding the traffic features to obtain a feature matrix of the network flow; wherein the traffic features include: HTTP version information used by the network flow and the number of network resources carried;

[0015] A website recognition result is obtained based on the feature matrix.

[0016] Furthermore, dividing the network traffic into network flows includes:

[0017] Collect network traffic from vantage points on the network;

[0018] The traffic is divided into different network flows in the form of traffic quintuples, wherein the traffic quintuples include: source IP, source port, destination IP, destination port and transport layer protocol.

[0019] Furthermore, the extracting of the traffic characteristics of the network flow includes:

[0020] Determine the start position of the HTTP phase through specific flags of the TLS protocol or QUIC protocol in the network flow;

[0021] Infer the HTTP version information used by the network flow based on the initial data packet transmission sequence direction in the HTTP stage;

[0022] By analyzing the number of packets sent by the client during the HTTP phase, the number of network resources transmitted in the network flow can be inferred.

[0023] Furthermore, determining the starting position of the HTTP phase through a specific flag of the TLS protocol or the QUIC protocol in the network flow includes:

[0024] Obtaining the protocol in the network flow;

[0025] In the case where the protocol in the network flow is the TLS protocol, determining the starting position of the HTTP stage by determining whether the content type field in the TLS data packet header sent by the client is application data;

[0026] When the protocol in the network flow is the QUIC protocol, the starting position of the HTTP stage is determined by the change from a long header to a short header.

[0027] Further, the inferring HTTP version information used by the network flow according to the initial data packet transmission sequence direction in the HTTP stage includes:

[0028] Determine whether the network flow is encrypted traffic based on the QUIC protocol;

[0029] When the network flow is encrypted traffic based on the QUIC protocol, determining that the version information of the network flow is HTTP / 3;

[0030] In the case where the network flow is encrypted traffic based on the QUIC protocol, the transmission direction of the first two data packets in the HTTP stage is obtained, and when the transmission direction of the first two data packets is out-in, the version information of the network flow is determined to be HTTP / 1.1, and when the transmission direction of the first two data packets is out-out, the version information of the network flow is determined to be HTTP / 2.

[0031] Furthermore, the inferring the number of network resources transmitted in the network flow by analyzing the number of data packets sent by the client in the HTTP stage includes:

[0032] According to the HTTP version information, build a protocol state machine model and define the transition conditions;

[0033] The protocol state machine model processes data packets from the client in the idle state, and when the data packets from the client meet the transition conditions and trigger the resource request state, the corresponding counter will be incremented, thereby obtaining the number of network resources transmitted in the network flow;

[0034] Wherein, when the HTTP version information is HTTP / 2 or HTTP / 3, the influence of control packets in the HTTP / 2 or HTTP / 3 version is eliminated by adding an initial step that does not expand the counter and using a packet size threshold.

[0035] Furthermore, the traffic characteristics are encoded to obtain a feature matrix of the network traffic, including:

[0036] Let the HTTP version of the i-th network flow be HTTP i ;

[0037] The elements of the characteristic matrix defining the network traffic Among them, v∈{1,2,3}, 1 represents the number of resources R i The corresponding network flow is HTTP / 1.1 version, and 2 represents the number of resources R i The corresponding network flow is HTTP / 1.1 version HTTP / 2 version, 3 represents the number of resources Ri The corresponding network flow is HTTP / 3 version.

[0038] Furthermore, when the application scenario is a closed world scenario, obtaining a website recognition result based on the feature matrix includes:

[0039] Collect website fingerprints and build a reference sample set;

[0040] Applying a bandpass filter to the reference sample set and retaining samples having a resource quantity within a set range with respect to the network traffic;

[0041] Calculating the similarity between the retained sample and the feature matrix of the network traffic based on a combination of Wasserstein distance and longest common similar subsequence distance;

[0042] A nearest neighbor algorithm is applied to the similarity between the retained sample and the feature matrix of the network traffic to obtain a website identification result; wherein the website identification result is a category of the website corresponding to the network traffic.

[0043] Furthermore, when the application scenario is an open world scenario, obtaining a website recognition result based on the feature matrix includes:

[0044] Collect website fingerprints and build a reference sample set;

[0045] Applying a bandpass filter to the reference sample set and retaining samples having a resource quantity within a set range with respect to the network traffic;

[0046] Calculating the similarity between the retained sample and the feature matrix of the network traffic based on a combination of Wasserstein distance and longest common similar subsequence distance;

[0047] A combination of the K nearest neighbor algorithm and the Gini coefficient is applied to the similarity between the characteristic matrix of the retained sample and the network traffic to obtain a website identification result; wherein the website identification result is whether the website corresponding to the network traffic is a sample of the monitored set and a sample that does not belong to the monitored set.

[0048] A cloud website identification device based on HTTP version inference, the device comprising:

[0049] A preprocessing module for dividing network traffic into network flows;

[0050] A feature extraction module is used to extract the traffic features of the network flow and encode the traffic features to obtain a feature matrix of the network flow; wherein the traffic features include: HTTP version information used by the network flow and the number of network resources carried;

[0051] The website identification module is used to obtain a website identification result based on the feature matrix.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects.

[0053] 1) The present invention is based on a lazy learning solution paradigm, thus avoiding the entire model training phase and providing higher potential for flexible deployment of website recognition.

[0054] 2) The design based on the wasserstein distance and the LCSS distance of the present invention has achieved the best accuracy performance in small sample problems and different monitoring website sets, and the present invention further improves the accuracy by weighted summing the two.

[0055] 3) The feature design of the present invention based on HTTP version information and the number of network resources not only achieves good recognition accuracy in a conventional closed world, but also achieves good recognition effects in a closed world with concept drift scenarios and an open world. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Summarize existing technical solutions for cloud-based website identification.

[0057] Figure 2 This is a process flow chart of the system of the present invention.

[0058] Figure 3 Execute the flow chart for the protocol state machine.

[0059] Figure 4 Annotate protocol state machine elements.

[0060] Figure 5 This is the website recognition accuracy result.

[0061] Figure 6 Recognition accuracy across network environments.

[0062] Figure 7 Recognition accuracy for concept drift environments.

[0063] Figure 8 Results for open world scene site identification. DETAILED DESCRIPTION

[0064] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0065] The encrypted traffic cloud website identification technology framework designed by the present invention consists of three processing stages: data preprocessing stage, feature extraction stage and website identification stage. Figure 2It is the system processing flow chart of the present invention. First, the data preprocessing stage is mainly responsible for fixing the network traffic to form processable data objects. In addition, the network traffic needs to be divided into network flows in the form of five-tuples. The subsequent feature extraction module mainly extracts a set of traffic features for each flow. The feature extraction stage needs to complete the inference of the HTTP version used by the current traffic and the number of network resources carried from the encrypted traffic, and extract the above two features for each flow in the current traffic sample; then examine and use the HTTP version and resource quantity feature information of the entire sample (multiple flows) to combine into a unique feature representation, which we call a website fingerprint. Finally, in the website identification stage, the website fingerprints of multiple websites are actively collected in advance and constructed using the above method to form a fingerprint reference set; when using this system to identify a website, the above method is used again to construct the website fingerprint of the sample to be tested, and the distance between the fingerprint and each fingerprint in the fingerprint reference set is calculated. For closed world scenes, the label of the sample with the minimum distance is directly used as the label of the current sample to be tested; for open world scenes, the distance sequence calculated above needs to be recalculated with the Gini coefficient of the sequence. If the Gini coefficient is greater than a certain threshold, it is determined that it does not belong to the target website. If the Gini coefficient is less than this threshold, the label of the current sample to be tested is calibrated according to the label of the sample with the minimum distance. Below we will introduce each module of the detection framework of the present invention one by one according to the stage processing order.

[0066] 1. Data preprocessing stage

[0067] (1) Network traffic collection module

[0068] This module is used to collect traffic data from a vantage point on the network, typically by using the tcpdump tool to capture pcap files from the network card. This module is used for initial traffic material acquisition, both in the user preparation phase (actively accessing websites of interest and collecting traffic data) and in the implementation phase (listening and collecting data from the network).

[0069] (2) Network flow partitioning module

[0070] After obtaining the complete traffic of a website visit, this module divides the traffic into different flows in the form of the traffic quintuple <source IP, source port, destination IP, destination port and transport layer protocol>. Considering the correlation between network flows, the present invention regards the <source IP, source port> and <destination IP, destination port> here as having unity, that is, the bidirectional flow on a network connection is used as the final data processing basis.

[0071] (II) Feature extraction stage

[0072] The module of the present invention proposes a method for inferring HTTP version information and the number of network resources from encrypted traffic. The method mainly infers the protocol mechanism characteristics in the traffic, solving the problem in the prior art that the application layer characteristics cannot be directly extracted due to encryption.

[0073] (1) HTTP phase inference module

[0074] In encrypted traffic, the TLS protocol encrypts the content of the application layer and also performs encryption obfuscation during the handshake, which makes it impossible to accurately understand which data packets belong to the application layer HTTP information by directly identifying encrypted data packets. Therefore, the inference of the HTTP stage in encrypted traffic is the basis for the subsequent inference of the HTTP version and the number of resources. The present invention determines the beginning of the HTTP stage by observing the specific flags of the TLS protocol or the QUIC protocol in the encrypted traffic. For HTTP / 1.1 and HTTP / 2 traffic based on TLS encryption, it can be determined whether to enter the HTTP stage by whether the 'content type' field in the TLS data packet header sent by the client is 'application data'; for HTTP / 3 traffic based on QUIC encryption, the beginning of the HTTP stage is marked by switching from a long header to a short header.

[0075] (2) HTTP version inference module

[0076] The present invention is designed to infer HTTP version information through the direction of the initial data packet transmission sequence in the traffic. In the HTTP stage, HTTP / 3 can be identified by its unique QUIC protocol, while HTTP / 1.1 and HTTP / 2 can be distinguished by the transmission direction of the first two data packets. Specifically, the transmission sequence of HTTP / 1.1 is "out-in" (i.e., a data packet is sent first, and then a data packet is received), while the transmission sequence of HTTP / 2 is "out-out" (i.e., two data packets are sent continuously) due to the requirements of its connection preface. Based on this, the present invention can accurately infer the HTTP version in the encrypted traffic.

[0077] (3) Resource quantity inference module

[0078] In an encrypted traffic environment, inferring the number of network resources is a challenge. The present invention proposes a resource quantity inference method based on client traffic, which infers the number of network resources transmitted in the traffic by analyzing the number of data packets sent by the client in the HTTP stage. Since encryption masks the specific content, the number of resource requests and traffic burst patterns of HTTP will not change, so the number of resources can be estimated by the statistical characteristics of client traffic. In addition, in order to avoid the influence of application layer control frames on resource quantity estimation, the present invention introduces a protocol state machine model to limit the inference deviation under specific conditions, thereby ensuring the accuracy of resource inference and ensuring a standardized description of the method. The designed protocol state machine model execution structure and element annotations for different HTTP versions are as follows: Figure 3 , Figure 4 As shown. Among them, Figure 3 (a) is the protocol state machine model of HTTP / 1.1, (b) is the protocol state machine model of HTTP / 2, and (c) is the protocol state machine model of HTTP / 3.

[0079] The protocol state machine of the present invention tracks the state of encrypted traffic in detail, such as Figure 3 As shown in the figure, the state machine starts to process the data packets from the client in the idle state and Figure 4 The transition conditions defined in identify resource requests. When a packet meets these conditions and triggers the resource request state, the corresponding counter will be incremented. This method only focuses on the number of resources transmitted in the data flow and is not affected by the encryption shielding content information.

[0080] It is worth noting that the packets generated by the application layer control functions introduced by HTTP / 2 and HTTP / 3 are independent of the number of network resources. The present invention uses two main methods to prevent these control packets: (1) adding an initial step that does not inflate the counter to offset the effect of setting packets; (2) using packet size thresholds to filter out control packets during resource transmission (specifically packets less than 40 bytes in HTTP / 2 and less than 120 bytes in HTTP / 3).

[0081] (4) Fingerprint construction module

[0082] The fingerprint construction module of the present invention is based on encoding the HTTP version of each connection and the corresponding resource quantity to construct an HTTP fingerprint for identifying website features in network traffic. For each connection in the network traffic (i.e., network flow), the present invention encodes the HTTP version information and the corresponding resource quantity, and finally forms a fingerprint represented by a matrix (referred to as H123 fingerprint in the present invention).

[0083] Suppose there are x network connections, where the HTTP version of the i-th connection is denoted as HTTP i , the HTTP version belongs to the set {1,2,3}, which represent HTTP / 1.1, HTTP / 2 and HTTP / 3 respectively. The number of resources in the i-th connection is denoted as R i The fingerprint is obtained through a matrix To represent, each column of the matrix corresponds to a network connection, and each row represents HTTP / 1.1, HTTP / 2, and HTTP / 3 versions respectively.

[0084] The element F of the matrix F v,i The definition is as follows:

[0085]

[0086] Where v∈{1,2,3} represents the corresponding HTTP version. This matrix accurately encodes the HTTP version of each connection and the number of resources, thereby forming the fingerprint feature of the present invention. In this way, the fingerprint building module of the present invention can generate a matrix structure that can characterize the characteristics of network traffic, encoding its HTTP version information and the number of resources for each network connection. This method can not only effectively distinguish different websites, but also provide highly reliable network feature fingerprints under the condition of encrypted traffic, which is helpful for subsequent website identification or traffic analysis tasks.

[0087] (III) Website Identification Stage

[0088] (1) Lazy Learning

[0089] The present invention relates to a classification method based on lazy learning. Lazy learning is different from traditional eager learning models. Eager learning models such as decision trees or neural networks build models in the training phase, while lazy learning delays the training process to the prediction phase. Lazy learning directly compares new samples with stored reference samples and calculates similarity or distance indicators to make classification judgments.

[0090] In the application of website recognition, the advantage of lazy learning is that there is no need to pre-train the model, making the addition and update of website categories or samples more flexible. In addition, by calculating the distance in the prediction stage, the overfitting problem caused by insufficient training data is avoided.

[0091] In order to improve the efficiency of lazy learning in large-scale datasets, this paper introduces a sample pre-screening process based on the number of resources. In the prediction stage, a bandpass filter is applied to the reference sample set, and only samples with resource numbers within 20% of the target sample are retained. This optimization measure significantly reduces unnecessary distance calculations, achieving about 85% comparison reduction in experiments.

[0092] (2) Distance metrics: Wasserstein distance and LCSS

[0093] In order to accurately classify website fingerprints in encrypted network traffic, the present invention designs a combined distance metric method. Through the analysis of H123 fingerprints, three main types of changes in fingerprint data are identified: sequence arrangement changes, data segmentation / aggregation changes, and time dislocation. Although these changes are small, they are inherent characteristics of fingerprint data and must be handled in the classification method.

[0094] In response to these changes, the present invention adopts a combination of Wasserstein distance and longest common similar subsequence (LCSS) distance to cope with the variability unique to encrypted traffic.

[0095] The Wasserstein distance measures the difference between probability distributions by calculating the minimum "work" required to transform one distribution into another. This method is particularly suitable for dealing with small displacements of sequence elements or aggregation / segmentation patterns in fingerprint data. The distance is calculated as follows:

[0096]

[0097] Where P and Q are probability distributions, γ represents the transition plan, inf represents the infimum, and Γ(P,Q) represents the set of all possible transition plans from P to Q.

[0098] LCSS distance measures local similarity by identifying common subsequences between two sequences, and is suitable for dealing with time series misalignment in fingerprints. In order to enhance the robustness of the algorithm, this paper introduces a relaxed similarity threshold ∈\epsilon∈, allowing slight deviations between elements to cope with these displacements.

[0099] By appropriately weighting and combining Wasserstein distance and LCSS distance, the present invention provides an effective measure for website fingerprints, thereby improving the accuracy and robustness of classification.

[0100] (3) Identification strategy

[0101] The classification method proposed in this invention can be applied to closed-world and open-world scenarios:

[0102] Closed world scenario (users only visit known websites): When all categories are known, the present invention uses the classic Nearest Neighbor (NN) algorithm for classification. This method can achieve high-precision classification within the known category range.

[0103] Open world scenario (users may visit unknown websites): In an open world scenario where the category is unknown, the present invention combines the K-Nearest Neighbors (KNN) algorithm with the Gini coefficient to determine whether the test sample belongs to the reference category. Specifically, first calculate the distance between the current sample to be tested and all reference samples (the number of reference samples is M*N, that is, the number of all categories*the number of samples in each category), and then take out the smallest 2N sample distances from these results. Then, the distribution of these distances is measured based on the distance between the test sample and the nearest 2N reference samples and the Gini coefficient. If the test sample belongs to a known category, the nearest 2N samples will contain N samples from the same category, and the distances of these samples are small, while the other N samples are from other categories, and their distances are relatively large. This will lead to uneven distribution of distances, resulting in a lower Gini coefficient. On the contrary, if the test sample does not belong to any known category, these distances will be more evenly distributed, resulting in a higher Gini coefficient. By setting a threshold for the Gini coefficient, it is possible to reliably distinguish between samples that belong to the monitored set and samples that do not belong to the set.

[0104] In the following embodiments, the present invention uses python 3.11 to write various modules of the system, and the main libraries used include scapy, numpy, transitions, etc.

[0105] In the fingerprint construction module, the present invention finally determines that the length of the H123 fingerprint is 50, which means that the fingerprint part with a length exceeding 50 will be truncated, and the part with a length less than 50 will be filled with 0; the similarity threshold is set to 0.24 in the LCSS distance; and the importance of the wasserstein distance and the LCSS distance is set to be the same (both weights are 0.5). The present invention actively constructs a large-scale data set to evaluate the actual effect of the present invention.

[0106] Closed world scenario - small sample problem. This task examines the ability of website recognition methods when the training samples are limited. Therefore, the number of training samples (for each website) is set to N = {1, 2, 3, 10, 30} and the recognition accuracy is used to evaluate the effects of various methods. Figure 5 The recognition effect comparison between the method of the present invention and the existing advanced methods is demonstrated. It can be seen that the method has a significant improvement in accuracy compared with the existing methods in small sample scenarios, especially when only one training sample (N=1) is required and the number of monitored websites is 320 (rand.), the recognition accuracy has exceeded 90%.

[0107] Closed world scenario - different collections of monitored websites. We designed and collected sample data for a variety of website collections, including the pop. website collection covering the 1,600 most popular websites; the rand. website collection covering 320 websites randomly selected from the top 100,000 popular websites; and the cens. website collection covering 320 websites that are susceptible to network censorship. The recognition task results for these three collections of monitored websites can still be found in Figure 5 The results show that this method performs well in various monitoring website collections. The recognition accuracy on the pop. collection is slightly lower than that on the other two. This is because there are more websites in this collection, which increases the difficulty of the task itself.

[0108] Closed World Scene - Time Cost. Figure 5 The execution time of various methods is shown on the right. Here we take 3 training samples as an example. Existing website recognition methods often require a lot of model training, and the model needs to be retrained every time the model accuracy decreases or a new website is to be recognized. However, the present invention, based on the lazy learning solution paradigm, avoids the entire model training phase and provides higher potential for flexible deployment of website recognition. The machine learning method achieves the best execution efficiency in the prediction phase, and the time overhead of this method is within an acceptable range.

[0109] Closed World Scenes - Different Distance Metrics. Figure 5 The lower part shows the accuracy records of website recognition using different distance metrics to process H123 fingerprints. The experimental results show that the wasserstein distance and LCSS distance have achieved the best accuracy performance in small sample problems and different monitoring website sets, and the weighted sum of the two distances in this invention further improves the accuracy, proving that the two distances are complementary to each other.

[0110] Closed world scenario - cross-network environment. In order to verify the robustness of the present invention in the cross-network environment scenario, traffic data was collected in three different network environments (no restrictions; simulated Starlink scenario - characterized by a high packet loss rate; simulated low-speed network scenario - characterized by a high network latency), and their two-by-two combinations were used in the training and testing processes. Figure 6 The experimental results show that the present invention has the best cross-network environment recognition capability among all the methods. The machine learning / deep learning-based methods have a serious drop in accuracy when facing data drift. The IP-based method has the strongest stability, but the recognition accuracy is low and cannot meet the application requirements.

[0111] Closed world scenario - concept drift. In order to verify the robustness of the present invention in the data concept drift scenario, we set up three data sets with time spans (time span is 0 days; time span is 18 days; time span is 30 days), and use early data in the training process and relatively fresh data in the testing process to verify the impact of concept drift on various website recognition methods. Figure 7 The experimental results show that the present invention has the best recognition accuracy in the concept drift scenario among all the methods. For example, when only one training sample is used, even if the time interval between the training data and the test data reaches 30 days, the present method can still recognize them with an accuracy rate close to 80%.

[0112] Open world scenario. In the open world scenario, it is assumed that the number of open world websites exceeds 100,000. In this scenario, various methods need to determine not only whether the current sample belongs to a monitored website, but also which specific website it belongs to. Figure 8 As shown in the figure, the present invention achieves better results than existing methods in many cases. Although the IP-based website recognition method works well in the task of simply distinguishing whether a website belongs to the monitoring set, the fine-grained classification effect in actual tasks is unsatisfactory. The present invention can achieve a final recognition F1 value of 85.9% using only one training sample.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A cloud website identification method based on HTTP version inference, characterized in that: The method comprises: Divide network traffic into network flows; Extracting the traffic features of the network flow, and encoding the traffic features to obtain a feature matrix of the network flow; wherein the traffic features include: HTTP version information used by the network flow and the number of network resources carried; A website recognition result is obtained based on the feature matrix.

2. The method according to claim 1, characterized in that The dividing of network traffic into network flows includes: Collect network traffic from vantage points on the network; The traffic is divided into different network flows in the form of traffic quintuples, wherein the traffic quintuples include: source IP, source port, destination IP, destination port and transport layer protocol.

3. The method according to claim 1, characterized in that The extracting of the traffic characteristics of the network flow includes: Determine the start position of the HTTP phase through specific flags of the TLS protocol or QUIC protocol in the network flow; Infer the HTTP version information used by the network flow based on the initial data packet transmission sequence direction in the HTTP stage; By analyzing the number of packets sent by the client during the HTTP phase, the number of network resources transmitted in the network flow can be inferred.

4. The method according to claim 3, characterized in that Determining the starting position of the HTTP phase through a specific flag of the TLS protocol or the QUIC protocol in the network flow includes: Obtaining the protocol in the network flow; In the case where the protocol in the network flow is the TLS protocol, determining the starting position of the HTTP stage by determining whether the content type field in the TLS data packet header sent by the client is application data; When the protocol in the network stream is the QUIC protocol, the starting position of the HTTP stage is determined by the change from a long header to a short header.

5. The method according to claim 3, characterized in that: The inferring the HTTP version information used by the network flow according to the initial data packet transmission sequence direction in the HTTP stage includes: Determine whether the network flow is encrypted traffic based on the QUIC protocol; When the network flow is encrypted traffic based on the QUIC protocol, determining that the version information of the network flow is HTTP / 3; In the case where the network flow is encrypted traffic based on the QUIC protocol, the transmission direction of the first two data packets in the HTTP stage is obtained, and when the transmission direction of the first two data packets is out-in, the version information of the network flow is determined to be HTTP / 1.1, and when the transmission direction of the first two data packets is out-out, the version information of the network flow is determined to be HTTP / 2.

6. The method according to claim 3, characterized in that: The method of inferring the number of network resources transmitted in the network flow by analyzing the number of data packets sent by the client in the HTTP stage includes: According to the HTTP version information, build a protocol state machine model and define the transition conditions; The protocol state machine model processes data packets from the client in the idle state, and when the data packets from the client meet the transition conditions and trigger the resource request state, the corresponding counter will be incremented, thereby obtaining the number of network resources transmitted in the network flow; Wherein, when the HTTP version information is HTTP / 2 or HTTP / 3, the influence of control packets in the HTTP / 2 or HTTP / 3 version is eliminated by adding an initial step that does not expand the counter and using a packet size threshold.

7. The method according to claim 1, characterized in that The traffic features are encoded to obtain a feature matrix of the network traffic, including: Let the HTTP version of the i-th network flow be HTTP i ; The elements of the characteristic matrix defining the network traffic Among them, v∈{1,2,3}, 1 represents the number of resources R i The corresponding network flow is HTTP / 1.1 version, and 2 represents the number of resources R i The corresponding network flow is HTTP / 1.1 version HTTP / 2 version, 3 represents the number of resources R i The corresponding network flow is HTTP / 3 version.

8. The method according to claim 1, characterized in that When the application scenario is a closed world scenario, the website recognition result is obtained based on the feature matrix, including: Collect website fingerprints and build a reference sample set; Applying a bandpass filter to the reference sample set and retaining samples having a resource quantity within a set range with respect to the network traffic; Calculating the similarity between the retained sample and the feature matrix of the network traffic based on a combination of Wasserstein distance and longest common similar subsequence distance; A nearest neighbor algorithm is applied to the similarity between the retained sample and the feature matrix of the network traffic to obtain a website identification result; wherein the website identification result is a category of the website corresponding to the network traffic.

9. The method according to claim 1, characterized in that: When the application scenario is an open world scenario, the website recognition result is obtained based on the feature matrix, including: Collect website fingerprints and build a reference sample set; Applying a bandpass filter to the reference sample set and retaining samples having a resource quantity within a set range with respect to the network traffic; Calculating the similarity between the retained sample and the feature matrix of the network traffic based on a combination of Wasserstein distance and longest common similar subsequence distance; A combination of the K nearest neighbor algorithm and the Gini coefficient is applied to the similarity between the characteristic matrix of the retained sample and the network traffic to obtain a website identification result; wherein the website identification result is whether the website corresponding to the network traffic is a sample of the monitored set and a sample that does not belong to the monitored set.

10. A cloud website identification device based on HTTP version inference, characterized in that: The device comprises: A preprocessing module for dividing network traffic into network flows; A feature extraction module is used to extract the traffic features of the network flow and encode the traffic features to obtain a feature matrix of the network flow; wherein the traffic features include: HTTP version information used by the network flow and the number of network resources carried; The website identification module is used to obtain a website identification result based on the feature matrix.

Images