A website identification method and device on a cloud based on HTTP version inference

Abstract

The application discloses a website identification method and device based on HTTP version inference on a cloud, and belongs to the technical field of encrypted traffic analysis. The method comprises the following steps: dividing network traffic into network flows; extracting traffic features of the network flows, and encoding the traffic features to obtain a feature matrix of the network traffic; wherein the traffic features comprise HTTP version information used by the network flows and the number of network resources carried by the network flows; and obtaining a website identification result based on the feature matrix. The application can provide a robust, accurate and lightweight website identification technology solution for practical applications such as network supervision and tracking of network criminal activities.

Description

Technical Field

[0001] This invention relates to the field of encrypted traffic analysis technology, and in particular to a method and apparatus for identifying cloud-based websites based on HTTP version inference. Background Technology

[0002] With the widespread adoption of cloud computing, an increasing number of websites and services are being deployed in cloud environments. However, cloud environments have some unique characteristics that make identifying the websites a user visits from encrypted traffic particularly complex and challenging. First, the number of hosts in a cloud environment is enormous, with a single cloud server often hosting multiple websites or application services. Second, due to the limited availability of public IP addresses, many cloud hosts do not have independent public IP addresses but communicate through shared egress IPs. This means that multiple websites may share the same egress IP, increasing the difficulty of accurately identifying the specific website being accessed.

[0003] In this context, effectively identifying specific cloud websites visited by users from encrypted traffic has become a critical technical requirement. This is essential for network supervision and tracking cybercrime, especially given the increasing use of encrypted protocols (such as HTTPS) in cloud services, where traditional plaintext-based identification methods are no longer applicable. Therefore, developing technologies capable of accurately identifying websites visited by users in encrypted traffic environments can not only enhance network security and management capabilities but also effectively counter malicious attempts to use cloud platforms to conceal criminal activities.

[0004] Existing encrypted traffic website identification technologies are mainly divided into two categories: identification methods based on side-channel information and identification methods based on traffic characteristics and classification models.

[0005] First, identification methods based on side-channel information rely on metadata information exposed beyond HTTPS encryption for website identification. Although the HTTPS protocol encrypts the transmitted content, unlike the Tor anonymous network, some side-channel information can still be utilized in the modern internet architecture. For example, by reverse-engineering the resolved domain name through the IP address at the network layer, the website visited by the user can be inferred; at the security layer, the SNI (Server Name Indication) field in the TLS handshake directly indicates the target website; and plaintext DNS requests at the application layer also explicitly expose information about the website visited by the user.

[0006] Secondly, identification methods based on traffic characteristics and classification models primarily analyze the transmission characteristics of encrypted traffic, such as packet size, direction sequence, and inter-packet time intervals. This method typically involves actively collecting traffic samples from different websites, extracting their transport layer features, and then using machine learning or deep learning models for classification within a supervised learning framework. These methods are widely used in existing encrypted traffic classification work, and are particularly suitable for identifying encrypted HTTPS traffic.

[0007] However, with the evolution of network technology, existing technologies are facing more and more challenges (summarized as follows). Figure 1 First, identification methods based on side-channel information are gradually becoming ineffective. The Internet Engineering Task Force (IETF) is actively introducing a series of new technologies to address the anonymity issues of HTTPS. For example, encrypted DNS protocols (such as DoH, DoT, and DoQ) effectively solve the privacy problem of plaintext DNS; TLS ECH technology hides the true SNI information in traffic, further improving user privacy; in addition, with the widespread application of cloud environments and the popularization of CDNs, more and more websites share the same IP address, forming IP co-addressing, making reverse lookup methods based on IP addresses unreliable.

[0008] Meanwhile, traffic feature-based classification models also face numerous limitations in practical applications. These methods exhibit poor robustness and low processing efficiency in real-world production environments, impacting their usability and application prospects. Specifically, existing methods rely on transport layer traffic features, which are closely related to network transmission conditions and are therefore susceptible to variations in network environments. In real-world production environments, users cannot exhaustively account for all possible network conditions, leading to inconsistencies between training and testing data, thus reducing model robustness. Furthermore, existing methods typically require a large number of training samples to support model accuracy, and once model performance degrades, new data must be collected and the model retrained, which is not only time-consuming but also highly complex and costly in practical deployment and maintenance.

[0009] These issues indicate that existing encrypted traffic website identification technologies are gradually losing their effectiveness in the face of the ever-evolving network environment, especially the cloud environment, and there is an urgent need for more robust and efficient new methods. Summary of the Invention

[0010] This invention addresses the issue that existing encrypted traffic identification schemes are gradually becoming ineffective and lack robustness and usability in cloud-based website identification scenarios. It proposes a cloud-based website identification method and device based on HTTP version inference, providing a robust, accurate, and lightweight website identification technology solution for practical applications such as network supervision and tracking cybercrime.

[0011] To achieve the above objectives, the technical solution of the present invention includes the following:

[0012] A method for identifying cloud-based websites based on HTTP version inference, the method comprising:

[0013] Divide network traffic into network flows;

[0014] Extract the traffic features of the network flow and encode the traffic features to obtain the feature matrix of the network flow; wherein, the traffic features include: the HTTP version information used by the network flow and the number of network resources carried;

[0015] The website identification result is obtained based on the feature matrix.

[0016] Furthermore, the process of dividing network traffic into network flows includes:

[0017] Collect network traffic from advantageous locations on the network;

[0018] Traffic is divided into different network flows according to the form of a traffic quintuple, which includes: source IP, source port, destination IP, destination port, and transport layer protocol.

[0019] Furthermore, the extraction of network flow traffic characteristics includes:

[0020] The starting position of an HTTP phase can be determined by specific flags in the network stream for either the TLS or QUIC protocol.

[0021] The HTTP version information used by the network flow can be inferred from the initial packet transmission sequence direction of the HTTP phase;

[0022] By analyzing the number of data packets sent by the client within an HTTP phase, the amount of network resources transmitted in the network stream can be inferred.

[0023] Furthermore, determining the start position of the HTTP phase through specific flags of the TLS or QUIC protocol in the network stream includes:

[0024] Obtain the protocol in the network stream;

[0025] When the protocol in the network stream is TLS, the start position of the HTTP phase is determined by whether the content type field in the TLS packet header sent by the client is application data.

[0026] When the protocol in the network stream is the QUIC protocol, the start position of the HTTP phase is determined by the change from a long header to a short header.

[0027] Furthermore, the step of inferring the HTTP version information used by the network flow based on the initial packet transmission sequence direction of the HTTP phase includes:

[0028] Determine whether the network flow is encrypted traffic based on the QUIC protocol;

[0029] In the case that the network stream is encrypted based on the QUIC protocol, the version information of the network stream is determined to be HTTP / 3;

[0030] In the case that the network flow is encrypted traffic based on the QUIC protocol, the transmission direction of the first two data packets in the HTTP stage is obtained. When the transmission direction of the first two data packets is out-in, the version information of the network flow is determined to be HTTP / 1.1. When the transmission direction of the first two data packets is out-out, the version information of the network flow is determined to be HTTP / 2.

[0031] Furthermore, the step of inferring the amount of network resources transmitted in the network stream by analyzing the number of data packets sent by the client during the HTTP phase includes:

[0032] Based on the HTTP version information, construct a protocol state machine model and define the transition conditions;

[0033] The protocol state machine model processes data packets from the client in the idle state, and when a data packet from the client meets the transition conditions and triggers the resource request state, the corresponding counter will increment to obtain the number of network resources transmitted in the network stream.

[0034] In the case where the HTTP version information is HTTP / 2 or HTTP / 3, the influence of control packets in HTTP / 2 or HTTP / 3 versions is eliminated by adding an initial step that does not cause the counter to swell and using a packet size threshold.

[0035] Further, the traffic features are encoded to obtain the feature matrix of the network traffic, including:

[0036] Let the HTTP version of the i-th network stream be denoted as HTTP. i ;

[0037] Define the elements of the feature matrix of the network traffic Where v∈{1,2,3}, 1 represents the number of resources R. i The corresponding network stream is HTTP / 1.1, and 2 represents the number of resources R. i The corresponding network stream is HTTP / 1.1 or HTTP / 2, and 3 represents the number of resources R.i The corresponding network stream is HTTP / 3.

[0038] Furthermore, in the case of a closed-world application scenario, the website identification result is obtained based on the feature matrix, including:

[0039] Collect website fingerprints and construct a reference sample set;

[0040] A bandpass filter is applied to the reference sample set, and samples whose resource quantity is within a set range relative to the network traffic are retained;

[0041] The similarity between the retained samples and the feature matrix of the network traffic is calculated based on a combination of Wasserstein distance and longest common similar subsequence distance.

[0042] The nearest neighbor algorithm is applied to the similarity between the retained samples and the feature matrix of the network traffic to obtain the website identification result; wherein, the website identification result is the category of the website corresponding to the network traffic.

[0043] Furthermore, in the case of an open-world application scenario, the website identification result is obtained based on the feature matrix, including:

[0044] Collect website fingerprints and construct a reference sample set;

[0045] A bandpass filter is applied to the reference sample set, and samples whose resource quantity is within a set range relative to the network traffic are retained;

[0046] The similarity between the retained samples and the feature matrix of the network traffic is calculated based on a combination of Wasserstein distance and longest common similar subsequence distance.

[0047] The website identification result is obtained by applying a combination of the K-nearest neighbor algorithm and the Gini coefficient to the similarity between the retained samples and the feature matrix of the network traffic; wherein, the website identification result is whether the website corresponding to the network traffic is a sample of the monitored set and a sample that does not belong to the monitored set.

[0048] A cloud-based website identification device based on HTTP version inference, the device comprising:

[0049] The preprocessing module is used to divide network traffic into network flows;

[0050] The feature extraction module is used to extract traffic features of the network flow and encode the traffic features to obtain the feature matrix of the network flow; wherein, the traffic features include: the HTTP version information used by the network flow and the number of network resources carried;

[0051] The website identification module is used to obtain website identification results based on the feature matrix.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects.

[0053] 1) This invention is based on a lazy learning paradigm, thus avoiding the entire model training phase and providing greater potential for flexible deployment of website recognition.

[0054] 2) The design based on Wasserstein distance and LCSS distance in this invention has achieved the best accuracy performance in small sample problems and different sets of monitoring websites. Furthermore, this invention achieves a further improvement in accuracy by weighting and summing the two distances.

[0055] 3) Based on the feature design of HTTP version information and the quantity of network resources, this invention has achieved good recognition accuracy not only in conventional closed worlds, but also in closed worlds with concept drift scenarios and open worlds. Attached Figure Description

[0056] Figure 1 A summary of existing technical solutions for identifying cloud-based websites.

[0057] Figure 2 This is a flowchart of the system processing of the present invention.

[0058] Figure 3 This is a flowchart of the protocol state machine execution process.

[0059] Figure 4 Comment out the protocol state machine elements.

[0060] Figure 5 The website identification accuracy results.

[0061] Figure 6 To improve the accuracy of cross-network environment identification.

[0062] Figure 7 To improve the accuracy of concept drift environment identification.

[0063] Figure 8 The results are for identifying open-world scene websites. Detailed Implementation

[0064] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0065] The encrypted traffic cloud website identification technology framework designed in this invention consists of three processing stages: data preprocessing, feature extraction, and website identification. Figure 2This is a system processing flowchart of the present invention. First, the data preprocessing stage is mainly responsible for fixing network traffic into processable data objects. It also needs to divide the network traffic into network streams according to a five-tuple format. The subsequent feature extraction module mainly extracts a set of traffic features for each stream. The feature extraction stage needs to infer the HTTP version used by the current traffic and the number of network resources it carries from the encrypted traffic, extracting the above two features for each stream in the current traffic sample. Then, it examines and utilizes the HTTP version and resource quantity feature information of the entire sample (multiple streams) to combine them into a unique feature representation, which we call a website fingerprint. Finally, in the website identification stage, multiple website fingerprints are proactively collected in advance and constructed using the aforementioned method to form a fingerprint reference set. When using this system to identify websites, the aforementioned method is used again to construct the website fingerprint of the sample to be tested, and the distance between this fingerprint and each fingerprint in the fingerprint reference set is calculated. For closed-world scenarios, the label of the sample with the minimum distance is directly used as the label of the current sample to be tested. For open-world scenarios, the Gini coefficient of the distance sequence obtained above needs to be calculated again. If the Gini coefficient is greater than a certain threshold, it is determined that it does not belong to the target website. If the Gini coefficient is less than this threshold, the label of the current sample to be tested is assigned according to the label of the sample with the minimum distance. Below, we will introduce each module of the detection framework of this invention one by one according to the stage processing order.

[0066] (I) Data Preprocessing Stage

[0067] (1) Network traffic acquisition module

[0068] This module is used to collect traffic data from advantageous locations on the network, typically by using the tcpdump tool to capture pcap files from the network interface card. This module is used for initial traffic data acquisition, and is also used in the user preparation phase (actively visiting websites of interest and collecting traffic data) and the implementation phase (listening to and collecting data from the network).

[0069] (2) Network Flow Partitioning Module

[0070] After obtaining the complete traffic from a website visit, this module divides the traffic into different flows according to the five-tuple of traffic <source IP, source port, destination IP, destination port, and transport layer protocol>. Considering the correlation between network flows, this invention treats <source IP, source port> and <destination IP, destination port> as having uniformity, that is, using a bidirectional flow on a network connection as the basis for the final data processing.

[0071] (II) Feature Extraction Stage

[0072] This invention proposes a method for inferring HTTP version information and the quantity of network resources from encrypted traffic. This method primarily solves the problem in existing technologies where application-layer features cannot be directly extracted due to encryption by inferring protocol mechanism characteristics from the traffic.

[0073] (1) HTTP phase inference module

[0074] In encrypted traffic, the TLS protocol encrypts application-layer content and also performs encryption obfuscation during the handshake. This makes it impossible to accurately determine which packets belong to application-layer HTTP information simply by identifying the encrypted packets. Therefore, inferring the HTTP phase in encrypted traffic is fundamental to subsequent inferences about the HTTP version and resource quantity. This invention determines the start of an HTTP phase by observing specific markers of the TLS or QUIC protocol in encrypted traffic. For TLS-encrypted HTTP / 1.1 and HTTP / 2 traffic, the start of the HTTP phase can be determined by whether the 'content type' field in the TLS packet header sent by the client is 'application data'. For QUIC-encrypted HTTP / 3 traffic, the start of the HTTP phase is marked by switching from a long header to a short header.

[0075] (2) HTTP version inference module

[0076] This invention designs a method to infer HTTP version information by analyzing the transmission sequence direction of the initial data packets in the traffic. Within the HTTP phase, HTTP / 3 can be identified through its unique QUIC protocol, while HTTP / 1.1 and HTTP / 2 can be distinguished by the transmission direction of the first two data packets. Specifically, the transmission sequence of HTTP / 1.1 is "out-in" (i.e., sending one data packet followed by receiving one data packet), while HTTP / 2, due to its connection preface requirements, has a transmission sequence of "out-out" (i.e., sending two data packets consecutively). Based on this, this invention can accurately infer the HTTP version in encrypted traffic.

[0077] (3) Resource Quantity Inference Module

[0078] Inferring the quantity of network resources in encrypted traffic environments is challenging. This invention proposes a resource quantity inference method based on client traffic. By analyzing the number of data packets sent by the client during an HTTP phase, the amount of network resources transmitted in that traffic is inferred. Since encryption masks the specific content, but the number of HTTP resource requests and the burst pattern of traffic remain unchanged, the resource quantity can be estimated using the statistical characteristics of client traffic. Furthermore, to avoid the influence of application layer control frames on resource quantity estimation, this invention introduces a protocol state machine model to limit inference bias under specific conditions, ensuring the accuracy of resource inference and the standardized description of the method. The execution structure and element annotations of the designed protocol state machine model for different HTTP versions are as follows: Figure 3 , Figure 4 As shown. Among them, Figure 3 In the diagram, (a) is the protocol state machine model for HTTP / 1.1, (b) is the protocol state machine model for HTTP / 2, and (c) is the protocol state machine model for HTTP / 3.

[0079] The protocol state machine of this invention performs detailed tracking of the state of encrypted traffic, such as... Figure 3 As shown. The state machine begins processing data packets from the client in the idle state, and according to... Figure 4 The transformation conditions defined in the code identify resource requests. When a data packet meets these conditions and triggers the resource request state, the corresponding counter increments. This method focuses only on the number of resources transmitted in the data stream and is unaffected by encrypted or masked content information.

[0080] It is worth noting that the application-layer control functions introduced in HTTP / 2 and HTTP / 3 generate packets that are independent of the amount of network resources. This invention uses two main methods to block these control packets: (1) adding an initial step that does not inflate the counter to counteract the effect of setting packets; (2) using packet size thresholds to filter out control packets (especially packets smaller than 40 bytes in HTTP / 2 and packets smaller than 120 bytes in HTTP / 3) during resource transfer.

[0081] (4) Fingerprint construction module

[0082] The fingerprint construction module of this invention constructs an HTTP fingerprint based on encoding the HTTP version of each connection and the corresponding number of resources, which is used to identify website characteristics in network traffic. For each connection (i.e., network flow) in network traffic, this invention encodes the HTTP version information and the corresponding number of resources, and finally forms a matrix-represented fingerprint (referred to as H123 fingerprint in this invention).

[0083] There are x network connections, where the HTTP version of the i-th connection is denoted as HTTP. i The HTTP version belongs to the set {1,2,3}, representing HTTP / 1.1, HTTP / 2, and HTTP / 3 respectively. The number of resources for the i-th connection is denoted as R. i Fingerprints are obtained through a matrix. This indicates that each column of the matrix corresponds to a network connection, and each row represents HTTP / 1.1, HTTP / 2, and HTTP / 3 versions, respectively.

[0084] The elements F of matrix F v,i The definition is as follows:

[0085]

[0086] Where v∈{1,2,3} represents the corresponding HTTP version. This matrix accurately encodes the HTTP version and resource quantity of each connection, thus forming the fingerprint feature of this invention. In this way, the fingerprint construction module of this invention can generate a matrix structure that can characterize network traffic features, encoding the HTTP version information and resource quantity for each network connection. This method can not only effectively distinguish different websites, but also provide highly reliable network feature fingerprints under encrypted traffic conditions, which is helpful for subsequent website identification or traffic analysis tasks.

[0087] (III) Website Identification Phase

[0088] (1) Lazy Learning

[0089] This invention relates to a classification method based on lazy learning. Unlike traditional eager learning models, such as decision trees or neural networks, which build the model during the training phase, lazy learning delays the training process until the prediction phase. Lazy learning makes classification decisions by directly comparing new samples with stored reference samples and calculating similarity or distance metrics.

[0090] In website recognition applications, lazy learning offers the advantage of eliminating the need for pre-trained models, allowing for greater flexibility in adding and updating website categories or samples. Furthermore, calculating distances during the prediction phase avoids overfitting issues caused by insufficient training data.

[0091] To improve the efficiency of lazy learning on large-scale datasets, this invention introduces a sample pre-screening process based on resource quantity. During the prediction phase, a bandpass filter is applied to the reference sample set, retaining only samples whose resource quantity is within 20% of the target sample. This optimization significantly reduces unnecessary distance calculations, achieving approximately 85% comparison reduction in experiments.

[0092] (2) Distance metrics: Wasserstein distance and LCSS

[0093] To accurately classify website fingerprints in encrypted network traffic, this invention designs a combined distance metric method. Through analysis of H123 fingerprints, three main types of intra-category variations in fingerprint data are identified: sequence arrangement variations, data segmentation / aggregation variations, and temporal misalignments. Although these variations are subtle, they are inherent characteristics of fingerprint data and therefore must be addressed in the classification method.

[0094] To address these variations, this invention employs a combination of Wasserstein distance and Longest Common Similar Subsequence (LCSS) distance to handle the variability unique to encrypted traffic.

[0095] The Wasserstein distance measures the difference between probability distributions by calculating the minimum "work" required to transform one distribution into another. This method is particularly well-suited for handling small shifts or aggregation / segmentation patterns in sequence elements of fingerprint data. The formula for calculating this distance is as follows:

[0096]

[0097] Where P and Q are probability distributions, γ represents the transfer plan, inf represents the infimum, and Γ(P,Q) represents the set of all possible transfer plans from P to Q.

[0098] LCSS distance measures local similarity by identifying common subsequences between two sequences, making it suitable for handling time-series misalignments in fingerprints. To enhance the robustness of the algorithm, this invention introduces a relaxed similarity threshold ∈ ε, allowing for slight deviations between elements to accommodate these shifts.

[0099] By appropriately weighting and combining Wasserstein distance and LCSS distance, this invention provides an effective metric for website fingerprinting, improving the accuracy and robustness of classification.

[0100] (3) Identification Strategy

[0101] The classification method proposed in this invention can be applied to both closed-world and open-world scenarios:

[0102] Closed-world scenario (users only visit known websites): When all categories are known, this invention employs the classic Nearest Neighbor (NN) algorithm for classification. This method can achieve high-precision classification results within the known category range.

[0103] Open-world scenario (users may visit unknown websites): In open-world scenarios where the categories are unknown, this invention combines the K-Nearest Neighbors (KNN) algorithm with the Gini coefficient to determine whether a test sample belongs to a reference category. Specifically, first, the distance between the current test sample and all reference samples (a total of M*N, which is the total number of categories multiplied by the number of samples in each category) is calculated. Then, the smallest 2N sample distances are selected. Next, the distribution of these distances is measured based on the distances between the test sample and the nearest 2N reference samples, combined with the Gini coefficient. If the test sample belongs to a known category, the nearest 2N samples will include N samples from the same category with smaller distances, while the other N samples will be from other categories with relatively larger distances. This results in an uneven distribution of distances, leading to a lower Gini coefficient. Conversely, if the test sample does not belong to any known category, these distances will be more evenly distributed, resulting in a higher Gini coefficient. By setting a threshold for the Gini coefficient, samples belonging to the monitored set and samples not belonging to the set can be reliably distinguished.

[0104] In the following embodiments, the system modules of this invention were written using Python 3.11, and the main libraries used include scapy, numpy, transitions, etc.

[0105] In the fingerprint construction module, this invention ultimately determines the length of the H123 fingerprint to be 50, meaning that fingerprints longer than 50 will be truncated, and those shorter than 50 will be padded with 0s. A similarity threshold of 0.24 is set in the LCSS distance; furthermore, the importance of Wasserstein distance and LCSS distance is set to be the same (both with a weight of 0.5). This invention actively constructs a large-scale dataset to evaluate its practical effectiveness.

[0106] Closed-world scenario – limited sample size problem. This task examines the performance of website recognition methods with a limited number of training samples. Therefore, the number of training samples (for each website) is set to N = {1, 2, 3, 10, 30}, and the recognition accuracy is used to evaluate the performance of various methods. Figure 5 The comparison of the recognition performance of the method of this invention with existing advanced methods is shown. It can be seen that the method of this invention has a significant improvement in accuracy compared with existing methods in small sample scenarios, especially when only one training sample (N=1) is needed and the number of monitored websites is 320 (rand.), the recognition accuracy has exceeded 90%.

[0107] Closed-world scenario—different surveillance website sets. Sample data for various website sets were designed and collected, including the pop. website set covering the 1600 most popular websites; the rand. website set covering 320 randomly selected websites from the top 100,000 most popular websites; and the cens. website set covering 320 websites vulnerable to internet censorship. The results of the identification task for these three surveillance website sets are still within [the relevant timeframe]. Figure 5 The results show that this method performs well across various monitoring website sets. However, its accuracy on the pop. set is slightly lower than the other two, which is because the pop. set contains more websites, increasing the processing difficulty of the task.

[0108] Closed-world scenarios – time consumption. Figure 5 The right side shows the execution time of various methods; here we take a method using 3 training samples as an example. Existing website recognition methods often require extensive model training, and the model needs to be retrained whenever the model accuracy drops or when new websites need to be identified. This invention, based on a lazy learning paradigm, avoids the entire model training phase, providing greater potential for flexible deployment of website recognition. In the prediction phase, the machine learning method achieves optimal execution efficiency, and the time cost of this method is within an acceptable range.

[0109] Closed-world scenarios – different distance metrics. Figure 5 The second half of the paper presents the accuracy records for website identification using different distance metrics to process H123 fingerprints. Experimental results show that Wasserstein distance and LCSS distance achieved the best accuracy performance in small sample problems and different sets of monitored websites. Furthermore, this invention achieves a further improvement in accuracy by weighted summing of the two distances, demonstrating that these two distances are complementary.

[0110] Closed-world scenario – cross-network environment. To verify the robustness of this invention in cross-network environment scenarios, traffic data was collected in three different network environments (unrestricted; simulated Starlink scenario – characterized by high packet loss rate; simulated low-speed network scenario – characterized by high network latency), and pairwise combinations of these environments were used during both the training and testing processes. Figure 6 Experimental results show that this invention has the best cross-network environment recognition capability among all methods. Methods based on machine learning / deep learning suffer a severe drop in accuracy when faced with data drift. The IP-based method has the strongest stability, but its recognition accuracy is low and cannot meet application requirements.

[0111] Closed-world scenario – concept drift. To verify the robustness of this invention in the data concept drift scenario, we set up three datasets with time spans (0 days; 18 days; 30 days), and used earlier data during training and fresher data during testing to verify the impact of concept drift on various website recognition methods. Figure 7 Experimental results show that the present invention has the best recognition accuracy in concept drift scenarios among all methods. For example, even when using only one training sample, the method can still recognize them with an accuracy of nearly 80% even when the time interval between training data and test data reaches 30 days.

[0112] Open-world scenario. In this scenario, it's assumed there are over 100,000 open-world websites. In this case, various methods not only need to determine if the current sample belongs to a monitored website, but also which specific website it belongs to. Experimental results are as follows... Figure 8 As shown, this invention achieves superior performance compared to existing methods in various situations. While IP-based website identification methods perform well in tasks involving simple distinctions of whether a website belongs to a monitoring set, their performance in fine-grained classification for real-world tasks is less than satisfactory. This invention, however, achieves a final identification F1 score of 85.9% using only one training sample.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for identifying cloud-based websites based on HTTP version inference, characterized in that, The method includes: Divide encrypted traffic into network flows; Extract the traffic features of the network flow and encode the traffic features to obtain the feature matrix of the encrypted traffic; wherein, the traffic features include: the HTTP version information used by the network flow and the number of network resources carried; The website identification result is obtained based on the feature matrix; The extraction of network flow traffic features includes: The starting position of an HTTP phase can be determined by specific flags in the network stream for either the TLS or QUIC protocol. The HTTP version information used by the network flow can be inferred from the initial packet transmission sequence direction of the HTTP phase; By analyzing the number of data packets sent by the client during an HTTP phase, the amount of network resources transmitted in the network stream can be inferred. The step of inferring the HTTP version information used by the network flow based on the initial packet transmission sequence direction of the HTTP phase includes: Determine whether the network flow is encrypted traffic based on the QUIC protocol; In the case that the network stream is encrypted based on the QUIC protocol, the version information of the network stream is determined to be HTTP / 3; When the network flow is encrypted traffic based on the TLS protocol, the transmission direction of the first two data packets in the HTTP phase is obtained. If the transmission direction of the first two data packets is out-in, the version information of the network flow is determined to be HTTP / 1.1; if the transmission direction of the first two data packets is out-out, the version information of the network flow is determined to be HTTP / 2. The step of inferring the amount of network resources transmitted in the network flow by analyzing the number of data packets sent by the client within the HTTP phase includes: Based on the HTTP version information, construct a protocol state machine model and define the transition conditions; The protocol state machine model processes data packets from the client in the idle state, and when a data packet from the client meets the transition conditions and triggers the resource request state, the corresponding counter will increment to obtain the number of network resources transmitted in the network stream. In the case where the HTTP version information is HTTP / 2 or HTTP / 3, the influence of control packets in HTTP / 2 or HTTP / 3 versions is eliminated by adding an initial step that does not cause the counter to swell and using a packet size threshold.

2. The method according to claim 1, characterized in that, The process of dividing encrypted traffic into network flows includes: Capture encrypted traffic from advantageous locations on the network; Traffic is divided into different network flows according to the form of a traffic quintuple, which includes: source IP, source port, destination IP, destination port, and transport layer protocol.

3. The method according to claim 1, characterized in that, Determining the start position of an HTTP phase by using specific flags from the TLS or QUIC protocol in the network stream includes: Obtain the protocol in the network stream; When the protocol in the network stream is TLS, the start position of the HTTP phase is determined by whether the content type field in the TLS packet header sent by the client is application data. When the protocol in the network stream is the QUIC protocol, the start position of the HTTP phase is determined by the change from a long header to a short header.

4. The method according to claim 1, characterized in that, Encoding the traffic features to obtain the feature matrix of the encrypted traffic includes: Let the HTTP version of the i-th network stream be denoted as HTTP. i ; Define the elements of the feature matrix of the encrypted traffic. Where v∈{1,2,3}, 1 represents the number of resources R. i The corresponding network stream is HTTP / 1.1, and 2 represents the number of resources R. i The corresponding network stream is HTTP / 2 version, and 3 indicates the number of resources R. i The corresponding network stream is HTTP / 3.

5. The method according to claim 1, characterized in that, In the case of a closed-world application scenario, the website identification result is obtained based on the feature matrix, including: Collect website fingerprints and construct a reference sample set; A bandpass filter is applied to the reference sample set, and samples whose resource quantity is within a set range as described in the encrypted traffic are retained; The similarity between the retained samples and the feature matrix of the encrypted traffic is calculated based on a combination of Wasserstein distance and longest common similar subsequence distance. The nearest neighbor algorithm is applied to the similarity between the feature matrix of the retained sample and the encrypted traffic to obtain the website identification result; wherein, the website identification result is the category of the website corresponding to the encrypted traffic.

6. The method according to claim 1, characterized in that, In an open-world application scenario, the website identification result is obtained based on the feature matrix, including: Collect website fingerprints and construct a reference sample set; A bandpass filter is applied to the reference sample set, and samples whose resource quantity is within a set range as described in the encrypted traffic are retained; The similarity between the retained samples and the feature matrix of the encrypted traffic is calculated based on a combination of Wasserstein distance and longest common similar subsequence distance. The website identification result is obtained by applying a combination of the K-nearest neighbor algorithm and the Gini coefficient to the similarity between the feature matrix of the retained sample and the encrypted traffic; wherein, the website identification result is whether the website corresponding to the encrypted traffic is a sample of the monitored set and a sample that does not belong to the monitored set.

7. A cloud-based website identification device based on HTTP version inference, characterized in that, The device includes: The preprocessing module is used to divide encrypted traffic into network streams; The feature extraction module is used to extract traffic features of the network flow and encode the traffic features to obtain the feature matrix of the encrypted traffic; wherein, the traffic features include: the HTTP version information used by the network flow and the number of network resources carried; The website identification module is used to obtain website identification results based on the feature matrix; The extraction of network flow traffic features includes: The starting position of an HTTP phase can be determined by specific flags in the network stream for either the TLS or QUIC protocol. The HTTP version information used by the network flow can be inferred from the initial packet transmission sequence direction of the HTTP phase; By analyzing the number of data packets sent by the client during an HTTP phase, the amount of network resources transmitted in the network stream can be inferred. The step of inferring the HTTP version information used by the network flow based on the initial packet transmission sequence direction of the HTTP phase includes: Determine whether the network flow is encrypted traffic based on the QUIC protocol; In the case that the network stream is encrypted based on the QUIC protocol, the version information of the network stream is determined to be HTTP / 3; When the network flow is encrypted traffic based on the TLS protocol, the transmission direction of the first two data packets in the HTTP phase is obtained. If the transmission direction of the first two data packets is out-to-in, the version information of the network flow is determined to be HTTP / 1.1; if the transmission direction of the first two data packets is out-to-out, the version information of the network flow is determined to be HTTP / 2. The step of inferring the amount of network resources transmitted in the network flow by analyzing the number of data packets sent by the client within the HTTP phase includes: Based on the HTTP version information, construct a protocol state machine model and define the transition conditions; The protocol state machine model processes data packets from the client in the idle state, and when a data packet from the client meets the transition conditions and triggers the resource request state, the corresponding counter will increment to obtain the number of network resources transmitted in the network stream. In the case where the HTTP version information is HTTP / 2 or HTTP / 3, the influence of control packets in HTTP / 2 or HTTP / 3 versions is eliminated by adding an initial step that does not cause the counter to swell and using a packet size threshold.

Images