A fine-grained webpage recognition method for complex network environments

Abstract

This invention relates to a fine-grained webpage identification method for complex network environments. First, webpage access traffic is segmented into five-tuples, and the segmented traffic is grouped using the SNI information contained in the data packets. Then, the data packets in specific groups are reassembled into TLS fragments, and their length information is extracted. During multiple accesses to the same webpage, ADUs representing webpage characteristics are repeatedly requested. By statistically analyzing the frequent items in the TLS fragment length information, feature vectors characterizing the webpage are constructed; these feature vectors are considered webpage fingerprints. This invention utilizes clustering to correct numerical fluctuations in feature vectors and designs a bag-of-words model to correct sequential fluctuations in feature vectors. Finally, the corrected feature vectors are input into a machine learning model for training, resulting in a classifier capable of accurately identifying webpages. This method can obtain stable fine-grained webpage fingerprints in complex network environments, enabling accurate identification of harmful webpage access behavior.

Description

Technical Field

[0001] This invention relates to a fine-grained webpage recognition method for complex network environments, belonging to the field of network security technology. Background technology:

[0002] Web pages are a crucial medium for information dissemination on the internet. Inevitably, harmful content exists within this vast amount of information, negatively impacting social stability. With the development of network communication technology, encryption protocols are widely used in end-to-end communication, making the tracking of harmful content more difficult. According to a statistical report from Google, as of September 2022, approximately 95% of web traffic through the Chrome browser was encrypted. Therefore, it is necessary to propose a method for efficiently identifying harmful web pages in encrypted environments.

[0003] Although the content transmitted on web pages is encrypted, different web pages are composed of different elements, and these elements exhibit different traffic behaviors during transmission. Therefore, by analyzing the traffic characteristics of web page visits, it is possible to obtain web page fingerprints that can accurately correlate with the content of the web pages.

[0004] Existing research has proposed several methods for constructing webpage fingerprints, which can be broadly categorized into coarse-grained and fine-grained webpage fingerprints based on their accuracy. Early webpage fingerprinting primarily focused on constructing fingerprints for the homepages of different websites, representing coarse-grained fingerprinting. With further research, researchers have begun to focus on finer-grained fingerprinting, capable of constructing fingerprints for different webpages within the same website. Fine-grained webpage fingerprints can more accurately reconstruct user browsing behavior, making them more suitable for applications such as identifying nuisance websites. Different webpages within the same website often have similar layouts, making them more difficult to distinguish based on traffic patterns, which places higher demands on the selection and extraction of webpage features.

[0005] However, when faced with more complex network environments, existing fine-grained webpage fingerprinting methods still fall short of regulatory requirements in terms of accuracy. This is because most existing methods construct webpage fingerprints using statistical features based on data packets. However, when the network environment changes, the size and order of data packets fluctuate to some extent. This fluctuation makes machine learning or deep learning classifiers trained based on such features insufficient in accuracy in real-world applications, failing to achieve fine-grained webpage identification. Summary of the Invention:

[0006] To address the issue of insufficient accuracy in webpage fingerprint construction due to network fluctuations, this invention proposes a fine-grained webpage identification method for complex network environments. The method first segments webpage access traffic into five-tuples and then groups the segmented traffic using the SNI information contained in the data packets. Next, the data packets in specific groups are reassembled into TLS fragments, and the length information of the TLS fragments is extracted to complete the data representation of the original traffic. During multiple accesses to the same webpage, ADUs representing webpage features are repeatedly requested. This invention constructs feature vectors representing webpages by statistically analyzing frequent terms in the TLS fragment length information; these feature vectors are considered webpage fingerprints. To eliminate the impact of complex network environments on the stability of webpage fingerprints, this invention uses clustering to correct numerical fluctuations in feature vectors and designs a bag-of-words model to correct sequential fluctuations in feature vectors. Finally, the corrected feature vectors are input into a machine learning model for training, resulting in a classifier capable of accurately identifying webpages. Compared with existing webpage identification methods, this method can obtain stable fine-grained webpage fingerprints in complex network environments and can be used for accurate identification of harmful webpage access behavior.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: a fine-grained webpage recognition method for complex network environments, the method comprising the following steps:

[0008] Step (1) Collect flow data and switch flows according to the five-tuple;

[0009] Step (2) analyzes the SNI information contained in the data packet and groups the traffic from step (1);

[0010] Step (3) reassembles the data packets from step (2) into TLS fragments and extracts the TLS fragment length information within a specific packet;

[0011] Step (4) Based on the results of step (3), cluster TLS fragment lengths with similar kernel density using kernel density prediction, and use the clustering results to eliminate the impact of TLS fragment length fluctuations on feature extraction.

[0012] Step (5) Based on the results of step (4), count the frequent items of TLS fragment length during webpage access to construct a feature vector representing the webpage.

[0013] Step (6) Based on the results of step (5), design a bag-of-words model to process feature vectors and eliminate the impact of TLS fragmentation order fluctuations;

[0014] Step (7) uses the results of step (6) to input the feature vector into the machine learning model for training, and obtains a model that can accurately identify web pages.

[0015] Furthermore, in step (2), the specific steps for grouping the traffic are as follows: parse the SNI information contained in the data packets of a single flow to obtain a five-tuple flow with an SNI tag. Flows with the same SNI tag are grouped together.

[0016] Furthermore, in step (3), the specific steps for extracting grouping information are as follows:

[0017] (3.1) Based on the characteristics of the website, select the packets with specific SNI tags in step (2) and reassemble them into TLS fragments according to the header information of the data packets in the packets;

[0018] (3.2) Extract the length information of the TLS fragments in (3.1) and construct the TLS fragment length sequence.

[0019] Furthermore, in step (4), the specific steps for eliminating the impact of TLS fragment length fluctuations on feature extraction using clustering results are as follows:

[0020] (4.1) Extract all TLS fragment length sequences belonging to the same SNI in step (3). Remove duplicate elements from all lengths in these sequences and denote them as set F. Sort the elements in F in ascending order, and denote the sorted result as sequence P. The i-th element in P is denoted as x. i ;

[0021] (4.2) The density of each value in sequence P is calculated using kernel density prediction, and the corresponding probability density function is constructed. The density calculation formula used is shown in formula (1):

[0022]

[0023] Where y is the independent variable, h is the window width of the kernel function, and x i Let be the i-th sample falling into the window, and N be the total number of samples falling into the window. K(x;h) is a positive kernel function. The kernel function used in this method is the Gaussian kernel function, which is defined as shown in formula (2):

[0024]

[0025] Where e is the natural constant, x is the independent variable, and h is the window width of the kernel function.

[0026] (4.3) Find the minimum point of the probability density function in (4.2), use the minimum point as the split point of sequence P, and divide the original TLS fragment length into several groups;

[0027] (4.4) Modify each data in the group in (4.3) to the average value of all data in that group.

[0028] Furthermore, in step (5), the specific steps for constructing the webpage fingerprint are as follows:

[0029] (5.1) Count the frequent items of TLS fragment length in a single connection under the same SNI. If the frequency of a certain TLS fragment length in a single connection exceeds the threshold, denote its length as k, and merge the m consecutive fragments of length k in the TLS sequence (m>1) with the next fragment;

[0030] (5.2) Count the most frequent items of TLS fragment length in multiple access behaviors of the same webpage under the same SNI. If the frequency of a certain TLS fragment length in multiple access behaviors of the same webpage exceeds the threshold, add the length to the set W;

[0031] (5.3) Count the most frequent items of TLS fragment length in multiple access behaviors of the same website under the same SNI. If the frequency of a certain TLS fragment length in multiple access behaviors of the same website exceeds the threshold, add the length to the set S;

[0032] (5.4) Analyze and extract webpage features. Select all TLS fragment lengths that are in set W but not in set S, and construct feature vectors that represent the webpages. These feature vectors are regarded as webpage fingerprints.

[0033] Furthermore, in step (6), the steps for designing the bag-of-words model to process features are as follows: First, all values ​​in all feature vectors generated in step (5) are deduplicated and used as the vocabulary of the bag-of-words model; then, the original feature vectors are processed into word vectors of equal length generated based on the vocabulary.

[0034] Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects.

[0035] (1) The fine-grained webpage identification method proposed in this invention constructs a fine-grained webpage fingerprint by statistically analyzing frequent terms of TLS fragment length. The webpage fingerprint constructed by the method can effectively characterize the differences between different webpages, and the trained classifier can accurately identify harmful webpages.

[0036] (2) The fine-grained webpage recognition method proposed in this invention utilizes clustering to correct numerical fluctuations in feature vectors and designs a bag-of-words model to correct sequential fluctuations in feature vectors. This method can ensure the stability of constructing webpage fingerprints in complex network environments, making the trained classifier more adaptable to complex and ever-changing real-world application scenarios compared to existing methods.

[0037] (3) The fine-grained webpage identification method proposed in this invention can be used in the field of identifying and monitoring public nuisance webpages, providing strong protection against the further spread of public nuisance information. Attached Figure Description

[0038] Figure 1 Overall architecture diagram of a fine-grained webpage recognition method for complex network environments;

[0039] Figure 2 Flowchart for clustering to eliminate numerical fluctuations;

[0040] Figure 3 A flowchart illustrating how the bag-of-words model eliminates order fluctuations. Detailed Implementation

[0041] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0042] Specific Implementation: The fine-grained webpage recognition method for complex network environments provided by this invention can achieve fine-grained recognition of monitored webpages in real-world scenarios where the network environment changes. Its process architecture is shown in the attached figure. Figure 1 As shown.

[0043] The implementation methods for each step will be described below:

[0044] Step (1) Collect flow data and switch flows according to the five-tuple;

[0045] In the first embodiment of the present invention, the user is browsing a monitored webpage, and the resulting traffic data is saved locally. Because the collected traffic data may contain a large amount of background traffic, it is necessary to use a 5-tuple (source IP, source port, destination IP, destination port, transport protocol) to cut the original data stream for subsequent analysis.

[0046] Step (2) analyzes the SNI information contained in the data packet and groups the traffic from step (1);

[0047] In one embodiment of the present invention, the specific steps for grouping traffic based on SNI information are as follows: parsing the header information of data packets in a single flow, finding data packets containing clienthello information, and parsing the SNI suite within them. This suite contains detailed SNI information, and the five-tuple flows are labeled according to the obtained SNI tags. Flows with the same SNI tag are grouped together; Table 1 lists the meanings and specific values ​​of the fields contained in the SNI suite.

[0048] Table 1: SNI Suite Field Information

[0049]

[0050]

[0051] Step (3) reassembles the data packets from step (2) into TLS fragments and extracts the TLS fragment length information within a specific packet;

[0052] In one embodiment of the present invention, the specific steps for reassembling TLS fragments and extracting TLS fragment length information are as follows:

[0053] (3.1) Based on the characteristics of the website, select the packets with ServerName field of item.jd.com, img14.360buyimg.com, and img11.360buyimg.com in step (2), and reassemble them into TLS fragments according to the header information of the data packets in the packets;

[0054] (3.2) Extract the length information of the TLS fragments from (3.1) and construct the TLS fragment length sequence. The TLS fragment length sequence in this embodiment is shown in Table 2.

[0055] Table 2 TLS Fragment Length Sequence

[0056]

[0057] Step (4) Based on the results of step (3), cluster TLS fragment lengths with similar kernel density using kernel density prediction, and use the clustering results to eliminate the impact of TLS fragment length fluctuations on feature extraction.

[0058] In one embodiment of the present invention, a flowchart illustrating how clustering results can eliminate the impact of TLS fragment length fluctuations on feature extraction is attached. Figure 2 As shown, the specific steps are as follows:

[0059] (4.1) Extract all TLS fragment length sequences belonging to the same SNI in step (3). Remove duplicate elements from all lengths in these sequences and denote them as set F. Sort the elements in F in ascending order, and denote the sorted result as sequence P. The i-th element in P is denoted as x. i The sorted sequence P contains the following TLS fragments: 79, 80, 81, 430, 431, 432, 700, 1349, 1350

[0060] (4.2) The density of each value in sequence P is calculated using kernel density prediction, and the corresponding probability density function is constructed. The density calculation formula used is shown in formula (1):

[0061]

[0062] Where y is the independent variable, h is the window width of the kernel function, and x iLet be the i-th sample falling into the window, and N be the total number of samples falling into the window. K(x;h) is a positive kernel function. The kernel function used in this method is the Gaussian kernel function, which is defined as shown in formula (2):

[0063]

[0064] Where e is the natural constant, x is the independent variable, and h is the window width of the kernel function.

[0065] (4.3) Find the minimum points of the probability density function in (4.2). Based on the data in (4.1), the calculated minimum points are 261, 572, and 1014. These three minimum points are used as the splitting points of sequence P, dividing the original TLS fragment length into four groups;

[0066] (4.4) Modify each data point within a group in (4.3) to the average of all data points in that group. The corrected sequence P is 80, 80, 80, 431, 431, 431, 700, 1349, 1349

[0067] Step (5) Based on the results of step (4), count the frequent items of TLS fragment length during webpage access to construct a feature vector representing the webpage.

[0068] In one embodiment of the present invention, the specific steps for constructing a webpage fingerprint are as follows:

[0069] (5.1) Analyze the most frequent TLS fragment lengths in a single connection under the same SNI. According to the statistics, under the SNI xxxxx.xxxxx.xxx, TLS fragments with a length of 16383 appear more than 70% of the time in a single connection. Merge consecutive TLS fragments with a length of 16383 in the TLS sequence with the next fragment.

[0070] (5.2) Count the most frequent items of TLS fragment length in multiple accesses of the same webpage under the same SNI. Under the SNI xxxxx.xxxxx.xxx, the TLS fragment with length 79, -148, -248, 16383, 9140, 3082 appears more than 80% of the time in multiple accesses of the same webpage. Add this length to the set W;

[0071] (5.3) Count the most frequent items of TLS fragment length in multiple access behaviors of the same website under the same SNI. Under the SNI xxxxx.xxxxx.xxx, the lengths are 79, -148, -248, 16383. The frequency of the TLS fragment length in multiple access behaviors of the same website exceeds 70%. Add this length to set S;

[0072] (5.4) Analyze and extract webpage features. Select all TLS fragment lengths that are in set W but not in set S to construct feature vectors representing webpages. Therefore, combining the results of (5.2) and (5.3), 9140,3082 is selected as the feature vector of the webpage, and these are regarded as webpage fingerprints.

[0073] Step (6) Based on the results of step (5), design a bag-of-words model to process feature vectors and eliminate the impact of TLS fragmentation order fluctuations;

[0074] In one embodiment of the present invention, a flowchart for designing the bag-of-words model to process features is attached. Figure 3 As shown, the specific steps are as follows: First, all values ​​in all feature vectors generated in step (5) are deduplicated and used as the vocabulary of the bag-of-words model; then, the original feature vectors are processed into word vectors of equal length generated according to the vocabulary.

[0075] Step (7) uses the results of step (6) to input the feature vector into the machine learning model for training, and obtains a model that can accurately identify web pages.

[0076] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A fine-grained web page identification method for complex network environment, characterized in that, The method comprises the following steps: Step (1) collects traffic data, and cuts the flow according to a five-tuple; Step (2) analyzes SNI information contained in a data packet, and groups the traffic in step (1); Step (3) re-groups the data packet in step (2) into TLS fragments, and extracts TLS fragment length information in a group with a specific SNI label; Step (4) based on the result of step (3), uses kernel density prediction to cluster similar TLS fragment lengths, and uses the clustering result to eliminate the influence of TLS fragment length fluctuations on feature extraction; Step (5) based on the result of step (4), counts the frequent items of TLS fragment lengths in web page access processes to construct a feature vector representing a web page; Step (6) based on the result of step (5), designs a bag-of-words model to process the feature vector, and eliminates the influence of TLS fragment sequence fluctuations; Step (7) uses the result of step (6), inputs the corrected feature vector into a machine learning model for training, and obtains a classifier capable of accurately identifying web pages; In step (4), the following sub-steps are included: (4.1) extract all TLS fragment length sequences belonging to the same SNI in step (3), remove repeated elements from all lengths in these sequences, and record the set as F. Arrange the elements in F in ascending order, and record the arrangement result as sequence P; (4.2) use kernel density prediction to calculate the density of each value in sequence P, and construct the corresponding probability density function. The density calculation formula used is shown in formula (1): where y is the independent variable, h is the window width of the kernel function, x i is the ith sample falling in the window, N is the total number of samples falling in the window, K(x; h) is a constant positive kernel function, and the kernel function selected by the present method is a Gaussian kernel function, which is defined as shown in equation (2): Where e is a natural constant, x is the independent variable, and h is the window width of the kernel function, (4.3) find the minimum point of the probability density function in (4.2), and use the minimum point as the segmentation point of sequence P to divide the original TLS fragment length into several groups; (4.4) modify each data in the group in (4.3) to the average value of all data in the group; In step (5), the following sub-steps are included: (5.1) count the frequent items of TLS fragment lengths in a single connection under the same SNI. If the frequency of a certain TLS fragment length in a single connection exceeds a threshold, record the length as k, and combine m consecutive TLS fragments with a length of k in the TLS sequence with the subsequent one, where m>1; (5.2) count the frequent items of TLS fragment lengths in multiple access behaviors of the same web page under the same SNI. If the frequency of a certain TLS fragment length in multiple access behaviors of the same web page exceeds a threshold, add the length to set W; (5.3) count the frequent items of TLS fragment lengths in multiple access behaviors of the same website under the same SNI. If the frequency of a certain TLS fragment length in multiple access behaviors of the same website exceeds a threshold, add the length to set S; (5.4) analyze and extract web page features, select all TLS fragment lengths in set W and not in set S, and construct a feature vector representing a web page. These feature vectors are regarded as web page fingerprints. ​ 2. The fine-grained web page identification method for complex network environment according to claim 1, characterized in that, The specific implementation steps of the step (2) are: analyzing the SNI information contained in the data in a single flow to obtain a five-tuple flow with an SNI label, and grouping the flows with the same SNI label into the same group.

3. The method of claim 1, wherein the method is a fine-grained web page recognition method for a complex network environment. The step (3) comprises the following sub-steps: (3.1) according to the website characteristics, selecting the group with a specific SNI label in the step (2), and recombining the data packets in the group into TLS fragments according to the header information of the data packets; (3.2) extracting the length information of the TLS fragments in the step (3.1) to construct a TLS fragment length sequence.

4. The fine-grained web page identification method for complex network environment according to claim 1, characterized in that, In the step (6), the specific process of designing the bag-of-words model to process the features is: firstly, removing the values in all the feature vectors generated in the step (5) to obtain a word list of the bag-of-words model; and then processing the original feature vectors into word vectors of the same length generated according to the word list.

Images