Tor core site discovery method based on hidden service association

Abstract

The application discloses a Tor core site discovery method based on hidden service association, which comprises the following steps: (1) a hidden service association algorithm: for the web sites with similar contents but different domain names, a hidden service association algorithm based on page structure and content is designed; (2) calculation of hidden service survival rate; (3) hidden service access volume measurement; (4) Tor core site discovery: the hidden services in each group clustered in the step (1) are analyzed by using the hidden service survival rate and the access volume obtained in the steps (2) and (3), and the core sites are identified. The application can realize the discovery of the core sites with higher analysis value in the Tor dark net.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of anonymity network, and particularly relates to a Tor core site discovery method based on hidden service association. BACKGROUND

[0002] Due to the strong anonymity of Tor, many criminals conduct illegal transactions and engage in illegal activities in it, such as gun trafficking, drug trafficking, privacy information trading and the like, in addition, some organizations also use it to carry out large-scale network attacks. In order to effectively regulate the dark web content, it is necessary to efficiently crawl the Tor dark web content. However, the importance of different hidden services in the dark web is different, and the amount of effective information is also greatly different, if the whole network crawler is used, a lot of valuable information cannot be obtained, resulting in low data quality. In addition, a large number of domain names in the dark web correspond to the same site content, that is, the web contents of different domain names are basically the same, which will cause a large amount of crawler analysis, storage and computing resources to be consumed in repeated site contents, seriously restricting the detection and mastery of the dark web space, therefore, it is necessary to discover the core site of the Tor dark web. SUMMARY

[0003] The application aims at the problem that the data quality of the dark web is low at present, and proposes a Tor core site discovery method based on hidden service association, which firstly associates the hidden services with different domain names but the same or similar contents, and then identifies the core site through the survival rate and access volume of the hidden service obtained by long-term detection.

[0004] The application adopts the following technical scheme:

[0005] A Tor core site discovery method based on hidden service association, the method comprises the following steps:

[0006] (1) Hidden service association algorithm: for the web sites with similar contents but different domain names, a hidden service association algorithm based on page structure and content is designed;

[0007] (2) Calculation of hidden service survival rate: whether the descriptor of the hidden service exists is used to indirectly determine whether the hidden service is online, and it is used as one of the characteristics of the core site judgment;

[0008] (3) Hidden service access volume measurement: the request condition of the hidden service blind public key is collected through the deployment of the hidden service directory server HSDir, and then the access volume of the hidden service is calculated by analysis and comparison;

[0009] (4) Tor core site discovery: the hidden service survival rate and access volume obtained by (2) and (3) are used to analyze the hidden services in each group of the clusters in (1), and the core sites are identified.

[0010] Further, the step (1) specifically comprises:

[0011] (11) Clustering by using the redirection link in the Response Header: since some domain names will return a 301 status code and automatically redirect to other pages after access, the Location field in the Response Header header will display the domain name of the redirected page, and therefore the domain name and the redirected domain name are clustered into a group;

[0012] (12) Clustering the same sites with meaningful titles: the title of the default page of a site in the dark web is defined as meaningless, including "Index of / ", "Apache2 Debian Default Page", "401 Authorization Required", Apache, Nginx, and the sites with meaningless titles and without title information are each classified into a group, while the sites with meaningful title information and the same title text are classified into a group;

[0013] (13) Clustering by combining HTML DOM tree, CSS style, and page keywords: a page is extracted from the group with meaningful titles, the DOM tree structure, class attribute value, id attribute value, and the first 20 keywords in the page are calculated, and the DOM tree structure similarity, class attribute value and id attribute value similarity, and page keyword similarity of each page are compared by using a similarity algorithm.

[0014] Further, the step (2) specifically comprises:

[0015] (21) Reading the domain name to be calculated for the hidden service survival rate from the database;

[0016] (22) Deploying multiple Tor processes, and sending a query request to the hidden server by the client through the Tor control protocol to realize concurrent execution of multiple processes;

[0017] (23) If the descriptor is in a non-exceptional state, it is judged whether the descriptor exists according to the returned information, and the result is saved: if the descriptor exists, the domain name is considered to be online; if it does not exist, the domain name is considered to be offline;

[0018] (24) If the descriptor query is abnormal and the number of queries does not exceed 5, the domain name is put back into the queue, and will be re-queried later, returning to step (22);

[0019] (25) According to the return information, save the detection result of calculating the survival rate of the hidden service.

[0020] Further, the step (3) specifically comprises:

[0021] (31) For each v3 domain name, calculate all blind public keys within a certain period;

[0022] (32) Compare the blind public key results calculated offline with the blind public key data collected from the hidden service directory server to obtain the total access of each v3 domain name;

[0023] (33) Divide the total access of each v3 domain name by the statistical days to calculate the daily average access of the hidden service v3 domain name.

[0024] Further, the step (4) specifically comprises:

[0025] (41) For each group clustered in (1), calculate the survival rate sr of each group j_i , the value of which is the maximum survival rate of all domain names in the group; the survival rate sr j_i is represented by the following formula:

[0026] Wherein, online_num is the online

[0027] (42) For each group clustered in (1), calculate the access view of each group j_i : For the website with declared mirror sites, view j_i is the sum of the access of all domain names in each group; and for the website without declared mirror sites, view j_i is the maximum value of the access of all domain names in each group;

[0028] (43) Model the discovery problem of the core site as a binary classification problem in machine learning, use the access, survival rate, number of similar pages and in-out degree as classification attributes, and use the XGBoost model to discover the core site;

[0029] (44) For the pages classified as core sites, calculate the classification discriminant probability x at the same time, based on the discriminant probability, further divide the identified core sites into three levels of importance, wherein the pages with x≥0.9 are regarded as the most important core sites, the pages with 0.75≤x<0.9 are regarded as the less important pages, and the pages with 0.5≤x<0.75 are regarded as the least important core sites.

[0030] Advantages: Compared with the prior art, the present application has the following advantages:

[0031] 1. Hidden service probe efficiency is improved: from the Tor client sending a request to the hidden service receiving, the whole process needs to pass through 15 hop onion routers, and using the hidden service probe method of the application, only 3 hop onion routers are needed, which significantly improves the hidden service probe efficiency.

[0032] 2. The traditional deployment hidden service directory server collects access volume scheme is based on Torv2 protocol, but the method can obtain the access volume of Torv3 hidden service through analysis and comparison by stripping Tor source code offline calculation v3 domain name blind public key.

[0033] 3. The existing Tor hidden service importance ranking does not consider the Tor protocol characteristics, and the core site in the application is combined with the Tor protocol characteristics, including hidden service survival rate and access volume, which can more effectively realize the hidden service core site discovery. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a schematic diagram of the clustering analysis algorithm of the application.

[0035] Figure 2 is a hidden service probe flowchart of the application.

[0036] Figure 3 is a system deployment diagram of the hidden service probe and access volume measurement of the application.

[0037] Figure 4 is a model training flowchart of the core site discovery of the application. DETAILED DESCRIPTION

[0038] The application designs and implements a Tor core site discovery technology based on hidden service association to discover the core site in the dark web. The hidden service association, hidden service probe, hidden service access volume measurement and core site discovery scheme are respectively included, which specifically includes:

[0039] 1. Hidden service association

[0040] The hidden service association algorithm includes three steps of clustering using the redirection link in the Response Header, clustering meaningful titles, and combining HTML DOM tree, CSS style, and page keywords.

[0041] Clustering using the redirection link in the Response Header: the Location field in the Response Header header will display the domain name of the redirected page, so in step one, the domain name and the redirected domain name are clustered into a group.

[0042] Clustering meaningful titles: The present invention considers that the titles of the default pages of Web servers (such as Apache, Nginx, etc.), such as "Index of / ", "Apache2 Debian Default Page", "401 Authorization Required", etc., are meaningless. Based on step one (the present invention considers that the group titles successfully clustered in step one are the titles of the redirected domain names), the sites with meaningless titles and no title information are grouped together, while the sites with meaningful title information and the same title text are grouped together.

[0043] Combining HTML DOM tree, CSS style, page keywords and other content clustering: This invention will extract a page from the group of meaningful titles, calculate the DOM tree structure, class attribute value, id attribute value and the first 20 keyword information of each page, and use the similarity algorithm to compare the DOM tree structure similarity, class attribute value and id attribute value similarity and page keyword similarity of each page. The overall process is as follows Figure 1 Specifically, we use the sequence comparison method to calculate the similarity between the DOM trees of each two pages, denoted as similarity1. We also use the Jaccard similarity coefficient to calculate the similarity between the class and id attribute values ​​of each two page documents, denoted as similarity2. We also use this coefficient to calculate the similarity between the keyword information of each two pages, denoted as similarity3. These three similarities are combined to determine whether two pages should be grouped.

[0044] 2. Hidden service detection solution

[0045] This scheme indirectly determines whether a hidden service is online by looking at the presence of its descriptor. By analyzing the Tor protocol, we found that before communicating with a hidden service, the client needs to query the hidden service directory server for the hidden service descriptor. During the query, the hidden service directory server's response can be summarized into three situations:

[0046] (1) Query successful: the descriptor exists and the query is returned successfully;

[0047] (2) Query failed: the descriptor does not exist;

[0048] (3) Query exception: The descriptor information is not returned for some reasons, including query timeout, hidden service directory server rejecting the request, etc.

[0049] Each hidden service sends its descriptor to the hidden service directory server periodically (no more than two hours), and the hidden service directory server also periodically cleans up expired descriptors, so whether a hidden service is online can be indirectly determined by whether the descriptor of the hidden service exists.

[0050] The whole live detection process is shown in Figure 2 The specific steps are as follows:

[0051] (1) Read the domain name to be tested from the database;

[0052] (2) Deploy multiple Tor processes, and the client sends a query request to the hidden server through the Tor control protocol to realize multi-process concurrent execution;

[0053] (3) If the descriptor is in a non-exceptional state, it will be determined whether the descriptor exists according to the returned information and the result will be saved: if the descriptor exists, it is considered that the domain name is online; if it does not exist, it is considered that the domain name is not online;

[0054] (4) If the descriptor query is abnormal and the number of queries does not exceed 5, the domain name will be put back into the queue and re-queried later, returning to step (2);

[0055] (5) Save the hidden service online detection result according to the returned information.

[0056] For each hidden service, the survival rate is sr, which can be expressed as follows:

[0057] Where online_num is the number of domain names measured online.

[0058] 3. Hidden service access volume measurement scheme

[0059] The hidden service domain name in the Tor network needs to be queried by HSDir when accessed, so the Tor source code can be modified accordingly to record and count the client access request. This is the overall idea of the hidden service access volume measurement method proposed in the present application.

[0060] When the client sends the descriptor Id value corresponding to the domain name to the selected HSDir, it will trigger the call of the cache_lookup_v3_as_dir function in the. / src / feature / hs / hs_cache.c file in the Tor source code to find out whether the descriptor Id value exists in the cache. If it is found, 1 is returned, otherwise 0 is returned. Therefore, the code in this function can be modified to record the client access request. However, HSDir cannot directly obtain the hidden service domain name, and when obtaining the access volume, only the blind public key can be seen, and the blind public key can be calculated offline. The specific process of measurement is as follows:

[0061] (1) For each domain name, calculate all blind public keys within a certain period;

[0062] (2) Compare the blind public key results calculated offline with the blind public key data collected from HSDir to obtain the total access volume of each v3 domain name;

[0063] (3) Divide the total access volume of each domain name by the number of statistical days to calculate the daily average access volume of the hidden service domain name.

[0064] Figure 3 The overall deployment scheme for hidden service probing and access volume detection is shown in the following table.

[0065] 4. Core site discovery scheme

[0066] The core site discovery scheme will combine the survival rate and access volume characteristics obtained in 2 and 3 for calculation. The overall algorithm flow is as follows:

[0067] (1) Calculate the survival rate and access volume of each group of hidden services: Let the survival rate of each group of domain names be sr j_i , which is the maximum survival rate of all domain names in the group; Let the access volume of each group be view j_i . For websites with declared mirror sites, view j_i is the sum of the access volumes of all domain names in each group; and for websites without declared mirror sites, view j_i is the maximum value of the access volumes of all domain names in each group.

[0068] (2) Data preprocessing: Perform data normalization on the access volume of each group of hidden services. Let the normalized access volume of each group of hidden services be view' j_i , then

[0069]

[0070] (3) Classification model training to obtain core sites: Model the core site discovery problem as a binary classification problem in machine learning, and use the preprocessed data as classification attributes. Use the XGBoost model to perform core site discovery. The overall flow of model training is shown in Figure 4 .

[0071] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application and do not limit the scope of protection. Although the present application has been described in detail with reference to the above examples, those skilled in the art should understand that the skilled person can make various changes, modifications or equivalent replacements to the specific embodiments of the application after reading the present application. However, these changes, modifications or equivalent replacements are within the scope of protection of the claims.

Claims

1. A Tor core site discovery method based on hidden service association, characterized in that: The method comprises the following steps: (1) Hidden service association algorithm: For websites with similar content but different domain names, a hidden service association algorithm based on page structure and content is designed; (11) Clustering using redirect links in Response Header: (12) Group similar sites with meaningful titles into one group: (13) Clustering based on HTML DOM tree, CSS style, and page keywords: (2) Calculate the hidden service survival rate: indirectly determine whether the hidden service is online by whether the hidden service descriptor exists, and use it as one of the features for core site judgment; (21) Read the domain name of the hidden service survival rate to be calculated from the database; (22) Deploy multiple Tor processes, and the client sends query requests to the hidden server through the Tor control protocol to achieve multi-process concurrent execution; (23) If the descriptor is in a non-abnormal state, the descriptor will be judged based on the returned information and the result will be saved: if the descriptor exists, the domain name is considered to be online; If it does not exist, the domain name is considered offline; (24) If the descriptor query is abnormal and the number of queries does not exceed 5, put the domain name back into the queue and requery it later, and return to step (22); (25) Based on the returned information, save the detection results of calculating the survival rate of the hidden service; (3) Hidden service access volume measurement: By deploying the hidden service directory server HSDir, we collect the requests for the blind public key of the hidden service, and then analyze and compare the hidden service access volume; (31) For each v3 domain name, calculate all blind public keys within a certain period; (32) Compare the offline calculated blind public key result with the blind public key data collected from the hidden service directory server to obtain the total number of visits for each v3 domain name; (33) Divide the total number of visits to each v3 domain name by the number of statistical days to calculate the average daily number of visits to the hidden service v3 domain name; (4) Tor core site discovery: Analyze the hidden services in each group clustered in (1) based on the hidden service survival rate and visit volume obtained in (2) and (3) to identify the core sites; (41) For each group clustered in (1), calculate the survival rate of each group , whose value is the maximum survival rate of all domain names in the group; survival rate The following formula expresses it: ； Among them, online_num is the total number of times the domain name is measured online, and check_num is the total number of times the domain name is measured; (42) For each group clustered in (1), calculate the number of visits for each group :For websites that have declared mirror sites, The sum of all domain name visits for each group; for websites that do not declare mirror sites, The maximum value of all domain name visits for each group; (43) The core site discovery problem is modeled as a binary classification problem in machine learning, with the number of visits, survival rate, number of similar pages, and in-and-out degree as classification attributes, and the XGBoost model is used for core site discovery; (44) For pages classified as core sites, the classification probability x is calculated at the same time. Based on the probability, the identified core sites are further divided into three levels of importance, among which pages with x ≥ 0.9 will be regarded as the most important core sites, pages with 0.75 ≤ x < 0.9 will be regarded as the second most important pages, and pages with 0.5 ≤ x < 0.75 will be regarded as the least important core sites.

2. A Tor core site discovery method based on hidden service association according to claim 1, characterized in that: The step (11) is specifically as follows: Since some domain names will return a 301 status code and automatically redirect to other pages after being accessed, the Location field in the Response Header will display the domain name of the redirected page. Therefore, the domain name and the redirected domain name are clustered into one group; The step (12) is specifically as follows: The titles of the default pages of sites on the dark web are defined as meaningless, including "Index of / ", "Apache2Debian Default Page", "401 Authorization Required", Apache, and Nginx. Sites with these meaningless titles and no title information are grouped separately, while sites with meaningful title information and the same title text are grouped together. The step (13) is specifically as follows: Clustering is performed by combining HTML DOM tree, CSS style, and page keywords: a page will be extracted from the group of meaningful titles, and the DOM tree structure, class attribute value, id attribute value, and the top 20 keyword information of each page will be calculated. The similarity of the DOM tree structure, class attribute value, id attribute value, and page keyword similarity of each page will be compared using a similarity algorithm.

Images