A method for analyzing and determining dark web access population from network traffic

Abstract

The present invention discloses a method for analyzing and identifying dark web access groups from network traffic. The method extracts dark web access traffic characteristics from network traffic data and categorizes the identification results according to the effectiveness of the characteristics, thereby identifying dark web access groups with different risk levels. The present invention can access massive amounts of network traffic and restore the traffic into network behavior logs. The restored DNS protocol data and network node access data are matched with dark web characteristics to preliminarily determine the dark web access groups and their corresponding risk levels. The method then further associates the groups with the mailing behavior data to improve the effectiveness of identification and ultimately output a tiered list of dark web access groups.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and analysis, and relates to the analysis of mobile network and fixed network traffic data, and specifically to a method for analyzing and determining the population of dark web visitors from network traffic. Background Art

[0002] Amidst increasingly complex network security threats, many malicious activities (such as worms, spam, and vulnerabilities) are hidden within the application layer payload of data packets. Traditional security protection technologies, relying solely on network and transport layer security detection techniques, are no longer able to meet network security requirements. DPI (Deep Packet Inspection) is a computer network packet filtering technology. It is a security mechanism that detects and controls network traffic passing through devices based on application layer information. It can detect and control network application layer information, ensure the security of data content, and improve network security.

[0003] The "dark web" refers to a hidden network that ordinary netizens cannot access through conventional means. Login requires specialized software, configuration, or authorization. Due to its anonymity, the dark web is a breeding ground for various crimes that exploit the internet. Currently, a common method for monitoring dark web users involves using machine learning to identify TOR traffic and subsequently analyze it. However, TOR traffic identified by machine learning can be misidentified, requiring significant time to verify its accuracy, hindering the efficiency of dark web user analysis. Summary of the Invention

[0004] In order to solve the above problems, the present invention discloses a method for analyzing and determining the dark web access population from network traffic, which can parse the dark web access and suspected delivery behavior from the network traffic, thereby identifying the corresponding dark web access behavior. The present invention extracts the traffic characteristics of dark web access from the network traffic data, and classifies the identification results according to the effectiveness of the characteristics, thereby identifying the dark web access population with different risk levels. Based on the traffic characteristics, the present invention identifies from several aspects, including the (IP, port) of the dark web relay, the (IP, port) of the dark web bridge, the DNS traffic of the dark web bridge database, the DNS traffic of the dark web feature domain name, and the traffic of the APP, specifically including:

[0005] Identify dark web access behavior through dark web relay (IP, port):

[0006] Visitors must first access the dark web through a three-step relay: an entry relay, an intermediate relay, and an exit relay. Therefore, accessing a dark web relay node (IP, port) can be considered a dark web access. Whether the access is successful can be determined by whether there is downstream traffic. Because dark web relays may be blocked by the state, traffic accessing dark web relays often only has upstream traffic, with no downstream traffic. People who engage in this behavior are at high risk for dark web access.

[0007] Identify dark web access behavior through the dark web bridge (IP, port):

[0008] Because accessing the dark web through relays often fails, a new method has emerged: using bridges. These allow users to bypass the firewall and directly access the dark web's entry relay, ultimately entering the dark web. Therefore, accessing a dark web bridge (IP address, port number) can be considered a dark web access. Whether the access is successful can be determined by whether there is downlink traffic. While bridges are more discreet than relays, they also carry the risk of being blocked by the government. Those who engage in this behavior are considered high-risk for accessing the dark web.

[0009] Identify dark web access behavior through DNS traffic in the dark web bridge database:

[0010] The Dark Web Bridge Database stores the resources of Dark Web Bridges. Those seeking to access the Dark Web can obtain the Dark Web Bridge's (IP address, port number) from the public network's Bridge Database, allowing them to access the Dark Web through the Bridge. Because the Bridge Database has been blocked domestically, it can only be accessed through a VPN. Regular traffic lacks traffic related to the Bridge Database domain name, and only a small amount of data related to the Bridge Database domain name can be captured in DNS protocol traffic. This data is often generated by a visitor triggering a program to access the Bridge Database without opening a VPN. Therefore, those who generate DNS traffic related to the Bridge Database domain name can be considered to have already taken the prelude to accessing the Dark Web and are considered a medium-risk group for Dark Web access.

[0011] Identify dark web access behavior through DNS traffic of dark web characteristic domain names:

[0012] Dark web website domain names have a uniform suffix signature. Furthermore, these domain names can only be accessed after entering the dark web through a relay; they cannot be successfully accessed directly from the surface web. Some visitors who are interested in accessing the dark web but lack technical expertise may find the dark web website's domain name online and attempt to access it directly from the surface web. This attempt will not succeed, but will generate DNS traffic, which can be identified by domain name signatures in this DNS traffic. Since these individuals have not successfully accessed the dark web and lack technical expertise, they are considered potential dark web users and pose a low risk of access.

[0013] Identify delivery behavior after placing an order on the dark web through the traffic of express delivery apps:

[0014] After identifying dark web access behavior through the above method, we can further monitor the delivery application traffic corresponding to people with dark web access behavior in the network traffic and mark it as the delivery behavior after placing an order on the dark web.

[0015] To achieve the above object, the technical solution of the present invention is as follows:

[0016] A method for analyzing and determining dark web access users from network traffic, comprising the following steps:

[0017] Step 1: Crawl the IP and port lists of dark web relays and bridges as the main features of dark web access behavior;

[0018] Step 2: Obtain network traffic and convert it into network behavior logs;

[0019] Step 3: Identify the behavior of accessing dark web relays and bridges from the network behavior log restored in step 2. When the network behavior log identifies the IP address or port in the dark web relay and bridge IP and port list obtained in step 1, it is considered that the behavior of accessing dark web relays and bridges has been identified. When the behavior has downstream traffic, the network account corresponding to the access behavior is given a high weight; when the behavior has no downstream traffic, the network account corresponding to the access behavior is given a medium weight.

[0020] Step 4: Identify attempts to access the bridge database using the domain name of the bridge database from the DNS data in the network behavior log restored in Step 2. If the DNS data contains the domain name of the bridge database, it is considered an attempted access behavior and the corresponding network account is given a medium weight.

[0021] In step 5, the domain names in the log data identified as dark web relay and bridge access behaviors in step 3 are extracted to form a domain name dataset. The domain name dataset is vectorized using the Word2Id model. The K-means clustering algorithm is used to calculate the converged K cluster centers. The domain names corresponding to the cluster centers are used as the characteristic domain names for accessing the dark web.

[0022] Step 6: After vectorizing the domain names in the daily network behavior logs, calculate the vector distance with the K cluster centers in step 5. Domain names with distances less than the set threshold are selected as candidate dark web access features and added to the characteristic domain names for dark web access obtained in step 5.

[0023] Step 7: Match the domain name of the DNS data in the network behavior log restored in step 2 with the characteristic domain name of accessing the dark web output in step 5 or step 6 to identify the behavior of attempting to access the dark web through the surface network. When the domain name of the DNS data matches the characteristic domain name of accessing the dark web, it is judged as an attempt to access the dark web website, and the network account corresponding to the access behavior is given a low weight;

[0024] Step 8: Group the dark web access behaviors captured in steps 3 to 5 by network account, and calculate the integral corresponding to each network account using the following formula:

[0025] integral=Σnumber of access behaviors*behavior weight

[0026] Step 9: Output the list of network accounts that access the dark web and the corresponding points to achieve the classification of dark web visitors.

[0027] Furthermore, in step 1, the IP is changed each time the IP and port list of the dark web relay and bridge is crawled.

[0028] Furthermore, in step 2, DPI technology is used to restore network traffic into traffic logs.

[0029] Furthermore, the network account is a mobile phone number, IMSI, IMEI or ADSL account.

[0030] Furthermore, in step 6, the domain name is manually analyzed and confirmed before being added to the characteristic domain name for accessing the dark web.

[0031] The beneficial effects of the present invention are:

[0032] The method of the present invention can access massive network traffic and restore the traffic into network behavior logs. It matches the restored DNS protocol data and network node access data with dark web features to preliminarily determine the people who access the dark web and the corresponding risk levels. It then combines the delivery behavior data to further associate with the people, improve the effectiveness of identification, and finally output a graded list of dark web access people. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flow chart of the method provided by the present invention for analyzing network traffic to determine the population of dark web visitors. DETAILED DESCRIPTION

[0034] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0035] The present invention provides a method for analyzing and determining dark web access population from network traffic, comprising the following steps:

[0036] Step 1: Use a web crawler to crawl the (IP, port) list of dark web relays and bridges. This is the main feature of dark web access behavior. Since the relay and bridge information crawled by the same IP within seven days is the same, the crawler needs to change the IP every time it crawls in order to obtain new relay and bridge information. The crawling action will continue to change the IP to improve the richness and completeness of the information.

[0037] Step 2: Obtain network traffic and convert it into network behavior logs. The present invention uses DPI technology to convert network traffic into traffic logs. Network traffic is obtained from the operator.

[0038] Step 3, identify the behavior of accessing dark web relays and bridges (IP, port) from the network behavior log restored in step 2 (when the IP or port in the (IP, port) list of dark web relays and bridges obtained in step 1 is identified in the network traffic data, it is considered that the behavior of accessing dark web relays and bridges is identified), and give different weights according to whether there is downstream traffic in the behavior. The presence of downstream traffic indicates that the captured behavior has successfully accessed the dark web and has a higher access risk. The corresponding network account (mobile phone number, IMSI, IMEI or ADSL account) is given a high weight. The absence of downstream traffic indicates that the captured behavior did not successfully access the dark web, but there may be successful access in the uncaptured behavior, and the corresponding network account is given a medium weight.

[0039] Step 4: Identify the behavior of attempting to access the bridge database through the domain name of the bridge database from the DNS data of the network traffic restored in step 2 (when the DNS data contains the bridge database domain name, it is judged as an attempted access behavior). Since the bridge database is generally accessed through VPN, the captured data is only a small part of the actual access behavior. The corresponding network account is likely to have uncaptured successful access to the dark web, so the corresponding network account is given a medium weight.

[0040] In step 5, the domain names in the log data of the dark web relay and bridge access behaviors determined in step 3 are extracted to form a domain name dataset. The domain name dataset is vectorized using the Word2Id model, and a suitable K value is selected. The K-means clustering algorithm is used to calculate the converged K cluster centers, and the domain names corresponding to the cluster centers are used as the characteristic domain names for accessing the dark web.

[0041] In step 6, domain names in daily network activity logs are vectorized and their vector distances are calculated with the K cluster centers from step 5. Domain names with distances below a set threshold are selected as candidate dark web access features. After manual analysis and confirmation, they are added to the dark web access feature domains output in step 5, continuously increasing the number of domain name features. Manual analysis ensures the accuracy of added domain names. Furthermore, if multiple domain names fall below the threshold, manual analysis can be used to select appropriate domain names for inclusion in the dark web access feature domains. The threshold should be adjusted based on the vector distance calculation results to select an appropriate number of domain names, avoiding difficulty selecting domain names or excessive results with a low percentage of valid domain names.

[0042] In step 7, the domain name of the DNS data in the network behavior log restored in step 2 is matched with the characteristic domain name of the dark web access output in step 5 or step 6 to identify the behavior of attempting to access the dark web through the surface network. When the domain name of the DNS data matches the characteristic domain name of the dark web access, it is judged as an attempt to access the dark web website, and the network account corresponding to the access behavior is given a low weight.

[0043] Step 8: Calculate the dark web access behaviors captured in steps 3 to 5, group them by network account, and calculate the integral corresponding to the network account as follows:

[0044] integral=Σnumber of access behaviors*behavior weight

[0045] Step 9: Output the list of network accounts that access the dark web and the corresponding points to achieve the classification of dark web access population.

[0046] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A method for analyzing and determining dark web access population from network traffic, characterized in that: The steps include: Step 1: Crawl the IP and port lists of dark web relays and bridges as the main features of dark web access behavior; Step 2: Obtain network traffic and convert it into network behavior logs; Step 3: Identify the behavior of accessing dark web relays and bridges from the network behavior log restored in step 2. When the network behavior log identifies the IP address or port in the dark web relay and bridge IP and port list obtained in step 1, it is considered that the behavior of accessing dark web relays and bridges has been identified. When the behavior has downstream traffic, the network account corresponding to the access behavior is given a high weight; when the behavior has no downstream traffic, the network account corresponding to the access behavior is given a medium weight. Step 4: Identify attempts to access the bridge database using the domain name of the bridge database from the DNS data in the network behavior log restored in Step 2. If the DNS data contains the domain name of the bridge database, it is considered an attempted access behavior and the corresponding network account is given a medium weight. In step 5, the domain names in the log data identified as dark web relay and bridge access behaviors in step 3 are extracted to form a domain name dataset. The domain name dataset is vectorized using the Word2Id model. The K-means clustering algorithm is used to calculate the converged K cluster centers. The domain names corresponding to the cluster centers are used as the characteristic domain names for accessing the dark web. Step 6: After vectorizing the domain names in the daily network behavior logs, calculate the vector distance with the K cluster centers in step 5. Domain names with distances less than the set threshold are selected as candidate dark web access features and added to the characteristic domain names for dark web access obtained in step 5. Step 7: Match the domain name of the DNS data in the network behavior log restored in step 2 with the characteristic domain name of accessing the dark web output in step 5 or step 6 to identify the behavior of attempting to access the dark web through the surface network. When the domain name of the DNS data matches the characteristic domain name of accessing the dark web, it is judged as an attempt to access the dark web website, and the network account corresponding to the access behavior is given a low weight; Step 8: Group the dark web access behaviors captured in steps 3 to 5 by network account, and calculate the integral corresponding to each network account using the following formula: integral=Σnumber of access behaviors*behavior weight Step 9: Output the list of network accounts that access the dark web and the corresponding points to achieve the classification of dark web visitors.

2. The method for analyzing and determining dark web access population from network traffic according to claim 1 is characterized in that: In step 1, the IP is changed each time the IP and port list of the dark web relay and bridge is crawled.

3. The method for analyzing and determining dark web access population from network traffic according to claim 1 is characterized in that: In step 2, DPI technology is used to restore network traffic into traffic logs.

4. The method for analyzing and determining dark web access population from network traffic according to claim 1, characterized in that: The network account is a mobile phone number, IMSI, IMEI or ADSL account.

5. The method for analyzing and determining dark web access population from network traffic according to claim 1 is characterized in that: In step 6, the domain name is manually analyzed and confirmed before being added to the characteristic domain name for accessing the dark web.

Images