A DNS tunnel detection method

Abstract

The application discloses a DNS tunnel detection method and belongs to the technical field of secure communication. The DNS tunnel detection method can solve the problem that the conventional DNS tunnel detection method is difficult to detect low-throughput DNS tunnels and distributed DNS tunnels; the application uses a set first aggregation key to aggregate data packets, uses a self-encoder to preliminarily detect and score the data packets, and detects conventional DNS tunnels; the set second aggregation key and third aggregation key are used to respectively aggregate the metadata of the first-order aggregation again, and the low-throughput DNS tunnels and the distributed DNS tunnels are detected from the communication object and domain name dimensions; the scores of the three self-encoders are sent into a self-encoder as a nonlinear voting mechanism, and finally the nonlinear voting mechanism makes a final judgment according to the scores of the data under the three aggregation keys to obtain a detection result. The application effectively improves the comprehensiveness of the DNS tunnel detection and can be applied to the detection of distributed DNS tunnels.

Description

Technical Field

[0001] The invention relates to a DNS tunnel detection method and belongs to the technical field of secure communications. Background Art

[0002] Traffic control at the network edge is crucial for ensuring information system security, effectively suppressing malicious traffic attacks. However, some attackers exploit tunneling technology to disguise malicious traffic as legitimate traffic, circumventing network edge security policies and conducting illegal activities. Tunneling is a data encapsulation technology that encapsulates the original data packet within another packet using a different protocol for transmission. Traditional tunneling is primarily based on transport layer or network layer protocols. Currently, tunneling technology is increasingly being implemented using application layer protocols, such as the Hypertext Transfer Protocol (HTTP), Domain Name System (DNS), and Secure Shell (SSH).

[0003] The Domain Name System (DNS) plays a vital role in the operation of today's Internet, providing a bidirectional mapping service between IP addresses and domain names. Since it was not originally designed for data transmission, traditional network security software or devices often allow DNS services to use User Datagram Protocol (UDP) port 53 by default. Furthermore, hosts typically unconditionally trust the responses from DNS servers, so DNS traffic can often propagate unimpeded at the edge of the network without strict security policies. Attackers exploit the aforementioned vulnerability and use a covert communication technique called DNS tunneling for data transmission. Figure 2 , attackers usually encode the data to be leaked and put it into the subdomain, and then establish communication with the controlled domain name server through normal DNS query.

[0004] DNS tunneling for data transmission was originally designed to bypass the network edge and gain free internet access. Due to its effectiveness in bypassing network security mechanisms, more and more services are adopting DNS tunneling. For example, a company proposed a method for distributing updated malware signatures through DNS tunneling, aiming to provide signature update services for antivirus clients. DNS-based remote control malware is considered the most dangerous network attack. Furthermore, DNS tunneling can be used for remote command and control information transmission and data exfiltration. Although DNS tunneling provides only low bandwidth for data transmission, attackers can still steal data or maintain communication with malware through the tunnel. Therefore, DNS tunneling detection technology is urgently needed in both academic and industrial fields.

[0005] DNS tunnel detection technology is of great significance for ensuring the security of information systems. DNS tunnel detection technology can effectively identify DNS tunnel traffic mixed in with normal traffic and promptly control channels where data leakage is occurring. For example, when facing double ransomware using DNS tunnels, the DNS tunnel function can effectively identify DNS tunnel traffic used for data leakage and promptly notify security personnel to stop the loss.

[0006] In the existing technology, DNS tunnel detection can be divided into load-based detection methods and flow-based detection methods according to the different detection subjects. The detection object of the load-based detection method is often a single data packet, while the detection object of the flow-based detection method is a session composed of data packets within a period of time. Load-based detection is usually better than flow-based detection in terms of time cost, but its detection ability for low-throughput DNS tunnel data packets similar to normal data packets is poor. Therefore, if you want to detect low-throughput DNS tunnels, you usually need to adopt a flow-based detection method. The first problem faced by flow-based detection research is how to aggregate scattered data packets. In existing research, aggregation keys exist in the form of (src_ip, src_port, protocol, dst_port, dst_ip), (src_ip, protocol, dst_ip), (domain), etc. Researchers usually conduct research based on the assumption that the DNS tunnel server has only one IP and only uses one domain name, and usually only use one aggregation key for aggregation in the research. However, in reality, attackers can effectively reduce the amount of information on a single domain name or address by interleaving multiple IP addresses and domain names, thereby reducing the risk of detection. This DNS tunneling technology, which utilizes multiple IP addresses and domain names to disperse information across different channels, is known as distributed DNS tunneling. Distributed DNS tunneling can be effectively integrated with botnets. By alternating between domain names and using DNS resolution load balancing technology, DNS query requests are distributed to different zombie hosts, which then forward them to controlled servers. The controlled servers aggregate and reassemble the messages to obtain the leaked data. Therefore, although DNS tunnel detection technology plays a key role in improving information system security, it still requires continuous optimization and development to cope with increasingly complex attack methods and practical application challenges. How to detect low-throughput DNS tunnels, and distributed DNS tunnels, which are more covert than conventional low-throughput DNS tunnels, has become an issue that cannot be ignored.

[0007] In summary, a DNS tunnel detection method is needed. Summary of the Invention

[0008] A brief overview of the present invention is provided below to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important aspects of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is simply to present certain concepts in a simplified form as a prelude to the more detailed description discussed later.

[0009] In view of this, in order to solve the problem in the prior art that traditional DNS tunnel detection methods are difficult to detect hidden distributed DNS tunnels, the present invention provides a DNS tunnel detection method.

[0010] The technical solution is as follows: A DNS tunnel detection method includes the following steps:

[0011] S1. Perform traffic preprocessing by obtaining a training traffic dataset, filtering the traffic dataset, and extracting and storing packet-level metadata to obtain a packet-level database;

[0012] S2. Perform first-order aggregation and detection based on IP-Domain based on the packet-level database to obtain the session-level metadata database, session-level feature library, and trained session-level anomaly detection model;

[0013] S3. Based on the session-level metadata database, we obtain trained domain-level anomaly detection models and communication-level anomaly detection models based on second-order aggregation and detection of IP addresses and domains.

[0014] S4. Based on the trained session-level anomaly detection model, domain-level anomaly detection model, and communication-level anomaly detection model, a nonlinear voter is used to perform comprehensive detection. If a DNS tunnel is detected, a high-level alert is generated.

[0015] Furthermore, the step S1 includes the following steps:

[0016] S11. Obtain a training traffic data set;

[0017] Specifically, the process of obtaining the training traffic dataset includes the normal traffic acquisition stage of virtual machine A as the client initiating DNS queries, the tunnel traffic acquisition stage of the client deploying the DNS tunnel tool on virtual machine A, and the traffic collection script deployment stage of virtual machine B as the local DNS server.

[0018] S12. Filter the DNS packets in the traffic data set to obtain a traffic data set after traffic filtering;

[0019] S13 reads the traffic data set after traffic filtering, extracts and stores packet-level metadata, and obtains a data packet-level database;

[0020] In S11, the normal traffic acquisition phase includes the following steps:

[0021] S1111. Set program-related parameters;

[0022] S1112. Select browser driver;

[0023] S1113. Select a search engine as the basis for subsequent searches;

[0024] S1114. Select a keyword thesaurus as a search seed;

[0025] S1115 uses an interrupt-continuation mechanism to determine whether there is an archive file in the keyword thesaurus. The archive file includes a total set of keywords and a complete set of keywords, which are archived by keywords to record all keywords used in each round of crawling and complete the archived record of keywords that have been used;

[0026] S1116. Randomly extract keywords. If there are no archived files remaining, this operation is regarded as a new round of crawling, and several pieces of data are randomly extracted from the keyword thesaurus files in each field;

[0027] S1117. Archive the keywords, that is, store the keywords extracted in this round into the total keyword set;

[0028] S1118. Read the keyword file. If there is an archive file remaining, this operation is regarded as a continuation of the previous round of crawling. The total set of keywords and the complete set of keywords are read and their difference is taken to obtain the unused keywords.

[0029] S1119. Perform a traversal search on the keywords used in this round. The keywords are considered as seed keywords. After the keyword search, they are stored in the keyword completion set.

[0030] S11110. Implement anti-crawler measures to prevent bots;

[0031] S11111. Determine whether the crawler is countered, that is, determine whether the search engine's anti-crawler measures are effective by detecting the structure of the web page obtained by the search;

[0032] S11112. Perform recommended jump avoidance, that is, when the current keyword can no longer produce search results, jump through the search engine's related recommendations;

[0033] S11113. Traverse the search results;

[0034] S11114. Jump to search page, that is, select sequential page jump or random page jump through user-set parameters;

[0035] S11115. When all the keywords used in this round of operation are traversed, the archive file recording the total set of keywords and the completed set of keywords is deleted;

[0036] In S11, the tunnel traffic acquisition phase includes the following steps:

[0037] S1121. Configure parameters for the DNS tunnel tool;

[0038] S1122. Select the file for DNS tunnel testing;

[0039] S1123 sends a DNS tunnel packet to build a DNS tunnel through the DNS tunnel server;

[0040] S1124. Read the response returned by the DNS tunnel tool server;

[0041] In S11, the stage of deploying the traffic collection script includes the following steps:

[0042] S1131. Select monitoring protocol;

[0043] S1132. Set BPF filtering conditions to quickly filter the underlying protocols of the transport layer, network layer, data link layer, and physical layer;

[0044] S1133. Sniff all packets on the selected network port, i.e., use Scapy to monitor traffic on the specified network adapter;

[0045] S1134. Rapidly filter traffic based on BPF filtering conditions;

[0046] S1135. Perform deep filtering at the application layer for traffic that meets the requirements of the underlying protocol;

[0047] S1136. Save the deep filtered traffic by appending it;

[0048] The S12 includes the following steps:

[0049] S121. Read the traffic data set, that is, read the saved DNS tunnel original traffic file;

[0050] S122. Traverse the DNS packets in the traffic data set;

[0051] S122. If the primary domain name queried in the DNS packet is the same as the domain name set for building the DNS tunnel, it is incrementally saved to the new traffic data set;

[0052] S123. If the primary domain name queried in the DNS packet is different from the domain name set for building the DNS tunnel, it is discarded;

[0053] S124. Obtain the traffic data set after traffic filtering;

[0054] The S13 includes the following steps:

[0055] S131 reads the traffic data set after traffic filtering, that is, reads the saved DNS tunnel original traffic file after traffic filtering;

[0056] S132 initializes the database based on the traffic data set information after traffic filtering and the selected packet-level metadata, so that each traffic data set after traffic filtering corresponds to a data table;

[0057] S133 traverses the DNS data packets in the traffic filtered data set;

[0058] S134 according to the set packet-level metadata, extract the packet-level metadata in the DNS data packet in the traffic data set after traffic filtering;

[0059] S135. Save the extracted packet-level metadata to the corresponding data packet-level metadata table of the database storing metadata, to obtain a data packet-level database.

[0060] Furthermore, the step S2 includes the following steps:

[0061] S21. Based on the packet-level database, a session-level aggregation key represented by src_ip, domain, and dst_ip is generated to obtain a session-level metadata database and a session-level feature library to complete the first-order aggregation based on IP-Domain.

[0062] S22. Based on the session-level feature library, perform tensor conversion and normalize the resulting session feature data tensor. Deep unsupervised training of the session-level autoencoder is performed using the normalized session-level feature data to obtain a trained session-level anomaly detection model, completing first-order IP-Domain-based detection.

[0063] The S21 includes the following steps:

[0064] S211 reads the packet-level metadata table in the packet-level database;

[0065] S212 based on the packet-level database data packet-level metadata table information, the selected session-level metadata database and session-level features to create a session-level database, each data table in the packet-level database corresponds to a session-level metadata table and session-level feature table;

[0066] S213 traverses each packet-level metadata table in the packet-level database;

[0067] S214. Read the packet-level metadata in the data packet-level metadata table;

[0068] S215. Extract the primary domain name from the domain name field in the package-level metadata;

[0069] S216. Generate a session-level aggregation key by extracting the source IP address denoted as src_ip and the destination IP address denoted as dst_ip from the packet-level metadata, and combining them with the primary domain name domain extracted in step S215 to obtain a session-level aggregation key denoted as src_ip, domain, dst_ip;

[0070] S217. Create session-level metadata entries. That is, if the currently processed aggregate key does not have a related record in the session-level metadata dictionary, then treat it as a new session, create related entries in the session-level metadata dictionary and the session-level time dictionary, and integrate the session-level metadata entries into session-level metadata.

[0071] S218. Update the session-level metadata, that is, if the currently processed aggregate key has a related record in the session-level metadata dictionary, update the session-level metadata dictionary and the session-level time dictionary;

[0072] S219. Obtaining session-level features by performing aggregate mean calculation on the session-level metadata;

[0073] S2110. The aggregation key and session-level metadata are saved in the session-level metadata table to obtain a session-level metadata database, and the aggregation key and session-level feature data are saved in the session-level feature table to obtain a session-level feature database;

[0074] The S22 includes the following steps:

[0075] S221. Read the session-level feature data table in the session-level feature library;

[0076] S222. Traverse each session-level feature data table in the session-level feature library;

[0077] S223 reads the characteristic data in each session-level characteristic data table. To avoid memory explosion, each reads the set N characteristic data;

[0078] S224. Perform tensor conversion to convert the read feature data from a tuple type to a tensor type suitable for the deep learning framework Pytorch;

[0079] S225. Process the session feature data tensor using the min-max normalization method to obtain normalized session feature data, and map each type of metadata value to the interval [0, 1].

[0080] S226. Use the normalized session-level feature data as input data for the session-level autoencoder to perform deep unsupervised training.

[0081] S227. Save the trained session-level anomaly detection model as a .pth file.

[0082] Furthermore, the step S3 includes the following steps:

[0083] S31. According to the session-level metadata database, a communication-level aggregation key represented by src_ip and dst_ip is generated to obtain a communication-level feature library to complete the communication-level feature aggregation;

[0084] S32. Generate domain-level aggregation keys based on the session-level metadata database to obtain domain-level feature libraries and complete domain-level feature aggregation;

[0085] S33. Use the standardized domain-level features and communication-level feature data to perform deep unsupervised training on the domain-level autoencoder and communication-level autoencoder, respectively, to obtain trained domain-level anomaly detection models and communication-level anomaly detection models, completing second-order detection;

[0086] The S31 includes the following steps:

[0087] S311. Read all data tables in the session-level metadata database;

[0088] S312 based on the data session-level metadata database session-level metadata table information and the selected communication-level features, create a communication-level feature database, each session-level metadata table corresponds to the communication-level feature table;

[0089] S313 traverses each session-level metadata table in the session-level metadata database;

[0090] S314. Read the session-level metadata in the session-level metadata table;

[0091] S315 generates a communication-level aggregation key, extracts the source IP represented as src_ip and the destination IP represented as dst_ip in the session-level metadata, and combines them to obtain a communication-level aggregation key represented as src_ip, dst_ip;

[0092] S316. Create a communication-level metadata entry, that is, if the currently processed aggregation key does not have a related record in the communication-level metadata dictionary, it is considered a new session, and a related entry is created in the communication-level metadata dictionary to be integrated into the communication-level metadata;

[0093] S317. Update the communication-level metadata, that is, if the currently processed aggregate key has a related record in the communication-level metadata dictionary, then update the communication-level metadata dictionary;

[0094] S318. Obtain communication-level features by calculating the communication-level metadata;

[0095] S319. The communication-level aggregation key and the communication-level feature are saved in the communication-level feature table to obtain a communication-level feature library;

[0096] The S32 includes the following steps:

[0097] S321. Read all session-level metadata tables in the session-level metadata database;

[0098] S322 based on the data session-level metadata table information and the selected domain-level features in the session-level metadata database, create a domain-level feature database, each session-level metadata table corresponds to the domain-level feature table;

[0099] S323. Traverse each session-level metadata table in the session-level metadata database;

[0100] S324. Read the session-level metadata in the session-level metadata table;

[0101] S325. Generate a domain-level aggregation key, that is, extract the primary domain name domain in the session-level metadata as the domain-level aggregation key;

[0102] S326. Create a domain-level metadata entry. If the currently processed aggregation key does not have a related record in the domain-level metadata dictionary, it is considered a new session, and a related entry is created in the domain-level metadata dictionary to be integrated into the domain-level metadata.

[0103] S327. Update the domain-level metadata, that is, if the currently processed aggregation key has a related record in the domain-level metadata dictionary, then update the domain-level metadata dictionary;

[0104] S328. Obtain domain-level features by calculating domain-level metadata;

[0105] S329. Save the domain-level aggregation key and domain-level features into the domain-level feature table to obtain a domain-level feature library.

[0106] Furthermore, the step S4 includes the following steps:

[0107] S41. Read the reconstruction errors of the trained session-level anomaly detection model, domain-level anomaly detection model, and communication-level anomaly detection model;

[0108] S42. Calculate the reconstruction errors output by the three anomaly detection models using a voting machine to obtain an anomaly score;

[0109] S43. If the anomaly score output by the voting machine is greater than the set threshold, an alarm is issued;

[0110] S44. If the anomaly score output by the voter is less than the set threshold, the current traffic is not processed.

[0111] The beneficial effects of the present invention are as follows: The present invention proposes a DNS tunnel detection method, which achieves a good balance between detection performance and detection time through two aggregations and detections, and the detection range is more comprehensive; the present invention uses session-level aggregation keys to aggregate DNS data packets in the first stage, and uses autoencoders to perform preliminary detection and scoring. This process is carried out in real time, and the detection target is a conventional DNS tunnel; the present invention then uses communication-level aggregation keys and domain-level aggregation keys to aggregate the first-order aggregated metadata again in the second stage, and uses autoencoders to score them respectively. The scores of the three autoencoders are then sent to an autoencoder as a nonlinear voting mechanism, and finally the nonlinear voting mechanism makes a final judgment based on the scores of the data under the three aggregation keys; the entire processing process of the second stage is near real-time, and its detection targets are low-throughput DNS tunnels and distributed DNS tunnels with good concealment; based on the principle of anomaly detection, the present invention uses unsupervised learning to train and test three autoencoders, thereby realizing the detection of different types of DNS tunnels. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0113] Figure 1 A flowchart of a DNS tunnel detection method is shown;

[0114] Figure 2 This is a schematic diagram of the DNS tunnel principle;

[0115] Figure 3 A schematic diagram of a flow chart of an embodiment of a DNS tunnel detection method;

[0116] Figure 4 This is a diagram of the deployment method for traffic preprocessing;

[0117] Figure 5 This is a diagram of the Selenium crawler's workflow;

[0118] Figure 6 This is a schematic diagram of the DNS tunnel workflow;

[0119] Figure 7 Schematic diagram of the workflow of virtual machine B;

[0120] Figure 8 This is a flow chart of traffic filtering;

[0121] Figure 9 A flowchart for extracting and saving package-level metadata;

[0122] Figure 10 Schematic diagram of the process of aggregating and extracting session-level metadata based on session-level aggregation keys;

[0123] Figure 11 This is a diagram of the training process for the session-level anomaly detection model.

[0124] Figure 12 A flowchart of communication-level session feature aggregation calculation based on communication-level aggregation keys is shown;

[0125] Figure 13 A flowchart of domain-level session feature aggregation calculation based on domain-level aggregation keys;

[0126] Figure 14 This is a schematic diagram of the principle of comprehensive detection and alarm;

[0127] Figure 15 Schematic diagram of the workflow of comprehensive detection and alarm. DETAILED DESCRIPTION

[0128] To make the technical solutions and advantages of the embodiments of the present invention more clearly understood, exemplary embodiments of the present invention are further described in detail below with reference to the accompanying drawings. It should be noted that the embodiments described are only a portion of the embodiments of the present invention, and are not an exhaustive list of all embodiments. It should be noted that the embodiments of the present invention and the features thereof may be combined with each other unless they conflict.

[0129] refer to Figures 1-15 This embodiment provides a DNS tunnel detection method, which specifically includes the following steps:

[0130] S1. Perform traffic preprocessing by obtaining a training traffic dataset, filtering the traffic dataset, and extracting and storing packet-level metadata to obtain a packet-level database;

[0131] S2. Perform first-order aggregation and detection based on IP-Domain based on the packet-level database to obtain the session-level metadata database, session-level feature library, and trained session-level anomaly detection model;

[0132] S3. Based on the session-level metadata database, we obtain trained domain-level anomaly detection models and communication-level anomaly detection models based on second-order aggregation and detection of IP addresses and domains.

[0133] S4. Based on the trained session-level anomaly detection model, domain-level anomaly detection model, and communication-level anomaly detection model, a nonlinear voter is used for comprehensive detection. If a DNS tunnel is detected (the anomaly score generated by the nonlinear voter is higher than the threshold), a high-level alarm is generated.

[0134] Specifically, refer to Figure 3 The present invention first acquires and filters the traffic in the traffic preprocessing stage, and obtains the packet-level metadata database after extraction; then performs first-order extraction, and aggregates the packet-level metadata with (src_ip, domain, dst_ip) as the aggregation key to obtain the IP-Domain metadata database; next, the IP-Domain metadata database is used for primary detection and early warning on the one hand, and for data aggregation under the (src_ip, dst_ip) and (domain) aggregation keys on the other hand; finally, the detection results of the metadata under the three aggregation keys are summarized to the voting organization, which conducts comprehensive evaluation and issues corresponding early warnings.

[0135] Furthermore, the step S1 includes the following steps:

[0136] S11. Obtain a training traffic data set;

[0137] Specifically, the process of obtaining the training traffic dataset includes the normal traffic acquisition stage of virtual machine A as the client initiating DNS queries, the tunnel traffic acquisition stage of the client deploying the DNS tunnel tool on virtual machine A, and the traffic collection script deployment stage of virtual machine B as the local DNS server.

[0138] S12. Filter the DNS packets in the traffic data set to obtain a traffic data set after traffic filtering;

[0139] S13. Read the traffic data set after traffic filtering, extract and store packet-level metadata;

[0140] In the S11, reference Figure 5 ,The normal traffic acquisition phase includes the following steps:

[0141] S1111 set program-related parameters, such as the number of search pages, waiting time after the recommended jump, waiting time after turning the page, web browsing time, sleep time before clicking, the number of keywords extracted from each vocabulary;

[0142] S1112. Select an appropriate browser driver, such as ChromeDriver;

[0143] S1113. Select an appropriate search engine as the basis for subsequent searches, such as Baidu, Sohu, or Bing.

[0144] S1114. Select an appropriate keyword lexicon as the search seed. To ensure comprehensive traffic collection, it is recommended to use a lexicon that covers various fields. Keywords in a single field may cause websites related to that field to appear repeatedly, reducing traffic diversity.

[0145] S1115. A break-and-resume mechanism is used to determine whether there are archive files in the keyword lexicon. The archive files include a total keyword set and a completed keyword set. The keyword archive records all keywords used in each round of crawling, and the keyword completion archive records the keywords that have already been used. To cope with complex and changing network conditions and ensure the overall integrity of the data generated by each program run, a break-and-resume mechanism is used to determine whether there are archive files remaining and whether the program has crashed, thereby deciding whether the current run is a new crawling round or a continuation of the previous crawling round.

[0146] S1116. Randomly extract keywords. If there are no archived files remaining, this operation is regarded as a new round of crawling, and several pieces of data are randomly extracted from the keyword thesaurus files in each field;

[0147] S1117. Archive the keywords, that is, store the keywords extracted in this round into the total keyword set;

[0148] S1118. Read the keyword file. If there is an archive file remaining, this operation is regarded as a continuation of the previous round of crawling. The total set of keywords and the complete set of keywords are read and their difference is taken to obtain the unused keywords.

[0149] S1119. Perform a traversal search on the keywords used in this round. The keywords are considered as seed keywords. After the keyword search, they are stored in the keyword completion set.

[0150] S11110. Pre-circumvent anti-crawler measures, namely, pre-circumvent possible crawler detection measures by disabling the automation control feature of the Blink rendering engine, setting the User-Agent option, modifying the experimental options excludeSwitches and useAutomationExtension, and using JavaScript to override the navigator.webdriver property;

[0151] S11111. Determine whether the crawler is countered, that is, determine whether the search engine's anti-crawler measures are effective by detecting the structure of the web page obtained by the search;

[0152] S11112. Avoid recommended redirection. That is, when the current keyword can no longer produce search results, redirect to related recommendations from Baidu's search engine. Due to the existence of anti-crawler mechanisms and the limitations of term search results, it is impossible to guarantee that each keyword will receive a sufficient number of search results. Therefore, a recommended redirection strategy is introduced. Since related recommendations are often keywords in the same field, the number of browsed topics in each field can be roughly the same.

[0153] S11113. After traversing the search results, the Selenium crawler will randomly stay on the page for a period of time to simulate user behavior and circumvent anti-crawler measures;

[0154] S11114. Jump to search page, i.e., select sequential page jump or random page jump based on user-defined parameters. Sequential page jump is more in line with user habits. Users who use search engines tend to only focus on highly ranked web pages, i.e., sequential page browsing. Random page jump can expand the coverage of web pages collected for traffic data and solve the problem of less visited web pages being ranked low in search results, thus making traffic more comprehensive.

[0155] S11115. When all the keywords used in this round of operation are traversed, the archive file recording the total set of keywords and the completed set of keywords is deleted;

[0156] In the S11, reference Figure 6 ,The tunnel traffic acquisition phase includes the following steps:

[0157] S1121. Configure parameters for the DNS tunnel tool, such as encoding, DNS resource type, and sending interval;

[0158] S1122 selects a suitable file for DNS tunnel testing. In this embodiment, several randomly generated files of 1KB size are selected;

[0159] S1123 sends a DNS tunnel packet to build a DNS tunnel through the DNS tunnel server;

[0160] S1124. Read the response returned by the DNS tunnel tool server;

[0161] In the S11, reference Figure 7 The deployment of traffic collection scripts includes the following steps:

[0162] S1131. Select the monitoring protocol, that is, select the type of network protocol to be collected, such as DNS, HTTP, etc.;

[0163] S1132. Set BPF filtering conditions to quickly filter the underlying protocols such as the transport layer, network layer, data link layer, and physical layer;

[0164] S1133. Sniff all packets on the selected network port, i.e., use Scapy to monitor traffic on the specified network adapter;

[0165] S1134. Perform a relatively shallow and rapid filtering of the traffic based on the BPF filtering conditions;

[0166] S1135. Perform deep filtering at the application layer for traffic that meets the requirements of the underlying protocol;

[0167] S1136. Save the deep filtered traffic by appending it;

[0168] In S12, reference Figure 8 , including the following steps:

[0169] S121. Read the traffic data set, that is, read the saved DNS tunnel original traffic file;

[0170] S122. Traverse the DNS packets in the traffic data set;

[0171] S122. If the primary domain name queried in the DNS packet is the same as the domain name set for building the DNS tunnel, it is incrementally saved to the new traffic data set;

[0172] S123. If the primary domain name queried in the DNS packet is different from the domain name set for building the DNS tunnel, it is discarded;

[0173] S124. Obtain the traffic data set after traffic filtering;

[0174] Specifically, the main content of traffic filtering is to filter normal traffic mixed in the DNS tunnel traffic file through specific domain names to ensure that the test set is sufficiently pure and the test results are more accurate;

[0175] In the S13, reference Figure 9 , including the following steps:

[0176] S131 reads the traffic data set after traffic filtering, that is, reads the saved DNS tunnel original traffic file after traffic filtering;

[0177] S132 initializes the database based on the traffic data set information after traffic filtering and the selected packet-level metadata, so that each traffic data set after traffic filtering corresponds to a data table;

[0178] S133 traverses the DNS data packets in the traffic filtered data set;

[0179] S134 according to the set packet-level metadata, extract the packet-level metadata in the DNS data packet in the traffic data set after traffic filtering;

[0180] S135. Save the extracted packet-level metadata to the corresponding data packet-level metadata table of the database storing metadata, to obtain a data packet-level database.

[0181] Specifically, referring to Table 1, which shows the packet-level metadata, packet-level metadata has advantages over the original traffic file, such as small storage space occupation and high usage efficiency. However, the disadvantage is that it needs to be inferred based on the subsequent aggregated metadata. Therefore, when the subsequent aggregated metadata changes, the packet-level metadata needs to be re-formulated and extracted. The packet-level metadata database stores the packet-level metadata extracted from each traffic data set in the form of a data table. In this embodiment, the lightweight database SQLite is used as the database for storing metadata.

[0182] refer to Figure 4 The client part consists of two virtual machines and a host machine. In the normal traffic acquisition stage, virtual machine A runs the Selenium crawler program, generates DNS requests by imitating the user's browsing behavior, and sends them to virtual machine B. Virtual machine B runs as a local DNS server, runs a traffic monitoring program on it, and captures all DNS traffic flowing through virtual machine B. The host machine sends commands to virtual machine A to control the operation of its crawler program and regularly receives traffic files captured by virtual machine B; the server side is composed of several DNS servers, among which the root domain name server and top-level domain name server are public servers. When acquiring normal traffic, the authoritative server is the server of the crawler target website, and when acquiring tunnel traffic, the DNS tunnel server as the authoritative domain name server is autonomously controlled. That is, in the tunnel traffic acquisition stage, the DNS tunnel tool client runs on virtual machine A, and the server side of various open source DNS tunnel tools runs on the DNS tunnel server, and DNS tunnel traffic is captured on virtual machine B.

[0183]

[0184] Table 1

[0185] Furthermore, the step S2 includes the following steps:

[0186] S21. Based on the packet-level database, a session-level aggregation key represented by src_ip, domain, and dst_ip is generated to obtain a session-level metadata database and a session-level feature library to complete the first-order aggregation based on IP-Domain.

[0187] S22. Reference Figure 11, according to the session-level feature library, tensor conversion is performed, the obtained session feature data tensor is standardized, and the session-level autoencoder is trained in deep unsupervised mode using the standardized session-level feature data to obtain a trained session-level anomaly detection model and complete the first-order detection based on IP-Domain;

[0188] In the S21, reference Figure 10 , including the following steps:

[0189] S211 reads the packet-level metadata table in the packet-level database;

[0190] S212 based on the packet-level database data packet-level metadata table information, the selected session-level metadata database and session-level features to create a session-level database, each data table in the packet-level database corresponds to a session-level metadata table and session-level feature table;

[0191] S213 traverses each packet-level metadata table in the packet-level database;

[0192] S214. Read the packet-level metadata in the data packet-level metadata table;

[0193] S215. Extract the primary domain name from the domain name field in the package-level metadata;

[0194] S216. Generate a session-level aggregation key by extracting the source IP address denoted as src_ip and the destination IP address denoted as dst_ip from the packet-level metadata, and combining them with the primary domain name domain extracted in step S215 to obtain a session-level aggregation key denoted as src_ip, domain, dst_ip;

[0195] S217. Create session-level metadata entries. That is, if the currently processed aggregate key does not have a related record in the session-level metadata dictionary, then treat it as a new session, create related entries in the session-level metadata dictionary and the session-level time dictionary, and integrate the session-level metadata entries into session-level metadata.

[0196] S218. Update the session-level metadata, that is, if the currently processed aggregate key has a related record in the session-level metadata dictionary, update the session-level metadata dictionary and the session-level time dictionary;

[0197] S219. Obtaining session-level features by performing aggregate mean calculation on the session-level metadata;

[0198] S2110. The aggregation key and session-level metadata are saved in the session-level metadata table to obtain a session-level metadata database, and the aggregation key and session-level feature data are saved in the session-level feature table to obtain a session-level feature database;

[0199] Specifically, refer to Figure 10 and Figure 11 The goal of first-order aggregation and detection is to detect common single-IP single-domain DNS tunnels. In these DNS tunnels, there is only one IP address serving as the DNS tunnel server, and the primary domain name is the same when the DNS tunnel is constructed. Therefore, session-level aggregation keys can effectively aggregate and highlight the anomalies of these DNS tunnels. Under normal circumstances, the IP addresses of both parties in a single DNS query session are fixed, that is, there is only the client IP address initiating the DNS query and the server IP address providing the DNS response, and there is only one queried domain name. Therefore, the metadata calculated based on the session-level aggregation key is called session-level metadata.

[0200] The work in the first-order aggregation stage revolves around the aggregation calculation of packet-level metadata using session-level aggregation keys to obtain session-level metadata. By aggregating and calculating the packet-level metadata in the packet-level metadata database, we obtain the IP-Domain metadata database and the IP-Domain feature database. Each data table in the above two databases corresponds to a traffic file. Since the start time of the same session under the aggregation key is the same, data updates and insertions will result in a large amount of duplicate data. Therefore, to save memory and speed up processing efficiency, time-related metadata is separated from other metadata during the session-level metadata aggregation calculation process. The IP-Domain metadata database and time database are used for second-order aggregation calculations, while the IP-Domain feature database is used for primary detection and early warning.

[0201] Refer to Table 2, which represents session-level metadata;

[0202] Refer to Table 3, which shows the session-level features.

[0203]

[0204] Table 2

[0205]

[0206] Table 3

[0207] In the S22, reference Figure 11 , including the following steps:

[0208] S221. Read the session-level feature data table in the session-level feature library;

[0209] S222. Traverse each session-level feature data table in the session-level feature library;

[0210] S223 reads the characteristic data in each session-level characteristic data table. To avoid memory explosion, each time the characteristic data is read, N pieces are set. In this embodiment, N is set to 10000.

[0211] S224. Perform tensor conversion to convert the read feature data from a tuple type to a tensor type suitable for the deep learning framework Pytorch;

[0212] S225. Process the session feature data tensor using the min-max normalization method to obtain normalized session feature data, and map each type of metadata value to the interval [0, 1].

[0213] S226. Use the normalized session-level feature data as input data for the session-level autoencoder to perform deep unsupervised training.

[0214] S227. Save the trained session-level anomaly detection model as a .pth file;

[0215] Specifically, we use PyTorch to implement the deep learning part of the anomaly detection model. In the first-order detection stage, based on the anomaly detection principle, we use a session-level autoencoder as the anomaly detection model and use RMSE as the loss function to perform feature detection on the traffic.

[0216] The detection model consists of an encoder and a decoder. The input data x, i.e., the normalized session-level feature data, is input into the encoder and encoded into an intermediate value m. The decoder reconstructs the original input from the intermediate value m to obtain the output data x'.

[0217] The intermediate value m is expressed as:

[0218] m=encoder(x)

[0219] The output data x' is expressed as:

[0220] x′=decoder(m)

[0221] The loss value e is expressed as:

[0222] e=loss(x′-x)

[0223] Among them, loss is the loss value calculation function;

[0224] The encoder decoder consists of a 12×10 fully connected layer, a 10×8 fully connected layer, an 8×6 fully connected layer, and a 6×4 fully connected layer, with ReLU activation function layers between each fully connected layer. The decoder consists of a 4×6 fully connected layer, a 6×8 fully connected layer, an 8×10 fully connected layer, and a 10×12 fully connected layer. The first three fully connected layers of the decoder are separated by ReLU activation function layers, and the last fully connected layer is followed by a Sigmoid activation function layer.

[0225] The detection model uses the Adam optimizer during training;

[0226] The loss RMSE(x,y) is expressed as:

[0227]

[0228] Among them, x i is the i-th element of the input data, y i is the i-th element of the output data, and k is the total number of elements of the input / output data.

[0229] Furthermore, the step S3 includes the following steps:

[0230] S31. According to the session-level metadata database, a communication-level aggregation key represented by src_ip and dst_ip is generated to obtain a communication-level feature library to complete the communication-level feature aggregation;

[0231] S32. Generate domain-level aggregation keys based on the session-level metadata database to obtain domain-level feature libraries and complete domain-level feature aggregation;

[0232] S33. Use the standardized domain-level features and communication-level feature data to perform deep unsupervised training on the domain-level autoencoder and communication-level autoencoder, respectively, to obtain trained domain-level anomaly detection models and communication-level anomaly detection models, completing second-order detection;

[0233] In the S31, reference Figure 12 , including the following steps:

[0234] S311. Read all data tables in the session-level metadata database;

[0235] S312 based on the data session-level metadata database session-level metadata table information and the selected communication-level features, create a communication-level feature database, each session-level metadata table corresponds to the communication-level feature table;

[0236] S313 traverses each session-level metadata table in the session-level metadata database;

[0237] S314. Read the session-level metadata in the session-level metadata table;

[0238] S315 generates a communication-level aggregation key, extracts the source IP represented as src_ip and the destination IP represented as dst_ip in the session-level metadata, and combines them to obtain a communication-level aggregation key represented as src_ip, dst_ip;

[0239] S316. Create a communication-level metadata entry, that is, if the currently processed aggregation key does not have a related record in the communication-level metadata dictionary, it is considered a new session, and a related entry is created in the communication-level metadata dictionary to be integrated into the communication-level metadata;

[0240] S317. Update the communication-level metadata, that is, if the currently processed aggregate key has a related record in the communication-level metadata dictionary, then update the communication-level metadata dictionary;

[0241] S318. Obtain communication-level features by calculating the communication-level metadata;

[0242] S319. The communication-level aggregation key and the communication-level feature are saved in the communication-level feature table to obtain a communication-level feature library;

[0243] In the S32, reference Figure 13 , including the following steps:

[0244] S321. Read all session-level metadata tables in the session-level metadata database;

[0245] S322 based on the data session-level metadata table information and the selected domain-level features in the session-level metadata database, create a domain-level feature database, each session-level metadata table corresponds to the domain-level feature table;

[0246] S323. Traverse each session-level metadata table in the session-level metadata database;

[0247] S324. Read the session-level metadata in the session-level metadata table;

[0248] S325. Generate a domain-level aggregation key, that is, extract the primary domain name domain in the session-level metadata as the domain-level aggregation key;

[0249] S326. Create a domain-level metadata entry. If the currently processed aggregation key does not have a related record in the domain-level metadata dictionary, it is considered a new session, and a related entry is created in the domain-level metadata dictionary to be integrated into the domain-level metadata.

[0250] S327. Update the domain-level metadata, that is, if the currently processed aggregation key has a related record in the domain-level metadata dictionary, then update the domain-level metadata dictionary;

[0251] S328. Obtain domain-level features by calculating domain-level metadata;

[0252] S329. Save the domain-level aggregation key and domain-level features to the domain-level feature table to obtain a domain-level feature library;

[0253] Specifically, refer to Figure 12 and Figure 13 The goal of second-order aggregation and detection is to detect distributed DNS tunnels. Distributed DNS tunnels can be used in conjunction with botnets. Attackers can deploy the DNS tunnel tool server on bot hosts, so that the IP of the DNS tunnel server is not the same. At the same time, attackers can register multiple domain names, or use the domain name generation algorithm DGA to generate multiple domain names. The characteristics of multiple IP × multiple domain names allow attackers to disperse the DNS tunnels that were originally concentrated in the same session and easily highlighted by session-level aggregation to different domain name sessions on different hosts. The present invention solves the problem of distributed DNS tunnels by combining communication-level aggregation keys with domain-level aggregation keys. In a communication process, the IPs of the two communicating parties are fixed, so the data aggregated based on the (src_ip, dst_ip) aggregation key is called the communication level. The role of the aggregation key (domain) is to aggregate all data under the same primary domain name, so we call the data aggregated based on the aggregation key (domain) the domain level.

[0254] In the second-order aggregation stage, since the packets aggregated by the first-order aggregation key (src_ip, domain, dst_ip) are subgroups of the second-order aggregation keys (src_ip, dst_ip) and (domain), the detection features under the second-order aggregation key can be directly obtained by aggregating and calculating the metadata under the first-order aggregation key;

[0255] The principle and process of the second-order detection stage are similar to those of the first-order detection stage, but the reconstruction error output by the detection model is not used as a separate warning reference, but is aggregated to the voting agency as input data for comprehensive detection;

[0256] Refer to Table 4, which shows the communication level characteristics;

[0257] Refer to Table 5, which shows domain-level features.

[0258]

[0259] Table 4

[0260]

[0261]

[0262] Table 5

[0263] Furthermore, the step S4 includes the following steps:

[0264] S41. Read the reconstruction errors of the trained session-level anomaly detection model, domain-level anomaly detection model, and communication-level anomaly detection model;

[0265] S42. Calculate the reconstruction errors output by the three anomaly detection models using a voting machine to obtain an anomaly score;

[0266] S43. If the anomaly score output by the voting machine is greater than the set threshold, an alarm is issued;

[0267] If the anomaly score output by the voting device is less than the set threshold, the current traffic is not processed and the traffic being tested is considered safe.

[0268] Specifically, refer to Figure 14 and Figure 15 , Threshold Checker represents the threshold checker, Alarm represents the alarm, and the comprehensive detection stage comprehensively analyzes the detection results of the three aggregation features to obtain the final result of whether the current DNS traffic is DNS tunnel traffic. Since the second-order aggregation requires a certain amount of running time, the real-time performance of the comprehensive detection is weaker than that of the first-order detection, but the comprehensive detection is stronger than the first-order detection in detection performance and can detect distributed DNS tunnels. In this embodiment, the voter used is a small autoencoder, which is trained using the reconstruction errors output by the above three anomaly detection models in the test phase. The training process is similar to that of the anomaly detection model.

[0269] Although the present invention has been described with respect to a limited number of embodiments, it will be apparent to those skilled in the art, having benefit of the foregoing description, that other embodiments are contemplated within the scope of the invention thus described. Furthermore, it should be noted that the language used in this specification has been selected primarily for readability and didactic purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention. Consequently, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A DNS tunnel detection method, characterized in that: The following steps are involved: S1. Perform traffic preprocessing by obtaining a training traffic dataset, filtering the traffic dataset, and extracting and storing packet-level metadata to obtain a packet-level database; S2. Perform first-order aggregation and detection based on IP-Domain based on the packet-level database to obtain the session-level metadata database, session-level feature library, and trained session-level anomaly detection model; S3. Based on the session-level metadata database, we obtain trained domain-level anomaly detection models and communication-level anomaly detection models based on second-order aggregation and detection of IP addresses and domains. S4. Perform comprehensive detection using a nonlinear voter based on the trained session-level, domain-level, and communication-level anomaly detection models. If a DNS tunnel is detected, a high-level alert is generated. Said S2 comprises the following steps: S21. Based on the packet-level database, a session-level aggregation key represented by src_ip, domain, and dst_ip is generated to obtain a session-level metadata database and a session-level feature library to complete the first-order aggregation based on IP-Domain. S22. Based on the session-level feature library, perform tensor conversion and normalize the resulting session feature data tensor. Deep unsupervised training of the session-level autoencoder is performed using the normalized session-level feature data to obtain a trained session-level anomaly detection model, completing first-order IP-Domain-based detection. The S21 includes the following steps: S211 reads the packet-level metadata table in the packet-level database; S212 based on the packet-level database data packet-level metadata table information, the selected session-level metadata database and session-level features to create a session-level database, each data table in the packet-level database corresponds to a session-level metadata table and session-level feature table; S213 traverses each packet-level metadata table in the packet-level database; S214. Read the packet-level metadata in the data packet-level metadata table; S215. Extract the primary domain name from the domain name field in the package-level metadata; S216. Generate a session-level aggregation key by extracting the source IP address denoted as src_ip and the destination IP address denoted as dst_ip from the packet-level metadata, and combining them with the primary domain name domain extracted in step S215 to obtain a session-level aggregation key denoted as src_ip, domain, dst_ip; S217. Create session-level metadata entries. That is, if the currently processed aggregate key does not have a related record in the session-level metadata dictionary, then treat it as a new session, create related entries in the session-level metadata dictionary and the session-level time dictionary, and integrate the session-level metadata entries into session-level metadata. S218. Update the session-level metadata, that is, if the currently processed aggregate key has a related record in the session-level metadata dictionary, update the session-level metadata dictionary and the session-level time dictionary; S219. Obtaining session-level features by performing aggregate mean calculation on the session-level metadata; S2110. The aggregation key and session-level metadata are saved in the session-level metadata table to obtain a session-level metadata database, and the aggregation key and session-level feature data are saved in the session-level feature table to obtain a session-level feature database; The S22 includes the following steps: S221. Read the session-level feature data table in the session-level feature library; S222. Traverse each session-level feature data table in the session-level feature library; S223 reads the characteristic data in each session-level characteristic data table. To avoid memory explosion, each reads the set N characteristic data; S224. Perform tensor conversion to convert the read feature data from a tuple type to a tensor type suitable for the deep learning framework Pytorch; S225. Process the session feature data tensor using the min-max normalization method to obtain normalized session feature data, and map each type of metadata value to the interval [0, 1]. S226. Use the normalized session-level feature data as input data for the session-level autoencoder to perform deep unsupervised training. S227. Save the trained session-level anomaly detection model as a .pth file; The S3 includes the following steps: S31. According to the session-level metadata database, a communication-level aggregation key represented by src_ip and dst_ip is generated to obtain a communication-level feature library to complete the communication-level feature aggregation; S32. Generate domain-level aggregation keys based on the session-level metadata database to obtain domain-level feature libraries and complete domain-level feature aggregation; S33. Use the standardized domain-level features and communication-level feature data to perform deep unsupervised training on the domain-level autoencoder and communication-level autoencoder, respectively, to obtain trained domain-level anomaly detection models and communication-level anomaly detection models, completing second-order detection; The S31 includes the following steps: S311. Read all data tables in the session-level metadata database; S312 based on the data session-level metadata database session-level metadata table information and the selected communication-level features, create a communication-level feature database, each session-level metadata table corresponds to the communication-level feature table; S313 traverses each session-level metadata table in the session-level metadata database; S314. Read the session-level metadata in the session-level metadata table; S315 generates a communication-level aggregation key, extracts the source IP represented as src_ip and the destination IP represented as dst_ip in the session-level metadata, and combines them to obtain a communication-level aggregation key represented as src_ip, dst_ip; S316. Create a communication-level metadata entry, that is, if the currently processed aggregation key does not have a related record in the communication-level metadata dictionary, it is considered a new session, and a related entry is created in the communication-level metadata dictionary to be integrated into the communication-level metadata; S317. Update the communication-level metadata, that is, if the currently processed aggregate key has a related record in the communication-level metadata dictionary, then update the communication-level metadata dictionary; S318. Obtain communication-level features by calculating the communication-level metadata; S319. The communication-level aggregation key and the communication-level feature are saved in the communication-level feature table to obtain a communication-level feature library; The S32 includes the following steps: S321. Read all session-level metadata tables in the session-level metadata database; S322 based on the data session-level metadata table information and the selected domain-level features in the session-level metadata database, create a domain-level feature database, each session-level metadata table corresponds to the domain-level feature table; S323. Traverse each session-level metadata table in the session-level metadata database; S324. Read the session-level metadata in the session-level metadata table; S325. Generate a domain-level aggregation key, that is, extract the primary domain name domain in the session-level metadata as the domain-level aggregation key; S326. Create a domain-level metadata entry. If the currently processed aggregation key does not have a related record in the domain-level metadata dictionary, it is considered a new session, and a related entry is created in the domain-level metadata dictionary to be integrated into the domain-level metadata. S327. Update the domain-level metadata, that is, if the currently processed aggregation key has a related record in the domain-level metadata dictionary, then update the domain-level metadata dictionary; S328. Obtain domain-level features by calculating domain-level metadata; S329. Save the domain-level aggregation key and domain-level features into the domain-level feature table to obtain a domain-level feature library.

2. A DNS tunnel detection method according to claim 1, characterized in that: Said S1 comprises the following steps: S11. Obtain a training traffic data set; Specifically, the process of obtaining the training traffic dataset includes the normal traffic acquisition stage of virtual machine A as the client initiating DNS queries, the tunnel traffic acquisition stage of the client deploying the DNS tunnel tool on virtual machine A, and the traffic collection script deployment stage of virtual machine B as the local DNS server. S12. Filter the DNS packets in the traffic data set to obtain a traffic data set after traffic filtering; S13 reads the traffic data set after traffic filtering, extracts and stores packet-level metadata, and obtains a data packet-level database; In S11, the normal traffic acquisition phase includes the following steps: S1111. Set program-related parameters; S1112. Select browser driver; S1113. Select a search engine as the basis for subsequent searches; S1114. Select a keyword thesaurus as a search seed; S1115 uses an interrupt-continuation mechanism to determine whether there is an archive file in the keyword thesaurus. The archive file includes a total set of keywords and a complete set of keywords, which are archived by keywords to record all keywords used in each round of crawling and complete the archived record of keywords that have been used; S1116. Randomly extract keywords. If there are no archived files remaining, this operation is regarded as a new round of crawling, and several pieces of data are randomly extracted from the keyword thesaurus files in each field; S1117. Archive the keywords, that is, store the keywords extracted in this round into the total keyword set; S1118. Read the keyword file. If there is an archive file remaining, this operation is regarded as a continuation of the previous round of crawling. The total set of keywords and the complete set of keywords are read and their difference is taken to obtain the unused keywords. S1119. Perform a traversal search on the keywords used in this round. The keywords are considered as seed keywords. After the keyword search, they are stored in the keyword completion set. S11110. Implement anti-crawler measures to prevent bots; S11111. Determine whether the crawler is countered, that is, determine whether the search engine's anti-crawler measures are effective by detecting the structure of the web page obtained by the search; S11112. Perform recommended jump avoidance, that is, when the current keyword can no longer produce search results, jump through the search engine's related recommendations; S11113. Traverse the search results; S11114. Jump to search page, that is, select sequential page jump or random page jump through user-set parameters; S11115. When all the keywords used in this round of operation are traversed, the archive file recording the total set of keywords and the completed set of keywords is deleted; In S11, the tunnel traffic acquisition phase includes the following steps: S1121. Configure parameters for the DNS tunnel tool; S1122. Select the file for DNS tunnel testing; S1123 sends a DNS tunnel packet to build a DNS tunnel through the DNS tunnel server; S1124. Read the response returned by the DNS tunnel tool server; In S11, the stage of deploying the traffic collection script includes the following steps: S1131. Select monitoring protocol; S1132. Set BPF filtering conditions to quickly filter the underlying protocols of the transport layer, network layer, data link layer, and physical layer; S1133. Sniff all packets on the selected network port, i.e., use Scapy to monitor traffic on the specified network adapter; S1134. Rapidly filter traffic based on BPF filtering conditions; S1135. Perform deep filtering at the application layer for traffic that meets the requirements of the underlying protocol; S1136. Save the deep filtered traffic by appending it; The S12 includes the following steps: S121. Read the traffic data set, that is, read the saved DNS tunnel original traffic file; S122. Traverse the DNS packets in the traffic data set; S122. If the primary domain name queried in the DNS packet is the same as the domain name set for building the DNS tunnel, it is incrementally saved to the new traffic data set; S123. If the primary domain name queried in the DNS packet is different from the domain name set for building the DNS tunnel, it is discarded; S124. Obtain the traffic data set after traffic filtering; The S13 includes the following steps: S131 reads the traffic data set after traffic filtering, that is, reads the saved DNS tunnel original traffic file after traffic filtering; S132 initializes the database based on the traffic data set information after traffic filtering and the selected packet-level metadata, so that each traffic data set after traffic filtering corresponds to a data table; S133 traverses the DNS data packets in the traffic filtered data set; S134 according to the set packet-level metadata, extract the packet-level metadata in the DNS data packet in the traffic data set after traffic filtering; S135. Save the extracted packet-level metadata to the corresponding data packet-level metadata table of the database storing metadata, to obtain a data packet-level database.

3. A DNS tunnel detection method according to claim 2, characterized in that: Said S4 comprises the following steps: S41. Read the reconstruction errors of the trained session-level anomaly detection model, domain-level anomaly detection model, and communication-level anomaly detection model; S42. Calculate the reconstruction errors output by the three anomaly detection models using a voting machine to obtain an anomaly score; S43. If the anomaly score output by the voting machine is greater than the set threshold, an alarm is issued; S44. If the anomaly score output by the voter is less than the set threshold, the current traffic is not processed.

Images