A multi-dimensional optimization-based fast DoH service measurement method and tool

Abstract

The application discloses a kind of fast DoH service measurement methods and tools based on multidimensional optimization.The method is:1) scanning target network, obtaining probe target;2) applying public domain name as base domain name, and based on base domain name selected a subdomain name as probe subdomain name;Then based on probe subdomain name, each probe target is assigned a unique subdomain name;3) select or deploy a DNS authority server as self authority server;Base domain name is deployed to self authority server, and each probe target is configured with a wildcard subdomain name based on each unique subdomain name;4) deploy probe tool on another DNS server, send the unique subdomain name assigned to each probe target, and collect return result;5) according to the return result collected and the log on self authority server, identify the DoH service quality of each probe target.Through the application, measurement can be completed in one active detection process.

Description

Technical Field

[0001] This invention relates to the technical field of computer science, and in particular to a fast DoH service measurement method and tool based on multi-dimensional optimization. Background Technology

[0002] The DNS service system is the cornerstone of the current internet interconnectivity. DNS resolution translates domain names into IP addresses, typically by sending requests to resolvers. From a component perspective, the DNS ecosystem includes roles such as the client, resolvers, root servers, and authoritative servers. DNS resolution requests are initiated by the client, sending a DNS query message to the resolver for recursive lookup. Resolvers receiving a query message handle it in two ways, depending on their settings: either performing an iterative query or forwarding the query message to other resolvers to continue the recursive lookup. During iterative lookup, the resolver first sends a domain name request to the root server. If the specified record is not received, it continues querying down to the next level of authoritative servers based on the returned NS records.

[0003] In traditional request methods, plaintext DNS resolution requests can be easily intercepted by third-party network eavesdroppers for service recommendations or censorship purposes. Therefore, to better protect user privacy, various encrypted DNS resolution communication methods have been proposed, such as DoT, DoH, and DoQ. For the purpose of protecting communication security and user privacy, several DoE (DNS over Encryption) methods have been proposed. Existing well-known DNS encryption methods include DoT, DoH, and DoQ. DoT is based on the TLS protocol, DoH on HTTPS, and DoQ on QUIC. DoE methods can effectively protect against line-based DNS hijacking. Line-based DNS hijacking methods are often combined with BGP hijacking, redirecting traffic flowing through the corresponding router to an attacker-specified device. On this device, DNS traffic is filtered, and false responses are made to the queries to carry out the hijacking. This hijacking method is relatively effective against plaintext DNS packets. However, for DNS packets transmitted over encrypted channels, this hijacking method not only fails to obtain the query ID and domain name from the DNS packet but also requires breaking through the underlying encryption protocol's authentication mechanism, thus increasing the difficulty of hijacking. Furthermore, the DoE method is also effective in protecting user privacy and preventing unauthorized third parties from using users' plaintext domain name query information to launch targeted advertising or precision marketing campaigns. In practical applications, the DoE method is mainly used in the iterative query phase, specifically when the client sends a request to the resolver. This includes both DNS requests issued by the system or proxy and DNS requests issued by the browser.

[0004] Among the various DoE methods mentioned above, the DoH method has attracted the most attention because it provides better covert DNS services by sharing public ports with normal web services on top of encrypted channels. In recent years, some studies have proposed methods to discover open DoHs in the wild without relying on public lists. Experimental results show that these methods can effectively measure DoH services. Identifying DoH services in the wild has multiple implications: First, measuring DoH services can discover DNS service resources that can provide services to other users; second, measuring DoH services can assess the development status of existing DoHs, evaluate the dependence and authenticity of existing DoH services, and provide a reference for deploying custom DoH servers; third, measuring DoH services is crucial for identifying potential malicious services. Currently, some malicious services, such as C&C communication, relay through normal DoH servers, so measuring DoH services can provide a basis for further identifying malicious traffic. Therefore, measuring DoH services is of great significance.

[0005] Regarding DoH (DoH) measurement, one study proposed a domain name-based DoH detection method, which discovered 5715 DoH servers in 2022. Although the domain name was appended to the request and various HTTP fields such as application and path were combined, the study also indicated that 87.4% of the machines could be accessed solely via IP address, while the remaining 12.6% could be accessed via both IP address and domain name. Another study proposed an efficient three-stage DoH detection method, which successfully identified 4354 DoH servers in 2022.

[0006] While the methods mentioned above can effectively discover existing DoH services, they still have problems with detection efficiency: first, the measurement process requires data collection in multiple stages, which consumes a lot of time in the actual DoH detection process; second, in terms of packet volume, a lot of invalid data packets need to be sent, which increases the cost of network operation. Summary of the Invention

[0007] To address the aforementioned issues, the present invention aims to provide a rapid DoH service measurement method and tool based on multi-dimensional optimization. This invention optimizes DoH services based on three dimensions. First, at the strategy level, it proposes a multi-field filtering method, filtering hosts that are highly unlikely to provide DoH services by filtering the field content of the real-time handshake. Second, at the implementation level, it is written in Golang, leveraging Golang's superior thread scheduling mechanism to optimize the detection of multiple targets. Finally, at the mechanism level, a wildcard domain name mechanism is introduced, enabling the measurement of DoH service availability, authenticity, and dependency through a single active detection process.

[0008] To achieve the above objectives, the specific technical solution adopted by the present invention is as follows:

[0009] A fast DoH service measurement method based on multi-dimensional optimization, the steps of which include:

[0010] 1) Use a port scanning tool such as Zmap to scan port 443 in the target network to find hosts with open ports as probe targets.

[0011] 2) Apply for a public domain name (hereinafter referred to as the base domain name) from the Internet Corporation for Assigned Names and Numbers (ICANN), and select a suitable subdomain based on the domain name to determine the probe subdomain (hereinafter referred to as the probe base domain name). Assign a unique subdomain (hereinafter referred to as the unique subdomain name) to the probe target of the open port in step 1) based on the probe base domain name.

[0012] 3) Deploy the base domain name to a DNS authoritative server that you can control (hereinafter referred to as your own authoritative server), find the union of all unique subdomains, and configure wildcard subdomains in the Zonefile file of your own authoritative server based on the result.

[0013] 4) Use a probe tool to run on another server, different from the self-owned authoritative server mentioned in step 2), send a unique subdomain assigned to each host to the open port host in step 1, and parse the returned data packet as the return result.

[0014] 5) Collect the detection results of the detection tools in step 3) and the logs on our own authoritative server, and perform usability, dependency and authenticity analysis. The specific analysis method is as follows.

[0015] Furthermore, in the wildcard domain name configuration, a unique subdomain needs to be configured for each probe target; after the domain name is allocated, a relevant probe tool needs to be developed, and the probe target and the corresponding unique subdomain need to be input into the probe tool for active probe; during active probe, the output results of the probe tool and the log files on the own authoritative server need to be collected; after collecting the data from both aspects, the availability, correlation and dependency analysis are performed.

[0016] Furthermore, a unique subdomain is configured for each probe target through the following steps: the address of the probe target is converted into a string and concatenated to the probe base domain name, and the concatenated complete domain name is assigned as the complete domain name (FQDN) to the probe target.

[0017] Further, configure your own authoritative server using the following steps: Map all these subdomains to a wildcard domain name record. Specifically, take the union of all assigned unique subdomains to obtain a wildcard domain name that covers all unique subdomains. Configure the obtained wildcard domain name in the authoritative server's Zonefile and map it to a private address, such as 127.0.0.1. This wildcard domain name and the private address then constitute a valid resource record in the Zonefile.

[0018] Furthermore, an active detection tool is constructed through the following steps:

[0019] 1) Based on multiple strategies employed during the handshake process, each probe target is evaluated in real-time to determine whether it provides DoH services. These strategies are applied sequentially. If the target meets the criteria for not providing DoH services, the session with the current probe target is terminated immediately, and no further probes are initiated.

[0020] 2) Use Golang for multithreaded scheduling, establish a thread pool with a specified number of threads, and implement a thread observation mechanism based on thread execution time. If a probe thread executes for more than 10 seconds, it terminates and returns an empty record. If a thread completes execution within 10 seconds, it parses the received return data packet, generates resource records, and returns them to the main thread for statistical analysis.

[0021] The strategies of this proactive detection tool include, but are not limited to, the version of the Transport Layer Security (TLS) certificate used, the Application Layer Protocol Field (ALPN), the Service Identity Field (SNI), and whether it is a self-signed certificate.

[0022] Furthermore, the availability of the DoH service is measured through the following steps: First, check if the tool's probe results show a DNS response status of NOERROR. If such a response is received, it indicates that the service of the probe target is available. Second, check the request logs of the authoritative server to see if any client has requested a unique subdomain assigned to the target. The presence of such a probe domain also indicates that the DoH service of the probe target is available.

[0023] Furthermore, the authenticity of the DoH service is measured through the following steps: the determination is completed through the response of the active probing tool. First, the response of the target received by the active probing tool is parsed to obtain the resource record of the target. Then, the resource record is judged. If it is consistent with the resource record configured in the domain name management file of the authoritative server, it means that the DoH service of the target is real; otherwise, the DoH service of the target is hijacked.

[0024] Furthermore, the dependency on the DoH service is measured through the following steps: Determining whether the request is completed via the logs of the proprietary authoritative server. First, the requested domain name in the record entries on the proprietary authoritative server is broken down. Then, the IP address corresponding to the unique subdomain of the probe target is compared with the IP address of the request for that unique subdomain recorded on the proprietary authoritative server. If they match, it indicates that the probe target carries a direct DoH service; otherwise, it indicates that the probe target carries an indirect DoH server.

[0025] This invention also provides a fast DoH service measurement tool based on multi-dimensional optimization, characterized in that it includes an input parsing module, a thread scheduling module, a network connection module, a policy filtering module, and an information output module;

[0026] The input parsing module is used to segment the file containing the target IP address to obtain multiple IP addresses, then detect the format and validity of each IP address, and store the IP addresses that pass the detection as an array object and input it into the thread scheduling module.

[0027] The thread scheduling module is used to manage threads using the coroutine mechanism of the Golang language, including thread startup, thread addition, thread monitoring, thread communication, and thread termination; for each started thread, a timer is started when the thread starts, and the thread is automatically terminated after the thread has been executed for a specified time.

[0028] The network connection module is used to manage the transition from IP sessions to DoH sessions. It first establishes IP and TCP connections, and then the policy filtering module determines whether early termination is necessary. After policy filtering, it sends the assigned unique subdomain to the target and collects the target's response. Upon receiving the response, it sends the target's IP address and the response to the result collection thread.

[0029] The policy filtering module is used to make a real-time judgment on whether each probe target provides DoH service based on multiple policies in the protocol handshake process. Each policy is judged in sequence. When the judgment is that the target does not provide DoH service, the probe on the current target is terminated.

[0030] The information output module is used to parse the response results and identify the DoH service quality of each of the probe targets.

[0031] The beneficial effects of this invention are as follows:

[0032] This invention optimizes DoH service detection from multiple dimensions. Specifically, at the mechanism level, it utilizes a wildcard domain name mechanism to reduce the cumbersome backend configuration work; at the strategy level, it uses an active field filtering mechanism to reduce the number of probe data packets sent; and at the implementation level, it uses a fast timeout detection mechanism to reduce false delivery. This enables efficient detection of target addresses, especially when discovering a large number of DoH services, significantly reducing the detection time for a large number of host targets and reducing the number of interactive data packets, thereby alleviating the pressure on the probe targets and improving the probe's lifespan. Compared with existing service detection methods, this invention has the following advantages:

[0033] 1. During the overall detection process, the three detection tasks can be completed in one active detection process, reducing configuration work and overall data transmission volume.

[0034] 2. Improve the scheduling efficiency of multi-threading and reduce the time required for detection during an active probing process, making it suitable for fast, large-scale DoH service discovery tasks.

[0035] 3. Reduce the amount of data transmitted during a single active detection process, alleviate the pressure on the detection target, and increase the probe's survival time. Attached Figure Description

[0036] Figure 1 This is the overall flowchart of the present invention.

[0037] Figure 2 This is a flowchart of the domain name allocation process.

[0038] Figure 3 This is a diagram illustrating wildcard domain name configuration.

[0039] Figure 4 This is a flowchart of the strategy filtering process.

[0040] Figure 5 This is a flowchart of the results analysis.

[0041] Figure 6 This is a graph showing the effect of thread count, accuracy, and detection time when detecting 10,000 DoH hosts. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the objectives, features, and advantages of the present invention more apparent and understandable, the core technology of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] This invention presents an efficient DoH service measurement method. From an overall perspective, this method optimizes the detection process in three dimensions: detection mechanism, filtering strategy, and tool implementation. Specifically, the detection mechanism reduces the probe packets required for service detection by using wildcards to organize concise request domains; the filtering strategy excludes targets that are unlikely to provide DoH services by comprehensively considering fields from the TLS handshake negotiation process; and the tool implementation, written in Golang, improves thread scheduling efficiency through optimized exception handling. Based on the response data obtained by this method and log data collected from a proprietary authoritative server, the current service status of DoH can be quickly measured from three dimensions: availability, dependency, and correctness. Experimental results show that this method can achieve at least a 20-70 times performance improvement compared to existing DoH measurement methods.

[0044] The overall flowchart of the present invention is as follows: Figure 1 As shown, the specific steps of the method are detailed below:

[0045] (1) Perform a port scan on port 443 in the target list. The method in this invention requires first performing a port openness test on the target list to exclude targets that do not need further in-depth probing. For the probing tools, existing tools such as zmap and nmap can be used for efficient probing. For the probing packets, a TCP SYN packet is selected to initiate a request to port 443 of the target, and then the target's response packet is observed. Targets with closed ports are excluded based on the response packet.

[0046] (2) Apply for a public domain name and deploy it on your own authoritative name server, and enable access logs. Apply for a public top-level domain name from ICANN or other domain name management organizations. After completing the application, point the NS record of the top-level domain or the NS record of the next-level subdomain to the IP address of your own authoritative name server. Then, set up a DNS resolution service on your own authoritative name server and configure the relevant domain name management files so that the applied public domain name can be successfully resolved through a controllable server. Finally, configure the logging function of the DNS resolution service, specify the log output file, so that every request to access the domain name can be recorded.

[0047] (3) Assign unique subdomains and wildcard domain names to the targets of the open ports. First, assign a unique subdomain to each probe target. The assigned domain name is then populated into the DoH information during the subsequent probe process to distinguish different probe targets. Then, character pattern extraction is performed on all assigned unique subdomains to obtain a wildcard domain name that can cover all subdomains, and a non-public address, such as 127.0.0.1, is configured for this wildcard domain name to avoid impacting other public services. Finally, the wildcard domain name is written to the domain name management file (zonefile).

[0048] (4) Use a probe tool to probe the target based on the assigned unique subdomain. For the open port target obtained in process (1), use the probe tool to actively request the domain name based on the domain name assigned in process (3). During the active domain name request, filter whether the target is a DoH service step by step based on the handshake information in the interaction process. First, check the TLS version and filter targets lower than TLS 1.2; then filter the ALPN, providing two options, HTTP 1.1 and H2, during the sending process, and determine the protocol to be selected in the next stage based on the target's selection, so as to reduce repeated probing of the two options. Finally, judge based on the SNI field, filter targets that do not contain DOH, DNS, or non-self-signed certificates, and retain targets whose field is an IP address. For the filtered targets, send the unique subdomain assigned to the target in process (3) and collect response information.

[0049] (5) Collect the response results of the probe tools and the logs on the controllable server for data analysis. Based on the response of the active probe tools obtained in process (4) and the access logs obtained after configuration in process (2), analyze the availability, authenticity, and dependency of the DoH service. The availability of the DoH service is determined by whether the response result of the probe tools gives a NOERROR response and whether there is a unique subdomain query corresponding to the target in the log records. The authenticity of the DoH service is determined by comparing whether the response result of the probe tools is consistent with the resource records in the zone file. The dependency of the DoH service is determined by comparing whether the target corresponding to the requesting machine and the requesting domain name in the log records is consistent to determine whether the target is a direct resolver or an indirect resolver.

[0050] This invention also discloses a fast DoH service measurement tool based on multi-dimensional optimization, which is implemented entirely in the Golang language. The tool includes: an input parsing module, a thread scheduling module, a network connection module, a policy filtering module, and an information output module; wherein:

[0051] 1. Input Parsing Module. This module reads the target list. It takes a file containing the IP addresses of the targets as input. Key functions include splitting multiple IP addresses, checking the format of the IP addresses, and verifying their validity. After passing these checks, the input parsing module stores the IP addresses as an array and inputs it into other thread scheduling modules.

[0052] 2. Thread Scheduling Module. The thread scheduling module manages the thread pool. It utilizes Golang's coroutine mechanism for thread management, including thread startup, thread addition, thread monitoring, thread communication, and thread termination. Generally, thread types include input threads, output threads, result collection threads, and probe threads. For each started probe thread, a timer begins upon startup, automatically terminating the thread after a specified time has elapsed. Through thread pool management, the number of probe threads is limited; a new probe thread is only started after an existing one has finished executing.

[0053] 3. Network Connection Module. The network connection module establishes the basic network connection with the target. It manages the transition from IP sessions to DoH sessions. The network connection module and the policy module together constitute the main execution body of the probe thread. The network connection module first establishes IP and TCP connections, then the policy filtering module determines whether early termination is necessary. After policy filtering, it sends the assigned unique subdomain to the target and collects the target's response. Upon receiving the response, it sends the target's IP address and the response result to the result collection thread.

[0054] 4. Policy Filtering Module. The policy filtering module pre-judgments potential DoH targets. Targets that are unlikely to contain DoH services are terminated early, thereby improving overall detection efficiency and reducing network throughput. First, the policy filtering module judges the TLS version sent by the target during the first TLS handshake, filtering targets with versions lower than TLS 1.2. After passing the TLS version filtering, in the next handshake, the ALPN field is set to HTTP 1.1 and H2, and the HTTP version for the next step is determined based on the target's selection. Finally, the module filters by the SNI field sent by the peer. If the peer's SNI field contains DNS, DoH, or other DNS service-related strings, or an IP address, or if other fields of the certificate indicate that the certificate is self-signed, the session continues to the next step; otherwise, the target is filtered.

[0055] 5. Information Output Module. The information output module outputs the results of the probe thread to external storage as a file in real time. The module needs to parse the probe results according to the thread output, extract and integrate information such as response status, resource records, response time, and whether the probe is authoritative, and append the target's IP address before outputting it to the external file in real time.

[0056] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A fast DoH service measurement method based on multi-dimensional optimization, comprising the following steps: 1) Scan the target network to identify hosts with open ports as probe targets; 2) Apply for a public domain name from the Internet Corporation for Assigned Names and Numbers (ICANN) as a base domain name, and select a subdomain as a probe subdomain based on the base domain name; then assign a unique subdomain name to each probe target based on the probe subdomain name; The method for configuring a unique subdomain for the detection target is as follows: convert the address of each detection target into a string and concatenate it to the detection subdomain, and use the concatenated complete domain name as the unique subdomain of the corresponding detection target; 3) Select or deploy a DNS authoritative server in the DNS service system as your own authoritative server; deploy the base domain name on the own authoritative server, and configure a wildcard subdomain for each probe target in the Zonefile file of the own authoritative server based on each unique subdomain; the method for configuring the own authoritative server is as follows: find the union of the unique subdomains of each probe target to obtain a wildcard domain name that covers each unique subdomain; configure the wildcard subdomain in the Zonefile file of the own authoritative server, and map a private address to the wildcard subdomain; use the wildcard subdomain and the private address as a valid resource record in the Zonefile file; 4) Select or deploy another DNS server in the DNS service system and deploy a probe tool on it. The probe tool sends a unique subdomain assigned to each probe target and collects the returned results. The method for constructing the probe tool is as follows: 41) Select multiple strategies based on the protocol handshake link in the handshake process as the filtering strategies of the probe tool, which are used to make real-time judgments on whether each probe target provides DoH service. The strategies are judged sequentially in serial order. When it is judged that the target does not provide DoH service, the probe on the current probe target is terminated; 42) Establish a thread pool containing a specified number of threads and a multi-thread scheduling mechanism, and establish a thread observation mechanism based on the thread execution time. When a thread executes for more than a set time, the execution of the thread is terminated. Each thread is used to send a unique subdomain assigned to a probe target and collect the returned results. 5) Based on the returned results collected in step 4) and the logs on the proprietary authoritative server, identify the DoH service quality of each probe target, including the availability, authenticity and dependency of the DoH service.

2. The method according to claim 1, characterized in that, The method for identifying the availability of the DoH service is as follows: determine whether the return result of each probe target A contains a message with a DNS response status of NOERROR. If so, the DoH service of the probe target A is determined to be available; or determine whether there is a user requesting a probe subdomain assigned to the probe target A in the request log of the proprietary authoritative server. If the probe subdomain exists, the DoH service of the probe target A is determined to be available.

3. The method according to claim 1, characterized in that, The method for identifying the authenticity of a DoH service is as follows: First, the return result of each probe target A is parsed to obtain the resource record that requested the probe target A. Then, the resource record is judged. If it is consistent with the resource record configured in the domain name management file of the proprietary authoritative server, the DoH service of the probe target A is judged to be genuine; otherwise, the DoH service of the probe target A is hijacked.

4. The method according to claim 1, characterized in that, The method for identifying the dependency of DoH service is as follows: First, the request domain name in the record entry of the proprietary authoritative server is broken down. Then, the IP address corresponding to the unique subdomain of the probe target A is compared with the IP address of the request for the unique subdomain recorded in the backend. If they match, it is determined that the probe target A carries a direct DoH service; otherwise, it is an indirect DoH server.

5. The method according to claim 1, characterized in that, The detection tool includes a detection target and its corresponding unique subdomain, which is used to actively detect the detection target.

6. The method according to claim 1, characterized in that, The strategy includes the version of the transport layer security protocol used, the application layer protocol field, the service identity identification field, and whether it is a self-signed certificate.

Images