A network domain name and address relationship information capturer

Abstract

The application discloses a network domain name and address relationship information catcher, comprising a filtering module, a processing module, a storage module and an output module; the filtering module is used for receiving all uplink and downlink data packets of the mirror interface through an interface L, filtering out DNS data packets and delivering the DNS data packets to the processing module; the processing module is used for processing the screened DNS data packets and generating an R table; the storage module is used for storing the R table and generating an S table based on the R table, wherein the S table is a table recording domain name and network IP address relationship information; and the output module is used for receiving domain name query requests of external subscribers through an interface U and returning query results to the external subscribers according to the S table. The application realizes the establishment of relatively real-time domain name and network IP address relationship information with low investment, low risk and low time delay.

Description

Technical Field

[0001] This invention belongs to the field of computer communication and communication security, and in particular relates to a capturer of network domain name and address relationship information. Background Technology

[0002] In computer networks based on TCP / IP address protocols, when a host accesses a target service provider (server) via a domain name, it needs to obtain the target service provider's (server's) network IP address through domain name translation. Currently, there are several main methods for hosts to obtain network IP addresses: First, the host locally stores a cache of network IP addresses for previously accessed domain names. For subsequent accesses, the host can directly obtain the network IP address from its local cache, without needing to comply with external protocols. Second, the host initiates a domain name resolution request to a pre-installed local DNS server (or proxy server), which provides the target service provider's (server's) network IP address; this process follows the DNS protocol. Third, the host initiates a domain name query request to a pre-installed third-party server using a non-DNS protocol. This process does not involve the DNS protocol but must comply with other protocols, typically HTTPS / SSL.

[0003] Based on network security policies, information security precautions, and other traffic labeling or filtering requirements, Internet Service Providers (ISPs) or enterprise network management departments need to identify, record, and update the domain name and network IP address relationships of traffic flowing through their forwarding nodes. Typical requirements include: 1. Public regulatory agencies require blocking access to specific domain names; 2. Internal or external information security monitoring agencies within an organization identify domain name security threats and require restrictions on specific domain names; 3. Users need to implement blacklist and whitelist control for domain names accessed by hosts.

[0004] For ISPs, existing technologies offer the following solutions for the aforementioned identification, recording, and updating:

[0005] First, ISPs typically provide managed DNS servers to hosts accessing their networks, using other protocols to announce the DNS server addresses during the initial access phase. For hosts using DNS provided by the ISP network, the specific network IP addresses of the target service providers (servers) they access are provided by the ISP's DNS; therefore, the relationship between the domain name and the network IP address can be obtained by the ISP.

[0006] Second, the gateway for a host to access the internet is usually on the ISP's network element. The ISP can provide a proxy for the DNS service on the gateway, forcing the DNS resolution requests of the host to non-ISPs to be processed by the proxy service of the ISP gateway. Similar to the first scheme, the ISP can also obtain the relationship between domain name and network IP address.

[0007] Third, since all uplink and downlink traffic of a host must pass through the ISP's forwarding network, theoretically, the ISP can configure relevant functions on devices at specific nodes of the network to capture and process raw network IP address data packets and obtain the domain name and network IP address relationship from the network layer.

[0008] For ISPs, solutions for identifying, recording, and updating the relationship between domain names and network IP addresses in their network traffic must consider the following key challenges: First, given the massive network traffic of ISPs, implementing the solution may incur additional performance overhead for critical network elements, negatively impacting the return on network investment. Second, compared to smaller networks, ISP networks are more complex and less flexible, making it difficult to replace or modify their expensive network element equipment according to specific solution requirements. Furthermore, the solution must consider the differences in management standards implemented by different network element management departments within the ISP (e.g., gateways and DNS are managed by two separate departments). When multiple departments are involved, conflicts between standards may occur, making the solution difficult to implement. Finally, the solution must consider the real-time nature of the domain name and network IP address relationship; introducing excessive processing latency will weaken the effectiveness of the solution.

[0009] Specifically, the existing technical solutions have the following corresponding defects:

[0010] The main drawback of the first approach is that the host must use the DNS declared by the ISP. Otherwise, the ISP's DNS can only update the domain name and IP address relationship based on data synchronized with other DNS servers. However, in real-world networks, the synchronization latency between DNS servers is typically measured in hours or days. This approach suffers from poor real-time performance and its accuracy is affected by other non-ISP DNS servers. Furthermore, according to existing network access protocols, the host can use non-ISP declared DNS servers, thus limiting the applicability of this approach. It's also worth noting that the ISP's DNS management department may be unable to provide non-DNS protocol standard offline domain name and IP address relationship information to other departments due to its own regulatory limitations. This could create difficulties for the department implementing the solution in managing domain name and IP address relationships later on.

[0011] For the second solution, although the ISP's gateway can force the round-trip packets between the host and the non-ISP DNS to be processed by the local proxy through proxy DNS, it has three drawbacks: First, proxy DNS obviously incurs additional performance overhead for the gateway device; second, proxy may lead to a decrease in the success rate of DNS resolution and may also introduce additional resolution latency, affecting the access quality of the host; finally, the existing gateway device must not only support the proxy function, but also support the processing and output of the relationship between domain name and network IP address. If the existing gateway device does not support this or has insufficient resources, then this solution is also unsuitable.

[0012] For the third option, although it is not bound by the DNS protocol, the network elements must support the configuration of related functions, including not only the configuration of relationship identification, recording, and updating, but also the configuration of relationship information processing and output. Furthermore, these functions also require hardware, software, and technical support. Since ISPs mostly purchase equipment from different communication equipment vendors, replacing or changing highly complex, inflexible, and expensive network elements will bring significant cost and change risks to ISPs. Summary of the Invention

[0013] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a network domain name and address relationship information capture device. This device solves the problem of how to establish relatively real-time domain name and network IP address relationship information with low investment, low risk, and low latency in high-traffic forwarding networks (typically ISPs), especially when third-party DNS servers or host-configured DNS servers are uncontrollable, existing network element hardware and software are difficult to change, and cross-supplier or cross-department technology cannot be coordinated and supported.

[0014] The value of the domain name and network IP address relationship information lies in its ability to be used to mark, filter, and intercept data packets between hosts and domain names on any forwarding network element, thereby meeting network security policies, information security precautions, and further developing traffic marking or filtering services.

[0015] To solve the above technical problems, the technical solution adopted by the present invention is: a network domain name and address relationship information capture device, including a filter module, a processor module, a storage module (Cache) and an output module (Exporter).

[0016] The filter module is equipped with an interface L, and the exporter module is equipped with an interface U.

[0017] The interface L is interconnected with the mirror interface on the gateway device;

[0018] The filtering module is used to receive all uplink and downlink data packets of the mirror interface through interface L, filter out DNS data packets, and hand them over to the processing module (Processor).

[0019] The processing module (Processor) is used to process the filtered DNS packets and generate an R table. The processing includes the following steps: extracting the summary information of each DNS packet, adding a timestamp, and recording it in the R table (R table full name: DNS relative list).

[0020] The storage module (Cache) is used to store the R table and generate the S table based on the R table. The S table is a table that records domain name and address relationship information.

[0021] The output module (Exporter) is used to receive domain query requests from external subscribers through interface U, and return the query results to the external subscribers according to table S.

[0022] The aforementioned capture device, the storage module (Cache), is also used to generate an Rb table based on an R table, including the following steps: selecting records in each row of the R table with a QR value of 1, then extracting the Name and Address relationship in the Answers field of the record, adding the current timestamp, and entering it into the Rb table;

[0023] If the domain query request received by the output module (Exporter) is a quick query and there is no corresponding query result in the current S table, then the query result is returned by matching the contents of the Rb table with the domain to be queried; a quick query means that the query only needs to return the relationship between the domain and the network IP address;

[0024] If the domain query request received by the output module (Exporter) is a detailed query and there is no corresponding query result in the current S table, then the query result is returned by matching the contents of the R table with the domain to be queried; a detailed query means that the query should return the relationship between the domain and the network IP address and detailed information.

[0025] In the aforementioned capture mechanism, the S-table serves as a carrier for information exchange between the output module (Exporter) and the storage module (Cache).

[0026] When the exporter receives a domain query request, if there is no corresponding query result in the current S table, it updates the domain in the S table and sends the updated S table to the storage module (Cache).

[0027] After receiving the S-table from the exporter, the storage module queries the domains newly updated in the S-table, saves the corresponding query results to the S-table, and returns the updated S-table to the exporter.

[0028] The aforementioned catcher, specifically the Processor, processes the filtered DNS packets to generate P and A tables. This processing includes the following steps:

[0029] Based on the value of the QR field in the flag bit of the DNS packet, the IP address, protocol, and port of the packet with a QR value of 0 are timestamped and recorded in the P table (full name of the P table: DNS Potential list); the IP address, protocol, and port of the packet with a QR value of 1 are timestamped and recorded in the A table (full name of the A table: DNS active list).

[0030] The aforementioned capture device also includes a test module (Collector), which is configured with an interface M. The interface L is interconnected with the interface M on the test module (Collector), and the traffic of the interface M is mirrored on the interface L. The interface M is an interface that can communicate with the Internet.

[0031] The test module is used to perform the following steps:

[0032] The first step is for the Collector to receive test instructions from the Processor and determine the source network IP address based on those instructions.

[0033] The second step is that the test module (Collector) obtains the testable DNS (ip+protocol+port) through the P table and A table, and obtains the domain name to be queried through the S table. The domain name to be queried is the domain name for which no corresponding result was found based on the R table.

[0034] The third step involves the test module (Collector) constructing a DNS request packet using the source network IP address, the DNS in table A, and the domain name to be queried in table S, and then accessing the DNS server.

[0035] The fourth step involves the test module (Collector) constructing a DNS request packet using the source network IP address, the DNS in the P table, and the domain name to be queried in the S table, and then accessing the DNS server.

[0036] Fifth, if the DNS server does not find any record for the domain, it considers the domain provided by the subscriber to be invalid, and the test module (Collector) supplements the DNS response with its own source network IP address;

[0037] When the exporter receives the S-table from the storage module, it checks the S-table. If the network IP address corresponding to a domain in the S-table is the source network IP address configured on the M-port by the catcher, then the domain is marked as an invalid domain.

[0038] The aforementioned catcher, the filter module (Filter), receives and sets multiple queues to receive DNS packets in a loop. The maximum number of packets that each queue can receive is adjustable. The specific looping method is as follows: each queue is pre-assigned an ordered sequence number. When the first queue is full, the filter module (Filter) automatically uses the next queue to continue receiving. When the last queue starts receiving, the first queue is immediately cleared and its sequence number is set to the last sequence number. At the same time, the sequence numbers of other queues are shifted forward by one position.

[0039] The aforementioned capture device, wherein the processing module is configured with a periodic timer T0 that starts when the capture device is powered on. When timer T0 expires, it retrieves the data packets in the first sequence of the current sequence from the filtering module for processing.

[0040] The aforementioned capture device, specifically the exporter module, is further configured to perform a validity check on the domain query requests from external subscribers, determining whether the subscriber's domain query request conforms to the following rules:

[0041] Rule 1: The number of domain entries queried in a single query cannot exceed the preset limit of the output module (Exporter);

[0042] Rule 2: Domain name (domain) string format, case sensitive, length range 1 to 255;

[0043] Rule 3: Domain names must not contain the following characters: "@", "\", ":", "*", """, "<", ">", "|", " / ", "#", "=", "'", and "&".

[0044] Rule 4: Domain names are allowed to begin with the wildcard character "**".

[0045] Rule 5: Domains are allowed to have 2-4 levels.

[0046] Rule 6: The period (.) symbol cannot be consecutive or appear at the end.

[0047] In the aforementioned capture device, the filtering module receives filtering rules from the processing module and filters out DNS data packets according to the filtering rules.

[0048] In the above-mentioned capture device, the filtering module obtains the filtering ratio r from the processing module of the capture device. If the processing module does not send the filtering ratio r, the filtering module receives all packets at a 1:1 ratio by default. Otherwise, the filtering module receives and samples uplink and downlink data packets in a manner that one packet is taken for every r packets.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] First, the capture device described in this invention, based on the collection and subsequent processing and calculation of network layer data packets, does not require the host to use a specific DNS, nor does it rely on operations on a specific DNS. The data packets collected by the capture device come from real-time traffic flowing through the gateway. Combined with the special processing methods of this invention, it can significantly reduce the update latency of domain name and network IP address relationships while ensuring the accuracy of the information. In addition, the capture device does not need to interoperate with network elements of other nodes except for obtaining mirrored traffic from the gateway, thus avoiding the problem of technical coordination between nodes.

[0051] Secondly, the relationship capture device described in this invention can be deployed alongside the ISP gateway device, using existing and well-established traffic mirroring technology to introduce the gateway's DNS packets into the capture device for processing. This is completely decoupled from the functionality of the existing network, avoiding the risks associated with existing technologies. Therefore, the capture device's operation does not depend on changes to the gateway device's forwarding method, does not significantly increase the gateway device's performance overhead, does not introduce additional DNS resolution latency, and has no impact on the network access of existing hosts.

[0052] Third, the traffic mirroring technology is widely supported by current gateway devices, especially ISP-level devices. Furthermore, no other functions are required from the device, and no hardware or software changes are needed. Therefore, the relationship capture device described in this invention can be deployed alongside most network elements, without considering the device manufacturer or requiring special functions, making it widely applicable and requiring low investment.

[0053] In summary, the beneficial effects of the present invention are as follows:

[0054] This invention can efficiently acquire and update domain name and IP address relationships without relying on DNS control, making it widely applicable. The solution uses a special processing method to control the update latency of these relationships to within seconds, a significant advantage compared to the hourly or day-based synchronization latency of existing DNS systems. The solution considers both existing domain name and IP address relationships and the query needs of subscribers, greatly reducing the computational requirements of the module through filtering, and establishing more comprehensive and accurate relationship data.

[0055] This invention only interfaces with network elements acting as gateways, without involving other network elements. Therefore, it eliminates cross-departmental management constraints and coordination difficulties, offering better usability compared to existing technologies. The traffic mirroring function required for the gateways connected to this invention is a mature existing technology, widely supported by various types of equipment from different manufacturers. The use of this function incurs minimal overhead on device performance, thus eliminating the need to replace or modify existing equipment, further expanding its applicability compared to existing technologies. This invention operates independently in parallel with the network management system, without interfering with the normal traffic forwarding of the connected gateway, without introducing outward latency, and without affecting the access quality of hosts. Compared to existing technologies, it significantly reduces the risk of changes to the existing network operation. This invention can autonomously capture, process, exchange, and store data, and provides a proprietary API interface to the outside world. The input and output processes do not rely on existing network equipment hardware or software support, nor do they require technical support. The performance design parameters of the capture device can be limited according to the traffic volume, resulting in lower cost and risk compared to existing technologies.

[0056] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0057] Figure 1 This is a diagram of the architecture of the present invention.

[0058] Figure 2 This is an example format for the R table of this invention.

[0059] Figure 3 This is a diagram illustrating the process of the Exporter module querying the S-table in this invention. Detailed Implementation

[0060] This invention is applicable to ISPs or enterprise network management departments because the ISP scenario is more typical than that of enterprise networks and usually covers the key network elements that may exist in enterprise networks. Therefore, this application describes the ISP scenario in this context.

[0061] like Figure 1As shown, a collector includes a filter module, a processor module, a cache module, an exporter module, and a collector module.

[0062] The filter module is equipped with interface L, the exporter module is equipped with interface U, and the collector module is equipped with interface M.

[0063] In this embodiment, the interface L of the filter module is interconnected with the mirror interface on the gateway device, and the interface L is also interconnected with the interface M on the collector module.

[0064] The filtering module is used to receive all uplink and downlink data packets about hosts on the mirror interface of the gateway device through interface L. Specifically, taking Internet Service Provider (ISP) as an example, the port traffic mirroring function is configured on the ISP's gateway device to copy all host traffic passing through the gateway device to another interface. The interface L for monitoring the network is configured to be directly connected to the gateway and receive all uplink and downlink data packets about hosts on the gateway.

[0065] Based on the filtering rules received from the Processor module, DNS packets are then filtered out from the uplink and downlink packets and handed over to the Processor module.

[0066] DNS packets are filtered out using the following methods: Because the DNS protocol uses a standardized structure and port number, the filtering module (Filter) filters packets by default based on the following characteristics:

[0067] Feature 1: The Protocol bit in the network IP address header of the packet is either UDP or TCP.

[0068] Feature 2: The source or destination port number of the packet's UDP header is 53;

[0069] Feature 3: The QR field of the flag bit in the packet is 0 or 1;

[0070] The filtering module temporarily retains packets that match the characteristics in the receiving queue and immediately discards packets that do not match the characteristics;

[0071] The filtering module supports obtaining the filtering ratio r and other filtering rules from the processor module of the capture unit. For the filtering ratio r, if the processor module does not send the filtering ratio r, the filtering module receives all packets at a 1:1 ratio by default. Otherwise, the filtering module receives and samples uplink and downlink packets one packet for every r packets. For other filtering rules, if the processor module does not send them, the filtering module uses the default rules mentioned above. Otherwise, in addition to the default rules, it also uses the sent rules for filtering again.

[0072] To receive packets matching the characteristics in a timely manner, the filtering module uses its own dedicated memory medium that supports high-speed read and write operations. It sets up multiple queues to receive packets in a loop. To balance the packet loss rate and receiving efficiency based on the number of packets, the maximum number of packets that each queue can receive is adjustable. The specific looping method is to pre-assign an ordered sequence number to each queue. When the first queue is full, the filtering module automatically uses the next queue to continue receiving. When the last queue starts receiving, the first queue is immediately cleared and its sequence number is set to the last sequence number. At the same time, the sequence numbers of other queues are shifted forward by one position.

[0073] In this embodiment, the processing module (Processor) is used to provide filtering rules to the filtering module (Filter), to provide test instructions to the testing module (Collector), and to provide P-table, A-table, and R-table to the storage module (Cache). The P-table, A-table, and R-table are obtained by the following method: after the filtering module (Filter) obtains the filtered packets, the processing module (Processor) needs to extract information from the data packets periodically, and then create the above-mentioned P-table, A-table, and R-table.

[0074] The filtered DNS packets are processed to obtain the P, A, and R tables. The processing includes the following steps:

[0075] Step A: Based on the value of the QR field in the flag bit of the DNS packet, timestamp the IP address, protocol, and port of the packets with a QR value of 0 and record them in the P table (full name of P table: DNS Potential list); timestamp the IP address, protocol, and port of the packets with a QR value of 1 and record them in the A table (full name of A table: DNS active list).

[0076] It should be noted that, according to the DNS protocol standard, a QR value of 0 represents a request, and a QR value of 1 represents a response. The purpose of classifying the P table and the A table is to distinguish between the DNS in the request packet sent by the host and the DNS in the response packet. This is because the DNS in the host's request may not be valid due to selection bias, while the DNS in the response is mostly valid. Therefore, theoretically, the A table is a subset of the P table.

[0077] Step B: Extract the summary information of each DNS packet, timestamp it, and record it in an R table (R table stands for DNSrelative list); the specific format is as follows: Figure 2 As shown;

[0078] It should be noted that the DNS Flag, DSN Queries, and DNSAnswer fields contain multiple subfields, including QR, Name, and Answers.

[0079] In practice, the processing module (Processor) starts an adjustable periodic timer T0 when the capture device is powered on. When the timer expires, it retrieves the data packet from the first sequence of the current sequence from the filtering module (Filter) and processes it according to steps A and B. The processed result is then handed over to the storage module (Cache) for management in the form of a table.

[0080] In this embodiment, the storage module (Cache) is used to manage tables P, A, R, and S, including providing storage space for each table, interacting with tables in other modules, and updating and aging tables.

[0081] Specifically, the storage module (Cache) is used to provide updates to the S table for the output module (Exporter) and to provide the P table, A table, and S table for the test module (Collector);

[0082] The storage module is also used to generate an Rb table from an R table, including the following steps:

[0083] Select the records in table R with a QR value of 1, then extract the Name and Address relationship in the Answers field of the record, add the current timestamp, and enter it into table Rb.

[0084] In practice, the storage module (Cache) obtains three tables (P table, A table, and R table) from the processing module (Processor), stores them in the database, performs subsequent processing, and then manages their validity. The specific steps are as follows:

[0085] First, because the R table has many fields, and in most cases the searcher may not need all the field information, too many fields are obviously detrimental to fast searching in these situations. Therefore, the R table is processed a second time to save a new abbreviated table Rb. Any operation searching the R table always first searches the Rb table. Only when the Rb table returns insufficient fields will the R table be searched second. This ensures search efficiency in most cases while providing complete field information for other searches.

[0086] The Rb table is generated by selecting records from the R table with a QR value of 1, extracting the Name and Address relationship from the Answers field of each record, adding the current timestamp, and then entering it into the Rb table. Specifically, a QR value of 1 indicates that the record is a DNS Response message. A DNS Response message normally contains an Answers field, which normally contains the domain name and its network IP address—the network IP address obtained by the host when resolving the domain name. Other abnormal cases, including returning a CNAME (Canonical Name, used in the DNS protocol to map one domain name to another), will not return the final network IP address of the domain name. Whether it's an abnormal packet or a record requiring recursive queries on the CNAME (where the CNAME mapping chain can be long, and recursion yields the final network IP address), these records will be filtered out by this method, ensuring the effectiveness of the filtering results. Abnormal packets and CNAME cases are entirely handled by the host or DNS server; the storage module (Cache) does not need to consider them.

[0087] The storage module (Cache) needs to manage the timeliness of records in the four tables (P table, A table, R table, and Rb table), and delete records that have not been updated for a long time to maintain a reasonable table size. This helps to reduce lookup latency and also helps to delete potentially expired domain name and network IP address relationship records in a timely manner, thereby improving the accuracy of the tables.

[0088] The storage module (Cache) pre-allocates a fixed amount of storage space for each table. Upon startup, it starts a periodic timer T1. When T1 expires, the storage module (Cache) iterates through each row of records in each table. As mentioned earlier, all records have timestamps. After the iteration is complete, the storage module (Cache) obtains the latest update time of all records, thus determining how long these records have remained unchanged. This time measure is represented by an integer multiple of T1, N0. When N0 exceeds a threshold, the record is deleted. The N0 value for each table is set independently and does not affect other tables.

[0089] In this embodiment, the export module is used to provide API communication to external subscribers through interface U. The export module is responsible for receiving API requests, maintaining the update and aging of the local S table, querying the local S table or sending an update to the storage module (Cache) based on the API request, checking the S table, marking invalid domains, and finally packaging the response information into an API message and returning it to the subscriber.

[0090] Specifically, interface U is the communication interface between the catcher and external subscribers. Subscribers are those objects that need to query the catcher for information on the relationship between domain names and network IP addresses.

[0091] The Exporter module is primarily responsible for providing subscribers with an API (Application Programming Interface) for inputting and querying domain records (a list of at least one domain). It also handles API transactions in a standard way for computer networks. Since interacting with data via API is existing technology, it will not be elaborated on here.

[0092] Next, the exporter module performs a validity check on the domain records provided by the subscriber, according to the following rules:

[0093] Rule 1: The number of domain entries queried in a single query cannot exceed the preset limit of the output module (Exporter);

[0094] Rule 2: Domain name (domain) string format, case sensitive, length range 1 to 255;

[0095] Rule 3: The following characters are not supported: "@", "\", ":", "*", """, "<", ">", "|", " / ", "#", "=", "'", and "&".

[0096] Rule 4: Wildcards "**" are allowed at the beginning;

[0097] Rule 5: Domains are allowed to have 2-4 levels.

[0098] Rule 6: The period (.) symbol cannot be consecutive or appear at the end.

[0099] The domain records checked by the exporter module are timestamped before being entered into the query process.

[0100] like Figure 3 As shown, the query process has two steps:

[0101] The first step is for the exporter to query its current S-table (S-table full name: DNS Subscriber list), which is obtained from the storage module (Cache). Furthermore, if the network IP addresses of all domains have been found, the exporter will directly package them into an API response message and return it to the subscribers, thus ending the query.

[0102] The second step is to create or update the S table of the exporter if the S table is empty, or if the previous query found that at least one domain did not find any results. Then the exporter will proceed to the next step of querying the cache module until the cache module returns the results and then package the API message and return it to the subscriber. Only then will the query end.

[0103] Regarding table S, the following should be noted:

[0104] 1. The S-table is the carrier of information exchanged between the output module (Exporter) and the storage module (Cache);

[0105] 2. The export module (Exporter) obtains the latest domain:IP retrieval information from the S table of the storage module (Cache), and the storage module (Cache) obtains the domains that subscribers are interested in from the S table of the export module (Exporter).

[0106] 3. The storage module (Cache) will search on its own, mainly from the R and Rb tables, because these two tables contain captured domain:IP relationships. Based on the domain in the S table, a portion of the current network can be retrieved. If no current network is retrieved, the test process will be triggered.

[0107] 4. Both the exporter and the cache maintain their own copy of the S table and update it according to their own methods. The exporter can update the S table because it receives queries from subscribers, and it can also receive and update the S table from the cache.

[0108] The domain:ip relationship is in the last field of the data recorded in the R and Rb tables. The domain:ip relationship exists as a long string in the last DNS Answer. Because a domain may have multiple IPs, its length is not fixed. This is a convention of the DNS standard protocol and is widely known, so it will not be elaborated here.

[0109] A subscriber's DNS request instructs them to query the IP address of a domain. In this case, the domain is the key for the search, and the content in the DNS answer is the value.

[0110] Regarding the second step mentioned above, the specific operation involves the exporter handing over its S-table to the cache for querying. To facilitate understanding the purpose of the S-table, it's important to clarify that the S-table is the standard message format for communication between the exporter and the cache. Each module maintains a copy, managed by its own management and updated through exchange. The exporter can perform two types of queries: first, a quick query, which only returns the domain name and IP address relationship without detailed information; second, a detailed query, which returns both the domain name and IP address relationship and detailed information. The specific method used is entirely determined by an option in the request sent by the subscriber via the API (API conventions are not key to this invention), and the exporter executes accordingly.

[0111] When the catcher boots up, the storage module (Cache) allocates a dedicated storage space for processing the S table. In addition, the storage module (Cache) starts a monitoring process for the S table. Whenever the Rb table or R table is updated, the storage module (Cache) immediately prompts the S table to match and complete the update.

[0112] When the storage module (Cache) receives a new S table from the exporter, it assumes there is a query task. The storage module (Cache) first compares the received S table content with its locally maintained S table content. If the received S table contains domains that it doesn't already have, it updates its own table accordingly; otherwise, it ignores them. This step ensures that the S table contains as many domains as possible that subscribers are interested in. Next, the storage module (Cache) determines the type of query. If it's a quick query, the S table is updated and immediately matched against the Rb table; if it's a detailed query, it matches against the R table. The distinction between the Rb and R tables is to improve query efficiency.

[0113] If at least one domain cannot be found during the above process, it indicates that either these domains have not appeared in the traffic of the gateway host before, or these domains are invalid. In this case, the storage module (Cache) will not immediately send the updated S table to the output module (Exporter). Instead, it will first send the current P table, A table, and S table to the collector's test module (Collector), and then enter a waiting process. The storage module (Cache) will only send the updated S table to the output module (Exporter) after at least one round of S table update actions.

[0114] As described above regarding the generation process of table R, table S will only be updated after at least one time T0. Therefore, for subscribers, the two actions of requesting and returning data to the output module (Exporter) via API can be handled asynchronously. This asynchronous approach means that after the requester initiates a request, it does not need to wait for the API query to complete, but can continue to perform other operations. When the API query is completed, the server can notify the requester that the result is ready in some way, i.e., allow waiting for the update.

[0115] In this embodiment, the test module is used to receive test control instructions issued by the processing module (Processor), and the test module (Collector) initiates a response test according to the instructions; initiating a response test means that the test module (Collector) uses interface M to initiate a test, so interface M must be an interface that can communicate with the Internet, and in the capture device of this invention, the traffic of interface M is mirrored to interface L.

[0116] The principle of the test: The test module uses the DNS (ip+protocol+port) records in the P table and A table to actively initiate a domain query for a specific domain, obtain the upstream and downstream DNS data packets locally, and then introduce the DNS data packets to the interface L of the filtering module for capture.

[0117] Test parameters include: source network IP address, start signal, end signal, number of attempts, time interval, and timeout.

[0118] The testing method is as follows:

[0119] The first step is that when the test module (Collector) starts, it has already received the test instructions from the processing module (Processor). Based on the instructions, it determines the source network IP address. The source network IP address refers to at least one network IP address configured in the test module (Collector) by the capture device. These are the source addresses used to initiate the test DNS.

[0120] The second step involves the test module (Collector) obtaining testable DNS (ip+protocol+port) through the P and A tables, and then obtaining all the domains that subscribers are interested in through the S table. The test module (Collector) then filters out the domains that do not have corresponding results based on the R table from the S table as the domains to be queried. Some of these domains may be invalid.

[0121] The third step involves the test module (Collector) constructing a DNS request packet using the source network IP address, the DNS in table A, and the domain in table S to access the DNS server.

[0122] The fourth step involves the test module (Collector) constructing a DNS request packet using the source network IP address, the DNS in the P table, and the domain name in the S table to access the DNS server.

[0123] Fifth, if the DNS server does not find any record for the domain, it considers the domain provided by the subscriber to be invalid, and the test module (Collector) then adds its own source network IP address to the DNS response packet;

[0124] It should be noted that DNS stands for Domain Name System. DNS refers to the IP address of the server that provides domain name translation. IP + protocol + port refers to the IP address, protocol, and port number of the DNS server, typically 8.8.8.8, udp, 53. The above test process is to generate DNS requests and responses for the domain name to be tested.

[0125] The test parameters in the above steps are adjustable to ensure the reliability of the test. The test steps and methods are unique to this invention, while parameter control is easy to implement. They are only used to illustrate the rationality of the test and will not be described in detail. It should be noted that the third and fourth steps are two types of tests and can be executed with different command parameters. The reason for executing the third step first is to allow the A table to complete the test first. Then the DNS response will reach the capture device first. In step 3, it can be seen from its type that the DNS in the A table all have response packets. Allowing this part of the test to be completed first helps to improve efficiency. In contrast, the DNS in the P table is only a request and may not be available. It is only used as a supplement to the A table, so it is executed later.

[0126] Since the M port has mirrored the traffic to the L port, the filter module can also use the method described above to filter out the DNS uplink and downlink packets of the M port and send them to the processor module. Then, according to the process described above, after a period of time, the S table of the storage module (Cache) will be updated through the Rb table and the R table, and then the update will be sent to the exporter module.

[0127] In this embodiment, when the exporter receives an update, it immediately saves and checks the S table. If the network IP address corresponding to the domain in the S table is the test source address configured by the catcher on the M port, the record is marked as an invalid domain. Then, a timer T2 is started for the S table. At the same time, the relationship between the domain and network IP address in this query, as well as other detailed information (if this query is a detailed query), are packaged into an API response message and returned to the subscriber through the U port.

[0128] Upon expiration of time T2, the Exporter clears its currently stored S-table and destroys timer T2. This timer allows the S-table sent by the Cache module to be temporarily retained in the Exporter for a period of time, in case subscribers initiate multiple queries within a short period. Since the Exporter can directly process some queries using its own S-table, this reduces the processing overhead and latency of other modules in the catcher system. Correspondingly, the purpose of the Exporter clearing the S-table upon T2's expiration is to ensure that the S-table's validity only lasts for T2. When a subscriber's request includes a previously unseen domain query, the Exporter can proactively trigger an update process involving the Cache module and other modules.

[0129] The storage module (Cache) also relies on the S table of the export module (Exporter). Without the S table, it can only obtain data from packets in the live network and cannot update the data based on subscriber queries. This is because the domain name and network IP address relationship data captured in the live network cannot always completely cover the domain name queried by the subscriber. This is because some domain names may not have any traffic activity in the live network, so it is necessary to actively query to obtain the domain name and network IP address relationship data.

[0130] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A capturer of network domain name and address relationship information, characterized in that, It includes a filtering module, a processing module, a storage module, and an output module; The filtering module is equipped with interface L, and the output module is equipped with interface U; The interface L is interconnected with the mirror interface on the gateway device; The filtering module is used to receive all uplink and downlink data packets of the mirror interface through interface L, filter out DNS data packets, and hand them over to the processing module. The processing module is used to process the filtered DNS packets and generate an R table, which stands for DNS relative list. The processing includes the following steps: extracting the summary information of each DNS packet, which includes information from the Flag, Queries and Answers fields of the DNS packet, and recording it in the R table with a timestamp. The storage module is used to store the R table and generate the S table based on the R table. The S table is a table that records the relationship information between domain names and network IP addresses. The output module is used to receive domain name query requests from external subscribers through interface U, and return query results to external subscribers according to table S; The filtering module receives filtering rules from the processing module and filters out DNS data packets according to the filtering rules.

2. A network domain name and address relationship information capturer according to claim 1, characterized in that, The storage module is also used to generate an Rb table based on an R table, including the following steps: selecting records in each row of the R table with a QR value of 1, then extracting the Name and Address relationship in the Answers field of the record, adding the current timestamp, and entering it into the Rb table; If the domain name query request received by the output module is a quick query and there is no corresponding query result in the current S table, then the query result is returned by matching the contents of the Rb table with the domain name to be queried; a quick query means that the query only needs to return the relationship between the domain name and the network IP address; If the domain name query request received by the output module is a detailed query and there is no corresponding query result in the current S table, then the query result is returned based on the content in the R table matched with the domain name to be queried; a detailed query means that the query should return the relationship between the domain name and the network IP address and detailed information.

3. A network domain name and address relationship information capturer according to claim 1, characterized in that, The S-table serves as a carrier for information exchange between the output module and the storage module. When the output module receives a domain name query request, if there is no corresponding query result in the current S table, it updates the domain name in the S table and sends the updated S table to the storage module. After receiving the S-table from the output module, the storage module queries the domain names newly updated in the S-table, saves the corresponding query results to the S-table, and returns the updated S-table to the output module.

4. A network domain name and address relationship information capturer according to claim 1, characterized in that, The processing module is used to process the filtered DNS data packets and generate P-tables and A-tables. The processing includes the following steps: Based on the value of the QR field in the flag bit of the DNS packet, packets with a QR value of 0 are timestamped and recorded in the P table, which is called the DNS Potential list; packets with a QR value of 1 are timestamped and recorded in the A table, which is called the DNS active list.

5. A network domain name and address relationship information capturer according to claim 4, characterized in that, It also includes a test module, which is configured with an interface M. The interface L is interconnected with the interface M on the test module, and the traffic of the interface M is mirrored on the interface L. The interface M is an interface that can communicate with the Internet. The test module is used to perform the following steps: The first step is for the test module to receive the test instructions from the processing module and determine the source network IP address based on the instructions. The second step is to use the P table and A table to obtain testable DNS servers identified by network IP address, protocol type and port number, and use the S table to obtain the domain name to be queried. The domain name to be queried is the domain name for which no corresponding result was found based on the R table. The third step involves the test module constructing a DNS request packet using the source network IP address, the DNS server information recorded in table A, and the domain name to be queried in table S, and then accessing the DNS server. The fourth step involves the test module constructing a DNS request packet using the source network IP address, the DNS server information recorded in the P table, and the domain name to be queried in the S table, and then accessing the DNS server. Fifth, if the DNS server does not find any record for the domain name, it considers the domain name provided by the subscriber to be invalid, and the test module supplements its own source network IP address into the DNS response packet; When the output module receives the S-table from the storage module, it checks the S-table. If the network IP address corresponding to the domain name in the S-table is the source network IP address configured on the M-port by the catcher, then the domain name is marked as an invalid domain name.

6. A network domain name and address relationship information capturer according to claim 1, characterized in that, The filtering module receives DNS packets in a loop using multiple queues. The maximum number of packets that each queue can receive is adjustable. The specific looping method is as follows: each queue is pre-assigned an ordered sequence number. When the first queue is full, the filtering module automatically uses the next queue to continue receiving. When the last queue starts receiving, the first queue is immediately cleared and its sequence number is set to the last sequence number. At the same time, the sequence numbers of other queues are shifted forward by one position.

7. A network domain name and address relationship information capturer according to claim 6, characterized in that, The processing module is equipped with a periodic timer T0 that starts when the capture device is powered on. When timer T0 expires, it retrieves the data packets from the first sequence of the current sequence from the filtering module for processing.

8. A network domain name and address relationship information capturer according to claim 1, characterized in that, The output module is also used to perform a validity check on the domain name query requests from external subscribers, and to determine whether the subscriber's domain name query request conforms to the following rules: Rule 1: The number of domain entries queried in a single query cannot exceed the preset upper limit of the output module; Rule 2: Domain name string format, case sensitive, length range 1 to 255; Rule 3: Domain names must not contain the following characters: "@", "\", ":", "*", "", "<", ">", "|", " / ", "#", "=", "'", and "&". Rule 4: Domain names are allowed to begin with the wildcard character "**"; Rule 5: Domain names are allowed to have 2-4 levels. Rule 6: The "." symbol cannot be consecutive or appear at the end.

9. A network domain name and address relationship information capturer according to claim 1, characterized in that, The filtering module obtains the filtering ratio r from the processing module of the capture device. If the processing module does not send the filtering ratio r, the filtering module receives all packets at a 1:1 ratio by default. Otherwise, the filtering module receives and samples uplink and downlink data packets one packet for every r packets.

Images