A large-scale DNS passive traffic query and response alignment method

Abstract

The application discloses a large-scale DNS passive flow query and response alignment method, which comprises the following steps: constructing a packet cache structure of an in-memory pool-hash pool-hash table, and completing DNS passive flow query and response alignment according to the packet cache structure; wherein the in-memory pool is a pre-applied memory segment, the head of the corresponding memory segment is the relevant information of the in-memory pool, and the memory of n pre-applied and pre-set structure bodies is arranged after the head; the hash pool is a pre-applied memory segment, the head of the corresponding memory segment is the relevant information of the hash pool, n pointers are arranged after the head, and the pointers are initialized as null, being an empty list with a size of n; and the hash table is a hash table using a cuckoo hash algorithm. The application solves the problem of packet cache difficulty in large-scale DNS passive monitoring flow query and response alignment.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, and in particular relates to a method for large-scale DNS passive traffic query and response alignment. Background Technology

[0002] The Domain Name System (DNS) is one of the most important infrastructures of the Internet, providing the most basic IP-domain name relationship lookup and resolution services. Its security is crucial to the security and stability of the entire Internet. Monitoring and analyzing DNS traffic on DNS servers is an important aspect of monitoring DNS system security risks, effectively detecting and providing early warnings of DNS system security incidents. DNS traffic monitoring and analysis typically requires aligning domain name query requests and responses within DNS traffic to detect anomalies such as denial-of-service, incorrect resolution, and timeouts on DNS servers, thereby promptly identifying security risks. In particular, analyzing domain name query requests and responses from backbone DNS servers on critical resolution links can promptly detect DNS system outages and network interruptions, which is of great significance for ensuring DNS security.

[0003] However, backbone DNS servers on critical links typically handle massive amounts of DNS traffic; for example, a recursive DNS server in a particular region may handle millions of queries per second. Aligning domain name query requests and responses from such massive amounts of DNS traffic, ensuring a one-to-one match between queries and responses, presents significant challenges. This is one of the key problems that needs to be addressed in the security monitoring of backbone DNS servers on critical links, and is of great importance for DNS risk warning and security assurance.

[0004] DNS query-response alignment requires caching query request packets and marking them with identifiers for easy lookup and pairing with response packets. The packet caching method directly impacts the efficiency of DNS query-response alignment. Large-scale DNS passive monitoring traffic scenarios place extremely high demands on DNS query-response alignment efficiency; therefore, the packet caching method is crucial to controlling the query and response efficiency of large-scale DNS passive monitoring traffic. There are two main packet caching methods: databases and hash tables. Each method has its advantages and disadvantages and cannot meet the query-response alignment requirements of large-scale DNS passive monitoring traffic.

[0005] Database-based packet caching methods require in-memory databases, but the throughput of a single in-memory database node for inserts and queries is only around 150,000 QPS, which cannot meet the throughput requirements of query and response alignment for large-scale DNS passive monitoring traffic. Using batch inserts and queries or pipelines can achieve a throughput of 1 million QPS, which can basically meet the throughput requirements of query and response alignment for large-scale DNS passive monitoring traffic. However, such a large volume of queries and writes may cause database crashes, excessive network card load, or excessive CPU load. Solving these problems usually requires building an in-memory database cluster. However, data consistency is difficult to guarantee with in-memory database clusters, load balancing is not easy to achieve, and in-memory database clusters will incur a performance loss of approximately 30%-40%, significantly increasing the load on servers and switches, leading to a substantial increase in maintenance costs.

[0006] Hash table-based packet caching methods offer fast response times for queries, insertions, and deletions because hash tables have O(1) time complexity. However, the response speed of a hash table is dependent on its size; the larger the hash table, the slower the response. This significantly reduces the response speed in large-scale DNS passive monitoring traffic scenarios, failing to meet requirements. Furthermore, using only hash tables means that if a large number of erroneous query requests are inserted into the hash table and not promptly deleted, the empty slots will fill up, causing a crash. Deleting erroneous queries requires traversing the hash table, greatly reducing query performance. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems of the prior art by disclosing a method for large-scale DNS passive traffic query and response alignment. This method solves the problem of difficult packet caching during query and response alignment of large-scale DNS passive monitoring traffic.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A large-scale DNS passive traffic query and response alignment method, the DNS passive traffic query and response alignment method comprising:

[0010] Construct a packet caching structure of memory pool-hash pool-hash table, and perform DNS passive traffic query and response alignment based on the packet caching structure;

[0011] The memory pool is a pre-allocated memory segment. The header of the corresponding memory segment contains information about the memory pool, and the header is followed by the memory of n pre-allocated, pre-defined structures.

[0012] The hash pool is a pre-allocated memory segment. The header of the corresponding memory segment contains relevant information about the hash pool. After the header are n pointers, which are initialized to null and are empty lists of size n, to cache the pointers of each query inserted into the memory pool.

[0013] The hash table is a hash table that uses the Cuckoo hash algorithm.

[0014] According to a preferred implementation, when the queryN packet arrives, the processing flow is as follows: (1) After the queryN packet is parsed, a memory segment is first requested from the mempool, the information of the queryN packet is stored in the memory segment, and the pointer ptrN of the memory segment is obtained;

[0015] (2) Get the next position N of the hashpool from the hashpool. If the content of the next position is null, put ptrN into the position and proceed to step (6).

[0016] If the content of the next position is not null, it means that the response packet respM packet corresponding to the queryM packet stored in the mempool pointed to by this position has been lost, so proceed to step (3);

[0017] (3) Remove ptrM from the hashpool and set its position to ptrN, then proceed to step (4);

[0018] (4) Obtain the corresponding queryM based on ptrM, return the memory of queryM to mempool, and proceed to step (5);

[0019] (5) After parsing queryM, the main content of the query packet is obtained, and the information is converted into a format and sent to other systems using MQ; at this time, it means that the queryM packet was lost and no respM packet arrived, and then continue to step (6);

[0020] (6) Combine the udp->src_port of queryN and the transaction_id in the queryN packet into a 32-bit key, and use the pointer ptr of positionN as the value;

[0021] According to a preferred embodiment, when a resp packet arrives, the processing flow is as follows:

[0022] (a) After the respN packet is parsed, combine the udp->dst_port of the respN packet and the transaction_id in the respN packet into a 32-bit key, and query the hash table.

[0023] If the corresponding value vN is found, which is ptr of positionN, then continue to step (b);

[0024] If not found, it means that the queryN packet corresponding to the respN is lost or the respN packet has been retransmitted. Set the delay to -1, convert the information and delay of the respN packet to the correct format, and continue to step (d).

[0025] (b) Remove vN from the hash table, and obtain ptrN based on the corresponding position of the hash pool pointed to by ptr of positionN, and set the corresponding positionN of the hash pool to null, then continue to step (c);

[0026] (c) Use ptrN to obtain queryN, thus completing the alignment of domain name query and response; use queryN to obtain the latency of the respN packet, convert the information and latency of the respN packet into a format to obtain the information of the domain name query response, and return the memory of queryN to mempool, then continue to step (d).

[0027] (d) Send the formatted information to other systems for processing using mq.

[0028] The aforementioned main solution of the present invention and its various further alternative solutions can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed by the present invention. Those skilled in the art, after understanding the solution of the present invention, will realize that there are many combinations based on existing technology and common knowledge, all of which are technical solutions to be protected by the present invention, and will not be exhaustively listed here.

[0029] The beneficial effects of this invention are:

[0030] The large-scale DNS passive traffic query and response alignment method of this invention can greatly improve the efficiency of traffic alignment, and can also obtain timed-out domain name query packets in a short time. The use of memory pool also ensures that the system allocates a fixed amount of memory at startup, so that excessive memory usage will not occur during operation, and at the same time, the generation of memory fragmentation is reduced, which greatly improves the performance of the system. Attached Figure Description

[0031] Figure 1 This is a diagram of the memory pool-hash pool-hash table structure constructed in this invention;

[0032] Figure 2 This is the initial state diagram of the data packet caching structure of this invention for processing queryN packets;

[0033] Figure 3 This is a schematic diagram illustrating the acquisition of the next position in the hashpool during the queryN packet processing process of the present invention.

[0034] Figure 4 This is a schematic diagram of hashpool operations during the queryN packet processing process of the method of this invention;

[0035] Figure 5 This is a schematic diagram illustrating mempool memory return during the queryN packet processing process of the method of this invention;

[0036] Figure 6 This is a schematic diagram illustrating the parsing of queryM during the queryN packet processing process of the method of this invention;

[0037] Figure 7 This is a schematic diagram of hash table operations during the queryN packet processing process of the method of the present invention;

[0038] Figure 8 This is a schematic diagram of the respN packet processing method of the present invention, and a schematic diagram of the hash table processing.

[0039] Figure 9 This is a schematic diagram of hash pool processing during the respN packet processing process of the present invention;

[0040] Figure 10 This is a schematic diagram illustrating the DNS information obtained during the respN packet processing process of the method of this invention;

[0041] Figure 11 This is a schematic diagram illustrating the transmission of processing results during the respN packet processing process of the method of the present invention. Detailed Implementation

[0042] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0043] This invention discloses a method for large-scale DNS passive traffic query and response alignment, which includes: constructing a packet caching structure of memory pool-hash pool-hash table, and completing DNS passive traffic query and response alignment according to the packet caching structure.

[0044] Specifically, refer to Figure 1 As shown, the memory pool (mempool) is a large pre-allocated block of memory. The header contains information about the memory pool, followed by the memory of n pre-allocated, pre-defined structures. The purpose of using a memory pool instead of malloc and free is to reduce memory fragmentation, facilitate memory management, and align with the CPU cache lines, thereby improving performance.

[0045] The hash pool is a large pre-allocated block of memory. The header contains hash pool information, followed by n pointers, initialized to null, which are empty lists of size n. The purpose is to cache pointers to each query inserted into the memory pool.

[0046] A hash table is a hash table that uses the Cuckoo hash algorithm.

[0047] The process of completing DNS passive traffic lookup and response alignment is shown below.

[0048] Assuming this is the state after running for a period of time, such as Figure 2 As shown. When the queryN packet arrives, the processing algorithm flow is as follows:

[0049] (1) As Figure 2 As shown, after queryN is parsed, a memory segment is first requested from mempool, the information of the queryN packet is stored in the memory segment, and a pointer ptrN to the memory segment is obtained.

[0050] (2) Obtain the next position N from the hashpool. If the content of the next position is null, then put ptrN into that position. Figure 3 As shown, proceed to step (6); if the content of the next position is not null, it means that the response packet respM packet corresponding to the queryM packet stored in the mempool pointed to by this position has been lost, then proceed to step (3);

[0051] (3) Figure 4 As shown, ptrM is retrieved from the hashpool and its position is set to ptrN, and then step (4) is performed;

[0052] (4) Figure 5As shown, obtain the corresponding queryM according to ptrM, return the memory of queryM to mempool, and proceed to step (5);

[0053] (5) Figure 6 As shown, after parsing queryM, the main content of the query packet is obtained, and the information is converted into a different format and sent to other systems using MQ. At this point, it indicates that queryM has been lost and no respM packet has arrived, and step (6) continues.

[0054] (6) Figure 7 As shown, the udp->src_port of queryN and the transaction_id in the queryN packet are combined into a 32-bit key, and the pointer ptr of positionN is used as the value. First, it is checked whether the key is already in the hash_table. If it exists, it means that the domain name query packet has been retransmitted, so no insertion is performed; if it does not exist, the key-value pair is inserted into the hash_table.

[0055] When the respN packet arrives, the processing algorithm flow is as follows:

[0056] (1) As Figure 8 As shown, after the respN packet is parsed, the udp->dst_port of respN and the transaction_id in the respN packet are combined into a 32-bit key. The hash_table is queried. If the corresponding value is found, that is, the ptr of positionN, the process continues to step (2). If it is not found, it means that the queryN packet corresponding to the respN is lost or the respN packet has been retransmitted. The delay is set to -1, and the information and delay of the respN packet are converted in format and the process continues to step (4).

[0057] (2) Figure 9 As shown, vN is deleted from the hash table, and ptrN is obtained according to the corresponding position of the hash pool pointed to by ptr of positionN. The corresponding position N of the hash pool is set to null, and step (3) is continued.

[0058] (3) Figure 10As shown, queryN is obtained using ptrN, and the alignment of domain name query and response is completed; the latency of the respN packet is obtained using queryN, and the information and latency of the respN packet are converted to obtain the information of the domain name query response. The memory of queryN is returned to mempool, and step (4) continues.

[0059] (4) Figure 11 As shown, the formatted information is sent to other systems for processing using message queues (MQ).

[0060] The large-scale DNS passive traffic query and response alignment method of this invention can greatly improve the efficiency of traffic alignment, and can also obtain timed-out domain name query packets in a short time. The use of memory pool also ensures that the system allocates a fixed amount of memory at startup, so that excessive memory usage will not occur during operation, and at the same time, the generation of memory fragmentation is reduced, which greatly improves the performance of the system.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for large-scale DNS passive traffic query and response alignment, characterized in that, The DNS passive traffic query and response alignment method includes: Construct a packet caching structure of memory pool-hash pool-hash table, and complete DNS passive traffic query and response alignment based on the packet caching structure; The memory pool is a pre-allocated memory segment. The header of the corresponding memory segment contains information about the memory pool, and the header is followed by the memory of n pre-allocated, pre-defined structures. The hash pool is a pre-allocated memory segment. The header of the corresponding memory segment contains information about the hash pool, followed by n pointers, which are initialized to null and are empty lists of size n. The hash table is a hash table that uses the Cuckoo hash algorithm; When the queryN packet arrives, the processing flow is as follows: (1) After the queryN packet is parsed, a memory segment is first requested from mempool, the information of the queryN packet is stored in the memory segment, and the pointer ptrN of the memory segment is obtained; (2) Get the next position N of the hashpool from the hashpool. If the content of the next position is null, put ptrN into the position and proceed to step (6). If the content of the next position is not null, it means that the response packet respM packet corresponding to the queryM packet stored in the mempool pointed to by this position has been lost, so proceed to step (3); (3) Remove ptrM from the hashpool and set its position to ptrN, then proceed to step (4); (4) Obtain the corresponding queryM based on ptrM, return the memory of queryM to mempool, and proceed to step (5); (5) After parsing queryM, the main content of the queryM packet is obtained, and the information is converted into a format and sent to other systems using mq; at this time, it means that the queryM packet was lost and no respM packet arrived, and then continue to step (6); (6) Combine the udp->src_port of queryN and the transaction_id in the queryN packet into a 32-bit key, and use the pointer ptr of positionN as the value; First, check if the key is already in the hash_table. If it exists, it means that the domain name lookup packet for the key was retransmitted, so no insertion is performed. If it does not exist, insert the key-value pair into the hash_table. When the respN packet arrives, the processing flow is as follows: (a) After the respN packet is parsed, combine the udp->dst_port of the respN packet and the transaction_id in the respN packet into a 32-bit key, and query the hash_table. If the corresponding value vN is found, which is ptr of positionN, then continue to step (b); If not found, it means that the queryN packet corresponding to the respN is lost or the respN packet has been retransmitted. Set the delay to -1, convert the information and delay of the respN packet to the correct format, and continue to step (d). (b) Remove vN from the hash table, and obtain ptrN based on the corresponding position of the hash pool pointed to by ptr of positionN, and set the corresponding positionN of the hash pool to null, then continue to step (c); (c) Use ptrN to obtain queryN, thus completing the alignment of domain name query and response; use queryN to obtain the latency of the respN packet, convert the information and latency of the respN packet into a format to obtain the information of queryN and respN domain name query response, and return the memory of queryN to mempool, then continue to step (d). (d) Send the formatted information to other systems for processing using MQ.

Images