An efficient domain name resolution first discovery method, device and medium based on DPDK and BloomFilter

Abstract

The present invention relates to an efficient domain name resolution first discovery method based on DPDK and BloomFilter. The DNS message flow is received in real time by DPDK, and each DNS message is analyzed. The BloomFilter algorithm is used to determine whether the domain name and request type in each DNS message resolution result are first discovered. The domain name field and request type field in the resolution result of the current DNS message are used to calculate H1, H2 and H3, and check whether BF[H1], BF[H2] and BF[H3] are all 1. If so, the current domain name and request type are not first discovered. If not, the current domain name and request type are first discovered, and then the first discovery result is output to a file or a database. The advantages of the present invention are: the present invention can output the first discovered domain name in real time when using a hardware device, which greatly shortens the output time of the relevant result data; using a relatively small amount of computing and memory resources, the real-time output function of the first discovery of the domain name resolution can be realized.

Description

Technical Field

[0001] The present invention relates to a domain name resolution first discovery software architecture, a DPDK and a BloomFilter algorithm application field, and in particular to an efficient domain name resolution first discovery method based on DPDK and BloomFilter. Background Art

[0002] Domain name resolution first discovery refers to finding the DNS request domain name and request type that appear for the first time in the network by analyzing the logs or response message streams generated by the DNS resolution server. Record them in a database or file for subsequent use. In addition, the ancillary fields that can be recorded may also include fields such as the requesting user address, request generation time, domain name resolution result, and response server address.

[0003] However, to complete the above tasks, the computing and storage resources of dozens or even dozens of clusters are required. As the number of domain names and the number of requests continue to increase, the consumption of resources is also increasing. In order to realize the first discovery function of domain name resolution, the existing invention must complete the deduplication of a large amount of domain name data, and at the same time, it is necessary to record the first discovered domain name and its related subsidiary fields. Its disadvantages are that it consumes a lot of computing and storage resources; second, the calculation time is long, generally taking several hours to complete, and due to the limitations of the invention itself, it takes a while to view the result data; third, the deployment is complex and the operation and maintenance cost is high. Summary of the invention

[0004] The purpose of the present invention is to provide an efficient domain name resolution first discovery method based on DPDK and BloomFilter algorithm to solve the above problems. The DPDK and BloomFilter algorithm are used to realize the real-time output function of domain name resolution first discovery with a single device and relatively small amount of computing and memory resources. The present invention solves the problems of high cost, difficult maintenance and complex deployment of traditional Hadoop cluster.

[0005] The technical solution to achieve the purpose of the present invention is:

[0006] An efficient domain name resolution first discovery method based on DPDK and BloomFilter includes the following steps:

[0007] S1. Receive DNS message streams in real time through DPDK and parse each DNS message.

[0008] S2. Use the BloomFilter algorithm to determine whether the domain name and request type in each DNS message resolution result are the first discovery:

[0009] S2-1. Calculate H1, H2 and H3 using the domain name field and request type field in the resolution result of the current DNS message; where H1 = (a+b)%m, H2 = (a+b*2)%m, H3 = (a+b*3)%m, a = MurmurHash64A(DT, Slen, seed), b = MurmurHash64A(DT, Slen, a), DT is the string concatenated by the domain name field D and the request type field T, Slen is the length of the DT string, seed is the set value, MurmurHash64A is the HASH function algorithm, and m is the total number of memory bits occupied by the BloomFilter algorithm;

[0010] S2-2. Check whether BF[H1], BF[H2] and BF[H3] are all 1. If so, the current domain name and request type are not the first discovery. Discard the current DNS message, extract the next DNS message to be processed, and return to S2-1. If not, the current domain name and request type are the first discovery. Output the first discovery result, extract the next DNS message to be processed, and return to S2-1. BF[HX] represents the HXth bit in m.

[0011] Furthermore: In S1, the multi-channel function of the network card is utilized through DPDK to receive the DNS response message flow.

[0012] Further: In S1, the resolution result of the DNS message includes the requesting user address, the request generation time, the request type, the domain name, and the response server address.

[0013] Further: In S2-2, when the current domain name and request type are first discovered, the corresponding resolution result and other field contents are recorded.

[0014] Further: In S2-2, when the current domain name and request type are first discovered, BF[H1], BF[H2] and BF[H3] are all set to 1, and the current DNS message is discarded, and the next DNS message is extracted.

[0015] In addition, the present invention provides an efficient domain name resolution first discovery device based on DPDK and BloomFilter, including:

[0016] Memory for storing computer programs;

[0017] The processor is used to implement the steps of the efficient domain name resolution first discovery method based on DPDK and BloomFilter as described above when executing the computer program.

[0018] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the efficient domain name resolution first discovery method based on DPDK and BloomFilter as described above are implemented.

[0019] Compared with the prior art, the present invention has the following significant advantages:

[0020] 1. The present invention proposes a method based on DPDK and BloomFilter algorithm, receives DNS message flow in real time through DPDK, adopts multi-channel network card queue to receive, parse and process messages, extracts fields such as request domain name, request user address, request generation time, request type, domain name resolution result, response server address, etc.; and then uses BloomFilter method to complete deduplication work with a single device;

[0021] 2. The present invention can complete the domain name resolution and discovery function of 5 million to 6 million QPS on a single device. It solves the problems of high resource consumption, long calculation cycle and high investment cost of traditional inventions. Compared with the original technical invention, its processing hardware cost is reduced by dozens of times, the calculation time is expected to be reduced by dozens of times, the structure is simple and the deployment is simple. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0023] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings.

[0024] An efficient domain name resolution first discovery method based on DPDK and BloomFilter, such as Figure 1 As shown, the following steps are included:

[0025] 1. Through DPDK, the multi-channel function of the network card is used to receive the DNS response message stream and parse out fields such as the requesting user address, request generation time, request type, domain name resolution result, and response server address.

[0026] 2. Concatenate the domain name field D and the request type field T into a string DT, and calculate H1, H2 and H3, which are a=MurmurHash64A(DT,Slen,seed), b=MurmurHash64A(DT,Slen,a), H1=(a+b)%m, H2=(a+b*2)%m, H3=(a+b*3)%m; seed can be any specified value, and MurmurHash64A is a universal HASH algorithm.

[0027] 3. Check whether BF[H1], BF[H2] and BF[H3] are all 1. If so, it proves that the domain name and request type are not the first discovery, and it is directly discarded and the process ends. Otherwise, go to the next step. Among them, BF[HX] represents the HXth bit in the total memory m.

[0028] 4. Set BF[H1], BF[H2], and BF[H3] to 1.

[0029] 5. Entering this step indicates that the domain name and request type are the first to be discovered, and records fields such as the requesting user address, request generation time, request type, domain name resolution result, and response server address, and then outputs the initial results to a file or database.

[0030] The present invention designs DPDK to quickly receive and parse data streams, and then uses the BloomFilter algorithm to quickly determine whether a domain name and request type have appeared, thereby determining whether the domain name and its type are discovered for the first time. The BloomFilter algorithm has a certain false positive rate, which means that domain name A has not appeared, but is mistakenly judged to have appeared. The calculation formula for the false positive rate is: f(m, k, n) = (1-e -kn / m ) k , where k represents the number of HASH functions in the BloomFilter algorithm, m is the total number of memory bits occupied, and n is the total number of estimated domain names and request types.

[0031] Taking 400GB memory (about 3.5 trillion bits), 10 billion domain names, and 3 HASH functions as an example, by adopting the technical solution of the present invention, the misjudgment rate is only 0.0000007, i.e., 7 / 10 million.

[0032] The present invention uses DPDK to receive and parse DNS response data streams in real time, providing intermediate data to be analyzed for real-time output of domain name resolution first-discovery results; using the BloomFilter algorithm, multiple HASH values ​​are used to set and query memory bits, so as to output in real time whether the domain name and type are first discovered using the computing and storage resources of a single device. This allows the determination of whether the domain name and type appear for the first time when using a single hardware device; at the same time, the first-discovered domain name can be output in real time, greatly shortening the output time of the relevant result data.

[0033] The present application also provides an efficient domain name resolution first discovery device based on DPDK and BloomFilter, comprising: a memory and a processor; a computer program is stored on the memory, and when the computer program is executed by the processor, the above-mentioned efficient domain name resolution first discovery method based on DPDK and BloomFilter is implemented.

[0034] The present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned efficient domain name resolution first discovery method based on DPDK and BloomFilter are implemented. The computer-readable storage medium may include: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0035] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0036] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An efficient domain name resolution first discovery method based on DPDK and BloomFilter, characterized in that: The steps include: S1. Receive DNS message streams in real time through DPDK and parse each DNS message. S2. Use the BloomFilter algorithm to determine whether the domain name and request type in each DNS message resolution result are the first discovery: S2-1. Calculate H1, H2 and H3 using the domain name field and request type field in the resolution result of the current DNS message; where H1 = (a+b)%m, H2 = (a+b*2)%m, H3 = (a+b*3)%m, a = MurmurHash64A(DT, Slen, seed), b = MurmurHash64A(DT, Slen, a), DT is the string concatenated by the domain name field D and the request type field T, seed is the set value, Slen is the length of the DT string, MurmurHash64A is the HASH function algorithm, and m is the total number of memory bits occupied by the BloomFilter algorithm; S2-2, check whether BF[H1], BF[H2] and BF[H3] are all 1. If so, the current domain name and request type are not the first discovery, discard the current DNS message, extract the next DNS message, and return to S2-1; if not, the current domain name and request type are the first discovery, output the first discovery result, extract the next DNS message, and return to S2-1; where BF[HX] represents the HXth bit in m; In S1, the resolution result of the DNS message includes the requesting user address, request generation time, request type, domain name, and response server address; In S2-2, when the current domain name and request type are first found, the corresponding resolution result is recorded.

2. According to claim 1, a method for efficiently discovering domain name resolution based on DPDK and BloomFilter is characterized in that: In S1, DPDK uses the multi-channel function of the network card to receive the DNS response message flow.

3. According to claim 1, a method for efficiently discovering domain name resolution based on DPDK and BloomFilter, characterized in that: In S2-2, when the current domain name and request type are first discovered, BF[H1], BF[H2] and BF[H3] are all set to 1, and the current DNS message is discarded, and the next DNS message to be processed is extracted.

4. An efficient domain name resolution first discovery device based on DPDK and BloomFilter, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the efficient domain name resolution first discovery method based on DPDK and BloomFilter as described in any one of claims 1 to 3 when executing the computer program.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the efficient domain name resolution first discovery method based on DPDK and BloomFilter as claimed in any one of claims 1 to 3 are implemented.

Images