A method and device for quickly matching DNS network domain name data

Abstract

The present invention discloses a method and device for quickly matching DNS network domain name data. The method includes: a Hadoop high-availability distributed cluster is used to perform multi-node design for the name node process in the distributed file system and the resource management process in the resource manager. If the main node fails, the backup node switches to active mode; a resource manager cluster mode is selected for processing DNS log data stored in the distributed file system, and Spark job operation parameters are designed by optimizing operators on the elastic data set of the directed acyclic graph; an improved double-array dictionary tree is used to calculate the fail() pointer using a breadth-first search algorithm, and each parent and child node is connected according to preset rules to create a matching output function; and a parallel Python module is installed. On the cluster, multi-threaded parallel computing and an enhanced AC automaton algorithm are combined to match the massive CNAME domain names in the DNS log data with a specified domain name library. The device includes a processor and a memory.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a method and device for quickly matching DNS network domain name data, more particularly to a technology for processing massive data through Spark distributed computing based on highly available Hadoop, and more particularly to efficient and fast matching of massive CNAME data in DNS logs. Background Art

[0002] As internet technology continues to evolve, DNS (Domain Name Server) network data is experiencing explosive growth. Telecom operators have discovered that their clients are reselling network traffic. To address this issue, they need to rapidly extract and match massive amounts of CNAME (Category Name) data from DNS logs. This, combined with DPI (Deep Packet Inspection) call record comparison and analysis, can improve refined internet operations and business management capabilities.

[0003] Scholars at home and abroad have conducted extensive research on how to handle large-scale data processing. Traditional MapReduce (a distributed computing program) within the Apache Hadoop framework, a distributed, intensive data processing platform, enables parallel computing and is effective for large data processing. However, it has certain drawbacks, such as limited expressiveness, high disk I / O overhead, and long processing times. To rapidly match massive amounts of domain name data in the DNS, a wide range of matching algorithms is needed. The single-pattern matching Brute Force (BF) algorithm, which relies on character-by-character search, is very inefficient. The Knuth-Morris-Pratt (KMP) and Boyer-Moore (BM) algorithms offer improved efficiency over BF, but their inability to perform multi-pattern matching is a limitation. The multi-pattern Aho-Corasick automaton (AC) matching algorithm, which achieves higher efficiency by constructing finite state automata to match multiple patterns in a single text scan, requires significant storage space. However, the AC automaton algorithm's dictionary tree construction consumes significant storage space, significantly impacting matching efficiency.

[0004] Although existing large-scale data matching algorithms have made great progress, these methods still have some shortcomings, resulting in unsatisfactory fast matching effects and low matching efficiency. Summary of the Invention

[0005] The present invention provides a method and apparatus for quickly matching DNS network domain name data. The present invention solves the problem of quickly matching massive CNAME data in DNS logs by combining a Spark (computing engine) distributed computing cluster based on highly available Hadoop with a parallel improved AC automaton algorithm. The highly available Spark distributed cluster on Hadoop solves the problem of high fault tolerance storage and high efficiency computing of massive DNS log data. The parallel improved multi-mode AC automaton algorithm is used to significantly reduce memory storage space and improve matching efficiency. Detailed description is provided below:

[0006] In a first aspect, a method for quickly matching DNS network domain name data comprises the following steps:

[0007] Implement keyless secure shell protocol communication, cluster clock synchronization, and firewall settings between servers that build multiple Linux systems;

[0008] On the Linux platform, decompress and install the compressed package to build a highly available distributed cluster. The cluster is used to implement a multi-node design for the name node process in the distributed file system and the resource management process in the resource manager. If the primary node fails, the standby node switches to active mode.

[0009] Select the resource manager cluster mode for processing DNS log data stored in a distributed file system, optimize operators on elastic datasets of directed acyclic graphs, and design Spark operating parameters;

[0010] Adopting the improved double array dictionary tree, using the breadth-first search algorithm to calculate the fail() pointer, and connecting each parent and child node according to the preset rules to create a matching function;

[0011] The parallel Python module is installed, and multi-threaded parallel computing and an enhanced AC automaton algorithm are combined on the cluster to perform matching calculations on the massive CNAME domain names in the DNS log data with the specified domain name library.

[0012] The cluster clock synchronization is specifically as follows:

[0013] Use the time synchronization command to install the time synchronization plug-in on the server; use the time write command to write the system time to the hard disk; finally, use the time synchronization command to enter in the terminal to ensure time synchronization between different machines in the cluster.

[0014] Furthermore, the firewall setting operation is specifically as follows:

[0015] Query the firewall status between each server; if the firewall status is active, disable the firewall of each server;

[0016] Ensure that the cluster firewall is permanently disabled by editing the file and changing the value of selinux (the Linux kernel's security subsystem) to disable.

[0017] The cluster is used to perform multi-node design on the name node process in the distributed file system and the resource management process in the resource manager as follows:

[0018] ① After decompressing and installing the Hadoop software installation package on the server, design the corresponding parameters in the configuration file under the installation directory; ② Distribute the Hadoop installation directory with the modified configuration file to other server nodes; ③ Then configure the JDK installation path in the yarn-env.sh file; ④ Open the startup command and start the JournalNode (a daemon process that keeps the standby node synchronized with the running master node) service on all nodes; ⑤ Initialize the NameNode on the master node; ⑥ Use the command to initialize the Hadoop high availability status in Zookeeper; ⑦ Start the HDFS command and YARN command on the master node; ⑧ Start the DFSZK Failover Controller on each NameNode node; ⑨ Check whether the process service command is normal.

[0019] Furthermore, the improved double array dictionary tree is:

[0020] The double array dictionary tree includes: base[] array and check[] array. Each element of the base array represents a dictionary tree node, and each element of the check array represents the predecessor state of a state, which is used to check whether the state transfer is correct.

[0021] base[s]+c=t

[0022] check[t]=s

[0023] Among them, s is the subscript of the current state, i.e. the parent node, t is the subscript of the transition state, i.e. the child node, and c is the encoding value of the input character. Based on the double array dictionary tree, the tail[] array is introduced to compress the non-public prefixes of the dictionary tree. The negative value of base[s] indicates a non-public prefix, and the absolute value indicates that the character suffix corresponding to the node is at the starting position of the tail[] array. If the tail string corresponding to the node s is b1, b2, ...b k , satisfying base[s]<0, then the corresponding string b1...b k Put it into the tail[] array.

[0024] pos=|-base[s]|

[0025] tail[pos+k-1]=b k (k≥1)

[0026] Furthermore, the enhanced AC automaton specifically operates as follows:

[0027] Among them, for the improved double array dictionary tree, the breadth-first search algorithm is used to calculate the fail() pointer, and each parent and child node is connected according to the preset rules to create a matching output function:

[0028] Traverse the input string, c is the character, sn is the state, n starts from 0, s0=0;

[0029] First press goto to jump: s(n+1)=base[sn]+c+1. If check[s(n+1)]=base[sn], the jump is successful and returns s(n+1). If the jump fails, check if s0==0. If so, return 0. Otherwise, continue to jump according to the fail pointer.

[0030] Jump according to the fail pointer, s(n+1)=base[fail[sn]]+c+1. If check[s(n+1)]=base[sn], the jump is successful and returns s(n+1). If the jump fails, check if s0==0. If so, return 0, otherwise continue to jump according to fail.

[0031] Verify the output table based on the return value. If output[s(n+1)]!=null, it means the match is successful, and all matching pattern strings are returned.

[0032] Continue processing with the next character.

[0033] The multi-threaded parallel computing and enhanced AC automaton algorithm on the cluster are specifically:

[0034] 1) Split the DNS log domain name T into s subsets, namely T = {t1, t2, ...t i}, i = 1, 2, ... s, subset t i ={tstr i1 , tstr i2 ,...tstr ij ,...},i=1,2,…s,j=12,…n,tstr ij represents the jth domain name in the i-th subset, the target domain name set P = {pstr1, pstr2, ...pstr k}, k = 1, 2, ... m, pstr k represents the kth domain name;

[0035] 2) Create all working nodes in the cluster and start the parallel computing cluster;

[0036] 3) Parallelize and assign the s subsets to the working nodes of the cluster;

[0037] 4) Submit the AC algorithm matching task for calculation.

[0038] In a second aspect, a device for quickly matching DNS network domain name data comprises: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the method steps described in the first aspect.

[0039] A third aspect is a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the method steps described in the first aspect.

[0040] The beneficial effects of the technical solution provided by the present invention are:

[0041] 1. The present invention adopts the Spark distributed cluster mode of highly available Hadoop to ensure high fault-tolerant storage and efficient real-time calculation of massive DNS log data;

[0042] 2. This invention improves the computing performance of Spark jobs by rationally designing Spark operating parameters and optimizing RDD (Resilient Dataset) operators;

[0043] 3. The present invention improves the dictionary tree of the multi-mode AC automaton algorithm by using a double-array dictionary tree method to improve the dictionary tree in the original AC algorithm, which greatly reduces the memory storage space and improves the matching efficiency;

[0044] 4. The present invention further improves the computing performance and matching speed of Spark by applying a parallel computing method combined with an improved multi-mode AC automaton. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a method for matching DNS (Domain Name Server) network domain name data;

[0046] Figure 2 Schematic diagram for starting a high-availability cluster service;

[0047] Figure 3 This is a diagram comparing the performance parameters of a Spark (computing engine) job before and after optimization.

[0048] Figure 4A schematic diagram of a web page with a large number of DNS (Domain Name Server) files to be processed;

[0049] Figure 5 To improve the schematic diagram of the double array dictionary tree;

[0050] Figure 6 A schematic diagram of memory analysis using an improved double array dictionary tree;

[0051] Figure 7 This is a comparison diagram before and after parallel processing of domain name data in DNS logs;

[0052] Figure 8 This is a schematic diagram of a job webpage based on a Spark distributed cluster;

[0053] Figure 9 This is a schematic diagram comparing the time efficiency of this method with other algorithms;

[0054] Figure 10 A schematic diagram of a device for matching DNS (Domain Name Server) network domain name data. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0056] Example 1

[0057] The main problem solved by the embodiment of the present invention is to propose a method for quickly matching DNS network domain name data in order to quickly match the massive domain name data in the DNS log. Figure 1 , the method comprises the following specific steps:

[0058] Step 101: To ensure time and file synchronization and stable communication between clusters, after setting up multiple Linux systems on the server, configure keyless SSH (Secure Shell) communication, clock synchronization, and firewall settings between the server clusters.

[0059] Step 102: On the Linux platform, decompress and install the JDK, Hadoop, Spark, and other related compressed packages to build a highly available distributed cluster. This ensures high fault-tolerant storage of massive DNS data, efficient computing, and high cluster reliability.

[0060] Among them, the high-availability distributed cluster mainly uses multi-node design for the NameNode (name node process) in the distributed file system HDFS and the ResourceManager (resource management process) in the resource manager YARN, so as to ensure that when the main node fails, the backup node will switch to the active mode in time, ultimately ensuring the real-time and stable operation of the cluster.

[0061] See also Figure 2 As you can see, by using Linux commands to enable the high availability cluster on the server, the specific background process of the high availability cluster service startup can be clearly seen. Figure 2 As can be seen, the name node processes and resource management processes of server nodes node01 and node02 have been designed for high availability. Node01 and node02 serve as master nodes for management, while the remaining nodes serve as slave nodes for computing. High availability is a technical term in this field, referring to multi-node cluster startup.

[0062] To ensure the storage of massive amounts of data, each node is configured as a DataNode (data node process), ensuring highly fault-tolerant storage of massive amounts of data. This highly available distributed cluster design not only ensures the stability and reliability of the cluster's 24 / 7 real-time operation, but also enables the cluster to store large amounts of data.

[0063] Step 103: Select the YARN cluster mode for processing DNS log data stored on HDFS (distributed file system), then optimize the operators of the directed acyclic graph RDD (elastic dataset) and reasonably design the Spark operation parameters to improve the computing performance of the Spark job.

[0064] See also Figure 3 It can be seen that through comparative experiments before and after tuning Spark operating parameters, it can be found that optimizing settings during Spark calculation and analysis will be 2 to 3 times more efficient than running with default Spark parameters. Obviously, reasonable parameter optimization can improve the computing performance of massive DNS log data on Spark distributed clusters.

[0065] Step 104: Improve the original AC automaton. Modifying the dictionary tree in the original AC automaton can effectively improve the storage and execution efficiency of the AC automaton.

[0066] The above modification of the dictionary tree in the original AC automaton is to adopt an improved double-array dictionary tree.

[0067] Step 105: After the above steps, the fail() pointer is calculated using the breadth-first search algorithm, and each parent and child node is connected according to the preset rules to create a matching function. At this point, the enhanced AC automaton is constructed;

[0068] Step 106: Install the parallel Python module and use multi-threaded parallel computing and an enhanced AC automaton algorithm on a highly available Hadoop Spark distributed cluster to further improve the rapid matching calculation between the massive CNAME domain names in the DNS log data and the specified domain name library.

[0069] In summary, the method designed by the embodiment of the present invention can solve the problem of rapid matching of massive CNAME data in DNS logs, and building a highly available Spark distributed computing cluster can ensure high fault-tolerant storage, high-efficiency computing and cluster reliability of massive DNS log data; the present invention can quickly read, clean and convert DNS log content, and through RDD operator optimization and Spark job performance parameter tuning, it is more efficient to calculate massive data, improves the efficiency of obtaining target domain name data and saves a lot of time; the embodiment of the present invention uses an improved dual-array dictionary tree to enhance the dictionary tree of the original AC automaton, and combines it with a parallel computing method, which can greatly reduce memory storage space and thus improve matching efficiency.

[0070] Example 2

[0071] The scheme in Example 1 is further introduced below in conjunction with specific calculation formulas and Tables 1 to 3.

[0072] The data source processed by the embodiment of the present invention is the DNS log data of the telecom operator. In order to analyze the problem of network traffic resale behavior by comparing the CNAME domain name in the DNS log with the target domain name library, a method for quickly matching DNS network domain name data is proposed. Figure 1-8 , further specifically describe the technical solutions of the embodiments of the present invention;

[0073] Step 201: To ensure time and file synchronization and stable communication between clusters, multiple Linux systems are installed on the servers, and keyless SSH (Secure Shell) communication, cluster clock synchronization, and firewall settings are implemented between the servers.

[0074] The clock synchronization operations for the server cluster are as follows:

[0075] First, use the install plugin command (for example: yum -y install ntp ntpdate) to install the time synchronization plugin on the server; second, use the insert command (for example: hwclock --systohc) to write the system time to the hard disk; finally, use crontab -e to enter the time synchronization command in the terminal (for example: * / 1**** / usr / sbin / ntpdatentp4.aliyun.com)

[0076] To ensure time synchronization between different machines in the cluster. Cluster clock synchronization is of great significance. It can ensure the real-time synchronization of cluster service process startup and the real-time response of cluster calculations.

[0077] The final server firewall configuration steps are as follows: First, use the systemctl statusfirewalld command to check the firewall status between each server. If the firewall status is active, use the systemctl stop firewalld command to disable the firewall on each server. To ensure that the firewall remains disabled after the cluster restarts, use the chkconfig iptables off command to permanently disable the cluster firewall. To ensure more real-time and stable cluster communication, edit the / etc / selinux / config file and change the selinux value to disable. Disabling the cluster firewall facilitates smooth real-time communication between servers and responsiveness between cluster processes.

[0078] Step 202: On the Linux platform, decompress and install the JDK, Hadoop, Spark and other related compressed packages to build a highly available distributed cluster, thereby ensuring high fault-tolerant storage of massive DNS data, high-efficiency computing and high reliability of the cluster. The cluster node configuration is shown in Table 1:

[0079] Table 1 Cluster node distribution

[0080]

[0081]

[0082] Among them, the method of making Hadoop highly available can be summarized into the following steps: ① After decompressing and installing the Hadoop software installation package on the server, design the corresponding high-availability parameters for the configuration files core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml, and slaves in its installation directory; ② Distribute the Hadoop installation directory with modified configuration files to other server nodes; ③ Then configure the JDK installation path in the yarn-env.sh file; ④ Open the zookeeper startup command and start the JournalNode service on all nodes; ⑤ Initialize the NameNode on the master node with the command: bin / hdfs NameNode-format; ⑥ Use the command bin / hdfszkfc-formatZK to initialize the Hadoop high-availability status in zookeeper; ⑦ Start the HDFS command and YARN command on the master node; ⑧ Start the DFSZK Failover Controller on each NameNode node; ⑨ Check whether the NameNode and ResouceManager process service commands of the high-availability design are normal.

[0083] Step 203: Select the YARN cluster mode for processing the DNS log data stored on the distributed file system HDFS, and then reasonably design the Spark operation parameters by optimizing the RDD (Elastic Dataset) operator of the directed acyclic graph. The present invention involves parameter optimization as shown in Table 2. There are three main parameter design methods: ① By designing parameters for the configuration files spark-defaults.conf and spark-env.sh in the Spark installation directory; ② By submitting the program through spark-submit and inputting parameters in the terminal; ③ By embedding parameter settings in the program. The embodiment of the present invention mainly adopts the third method.

[0084] Table 2 Spark job tuning parameters

[0085]

[0086]

[0087] Among them, reference Figure 4 As you can see, the path of DNS log data stored in the distributed file system is: / DNSFile / , and you can see the block size and corresponding permissions of each file. After starting the Hadoop service, use the remote tool to upload the compressed DNS log data package to be processed to Hadoop's HDFS. Figure 4As can be seen, each file is backed up three times, which fully ensures high fault tolerance of the stored data. Once the storage path of the DNS log data is known, the path can be written through code and then the data content can be analyzed and executed.

[0088] Step 204: improving the original AC automaton by modifying the dictionary tree in the original AC automaton and using the improved double-array dictionary tree, which can effectively improve the storage and execution efficiency of the AC automaton;

[0089] Among them, the improved dual-array dictionary tree introduces the tail[] array on the basis of the dual-array dictionary tree. The dual-array dictionary tree can effectively reduce storage space without affecting the query rate. Its core is the base[] array and the check[] array. Each element of the base array represents a dictionary tree node, that is, a state. Each element of the check array represents the predecessor state of a state. The check array is used to verify whether the state transfer is correct. The initial state sets base and check to 0. If base and check are both 0 during construction, it indicates that the state is idle. The base and check arrays can be used to represent the dictionary tree, and they satisfy the following constraints:

[0090] base[s]+c=t (1)

[0091] check[t]=s (2)

[0092] Among them, s is the subscript of the current state, that is, the parent node, t is the subscript of the transition state, that is, the child node, and c is the encoding value of the input character.

[0093] On this basis, further improvements are made by introducing tail[] array to compress the non-public prefixes of the dictionary tree. For example, the double array improvement is performed on the pattern string P = {"bachelor#","bcs#","badge#","baby#"}, such as Figure 5 As shown, the negative value of base[s] indicates a non-public prefix, and the absolute value indicates that the character suffix corresponding to the node is at the starting position of the tail[] array. If the tail string corresponding to node s is b1, b2, ... b k , satisfying base[t]<0, then the corresponding string b1…b k Put it into the tail[] array, specifically satisfying formulas (3) and (4).

[0094] pos=|-base[s]| (3)

[0095] tail[pos+k-1]=b k (k≥1) (4)

[0096] from Figure 6 As can be seen, when the number of strings is 5, the Trie memory usage is 3240 bytes, while the memory usage of the improved double-array Trie is 176 bytes, accounting for 5.43% of the original AC automaton Trie memory. When the number of strings is increased to 20, the Trie memory usage is 12528 bytes, while the improved double-array Trie memory usage is 396 bytes, accounting for 3.16% of the original AC automaton Trie memory usage. Clearly, the improved double-array Trie significantly reduces the memory usage of strings.

[0097] Table 3 Comparison of string memory usage

[0098]

[0099] Step 205: After the above steps, the fail() pointer is calculated using the breadth-first search algorithm, and each parent and child node is connected according to the preset rules to create a matching function. At this point, the enhanced AC automaton is constructed. The specific matching process is shown in Table 4.

[0100] Table 4 Enhanced AC automaton matching process

[0101]

[0102]

[0103] Step 206: Install the parallel Python module and use multi-threaded parallel computing and an enhanced AC automaton algorithm on a highly available Hadoop Spark distributed cluster to further improve the rapid matching calculation of the massive CNAME domain names in the DNS log data with the specified domain name library.

[0104] The parallel computing method for DNS log data can be summarized as follows:

[0105] 1) Split the DNS log domain name T into s subsets, namely T = {t1, t2, ...t i}(i=1, 2, ...s), subset t i ={tstr i1 , tStr i2 ,...tstr ij ,...}, (i=1, 2,...s, j=12,...n, tstr ij represents the jth domain name in the i-th subset), the target domain name set P = {pstr1, pstr2, ...pstr k}(k=1,2,…m,pstr k represents the kth domain name);

[0106] 2) Create all working nodes in the cluster and start the parallel computing cluster;

[0107] 3) Parallelize and assign the s subsets to the working nodes of the cluster;

[0108] 4) Submit the AC algorithm matching task for calculation.

[0109] After the above process is prepared, execute the packaged code. In the Spark distributed cluster, there are generally three ways to execute the code: ①spark-shell mode, open the Spark running terminal, and then enter the corresponding command to execute; ②spark-submit mode, write the written code path and corresponding execution parameters, and then submit the execution task; ③ run the packaged execution parameters and data path code through the python3 running instruction. The embodiment of the present invention uses the third method. Figure 7 It can be seen that using the parallel processing method in Spark computing can improve the efficiency of big data processing and shorten the running time.

[0110] In summary, the embodiment of the present invention adopts the highly available Spark distributed cluster of Hadoop in the above steps 201 to 206 to ensure high fault-tolerant storage and high-efficiency real-time calculation of massive DNS log data.

[0111] Example 3

[0112] The following combination Figure 8 and Figure 9 The feasibility of the schemes in Examples 1 and 2 was verified, as described below:

[0113] pass Figure 8 As you can see, when executing Spark tasks, you can view the tasks and judge the running status from the browser interface, and confirm the execution status of the job by viewing the history of each job, so as to timely adjust the corresponding parameters of the program and the relevant parameters of the Spark and Hadoop configuration files.

[0114] The present invention also compares the performance of other methods. Figure 9As can be seen, as the number of CNAME domains in DNS logs increases, matching time also increases accordingly. However, this method (a method for quickly matching DNS network domain data, abbreviated as PPEACS) has the shortest execution time among many algorithms and is stable. Furthermore, as the number of CNAME domains in DNS doubles, matching time does not double. When the number of CNAME domains in DNS reaches 376,800, the PPEACS method's single CNAME domain matching speed is approximately seven times that of the original AC automaton algorithm. Clearly, this method has significant advantages over the AC automaton algorithm and is more efficient. Similarly, compared to the improved AC-BM algorithm (an improved combination of the AC automaton algorithm and the BM algorithm), the speed is approximately two to three times faster, which also shows its advantages.

[0115] Example 4

[0116] A device for quickly matching DNS network domain name data, see Figure 10 The device includes: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to perform the following method steps:

[0117] Implement keyless secure shell protocol communication, cluster clock synchronization, and firewall settings between servers that build multiple Linux systems;

[0118] On the Linux platform, decompress and install the compressed package to build a highly available distributed cluster. The cluster is used to implement a multi-node design for the name node process in the distributed file system and the resource management process in the resource manager. If the primary node fails, the standby node switches to active mode.

[0119] Select the resource manager cluster mode for processing DNS log data stored in a distributed file system, optimize operators on elastic datasets of directed acyclic graphs, and design Spark operating parameters;

[0120] Adopting the improved double array dictionary tree, using the breadth-first search algorithm to calculate the fail() pointer, and connecting each parent and child node according to the preset rules to create a matching function;

[0121] The parallel Python module is installed, and multi-threaded parallel computing and an enhanced AC automaton algorithm are combined on the cluster to perform matching calculations on the massive CNAME domain names in the DNS log data with the specified domain name library.

[0122] Cluster clock synchronization is as follows:

[0123] Use the time synchronization command to install the time synchronization plug-in on the server; use the time write command to write the system time to the hard disk; finally, use the time synchronization command to enter in the terminal to ensure time synchronization between different machines in the cluster.

[0124] Furthermore, the firewall setting operations are as follows:

[0125] Query the firewall status between each server; if the firewall status is active, disable the firewall of each server;

[0126] Ensure that the cluster firewall is permanently disabled by editing the file and changing the value of selinux (the Linux kernel's security subsystem) to disable.

[0127] The cluster is used to implement multi-node design for the name node process in the distributed file system and the resource management process in the resource manager. Specifically:

[0128] ① After decompressing and installing the Hadoop software installation package on the server, design the corresponding parameters in the configuration file under the installation directory; ② Distribute the Hadoop installation directory with the modified configuration file to other server nodes; ③ Then configure the JDK installation path in the yarn-env.sh file; ④ Open the startup command and start the JournalNode (a daemon process that keeps the standby node synchronized with the running master node) service on all nodes; ⑤ Initialize the NameNode on the master node; ⑥ Use the command to initialize the Hadoop high availability status in Zookeeper; ⑦ Start the HDFS command and YARN command on the master node; ⑧ Start the DFSZK Failover Controller on each NameNode node; ⑨ Check whether the process service command is normal.

[0129] Furthermore, the improved double array dictionary tree is:

[0130] The double array dictionary tree includes: base[] array and check[] array. Each element of the base array represents a dictionary tree node, and each element of the check array represents the predecessor state of a state, which is used to check whether the state transfer is correct.

[0131] base[s]+c=t

[0132] check[t]=s

[0133] Among them, s is the subscript of the current state, i.e. the parent node, t is the subscript of the transition state, i.e. the child node, and c is the encoding value of the input character. Based on the double array dictionary tree, the tail[] array is introduced to compress the non-public prefixes of the dictionary tree. The negative value of base[s] indicates a non-public prefix, and the absolute value indicates that the character suffix corresponding to the node is at the starting position of the tail[] array. If the tail string corresponding to the node s is b1, b2, ... b k , satisfying base[s]<0, then the corresponding string b1…b k Put it into the tail[] array.

[0134] pos=|-base[sl|

[0135] tail[pos+k-1]=b k (k≥1)

[0136] Furthermore, the operation of the enhanced AC automaton is:

[0137] Among them, for the improved double array dictionary tree, the breadth-first search algorithm is used to calculate the fail() pointer, and each parent and child node is connected according to the preset rules to create a matching output function:

[0138] Traverse the input string, c is the character, sn is the state, n starts from 0, s0=0;

[0139] First press goto to jump: s(n+1)=base[sn]+c+1. If check[s(n+1)]=base[sn], the jump is successful and returns s(n+1). If the jump fails, check if s0==0. If so, return 0. Otherwise, continue to jump according to the fail pointer.

[0140] Jump according to the fail pointer, s(n+1)=base[fail[sn]]+c+1. If check[s(n+1)]=base[sn], the jump is successful and returns s(n+1). If the jump fails, check whether s0==0. If so, return 0. Otherwise, continue to jump according to fail.

[0141] Verify the output table based on the return value. If output[s(n+1)]!=null, it means the match is successful, and all matching pattern strings are returned.

[0142] Continue processing with the next character.

[0143] Specifically, the multi-threaded parallel computing and enhanced AC automaton algorithm on the cluster are as follows:

[0144] 1) Split the DNS log domain name T into s subsets, namely T = {t1, t2, ...t i}, i = 1, 2, ... s, subset t i ={tstr i1 , tstr i2 ,...tstr ij ,...},i=1,2,…s,j=12,…n,tstr ij represents the jth domain name in the i-th subset, the target domain name set P = {pstr1, pstr2, ...pstr k}, k = 1, 2, ... m, pstr k represents the kth domain name;

[0145] 2) Create all working nodes in the cluster and start the parallel computing cluster;

[0146] 3) Parallelize and assign the s subsets to the working nodes of the cluster;

[0147] 4) Submit the AC algorithm matching task for calculation.

[0148] It should be noted here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0149] The execution subjects of the above-mentioned processor 1 and memory 2 can be computers, single-chip microcomputers, microcontrollers and other devices with computing functions. In specific implementation, the embodiment of the present invention does not limit the execution subject and it is selected according to the needs of actual application.

[0150] Data signals are transmitted between the memory 2 and the processor 1 via the bus 3 , which will not be described in detail in the embodiment of the present invention.

[0151] Example 5

[0152] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored program, and when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiment.

[0153] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0154] It should be noted here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0155] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part.

[0156] The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted via computer-readable storage media. Computer-readable storage media can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. Available media can include magnetic media or semiconductor media, etc.

[0157] Unless otherwise specified, the embodiments of the present invention do not limit the models of the components. Any component that can perform the above functions may be used.

[0158] Those skilled in the art will understand that the accompanying drawings are only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for quickly matching DNS network domain name data, characterized in that: The method comprises: Implement keyless secure shell protocol communication, cluster clock synchronization, and firewall settings between servers that build multiple Linux systems; On a Linux platform, decompress and install the compressed package to build a high-availability distributed Hadoop cluster. The cluster is used to implement a multi-node design for the name node process in the distributed file system and the resource management process in the resource manager. If the primary node fails, the backup node switches to active mode. Select the resource manager cluster mode for processing DNS log data stored in the distributed file system, optimize operators on the elastic dataset of the directed acyclic graph, and design Spark job operation parameters; Adopting the improved double array dictionary tree, using the breadth-first search algorithm to calculate the fail() pointer, and connecting each parent and child node according to the preset rules to create a matching function; Install the parallel Python module and combine multi-threaded parallel computing and the enhanced AC automaton algorithm on the cluster to match the massive CNAME domain names in the DNS log data with the specified domain name library; Wherein, the improved double array dictionary tree is: The double array dictionary tree includes: base[] array and check[] array. Each element of the base array represents a dictionary tree node, and each element of the check array represents the predecessor state of a state, which is used to check whether the state transfer is correct. base[s]+c=t; check[t]=s; Where s is the subscript of the current state, i.e. the parent node; t is the subscript of the transition state, i.e. the child node; and c is the encoding value of the input character. Based on the double-array dictionary tree, the tail[] array is introduced to compress the non-public prefixes of the dictionary tree. The negative value of base[s] indicates a non-public prefix, and the absolute value indicates that the character suffix corresponding to the node is at the starting position of the tail[] array. If the tail string corresponding to the node s is b1, b2, ... b k , satisfying base[s]<0, then the corresponding string b1…b k Put it into the tail[] array: pos=|-base[s]|; tail[pos+k-1]=b k (k≥1)。 2. The method for quickly matching DNS network domain name data according to claim 1, characterized in that: The cluster clock synchronization is specifically as follows: Use the time synchronization command to install the time synchronization plug-in on the server; use the time write command to write the system time to the hard disk; finally, use the time synchronization command to enter in the terminal to ensure time synchronization between different machines in the cluster.

3. The method for quickly matching DNS network domain name data according to claim 1, characterized in that: The firewall setting operation is specifically as follows: Query the firewall status between each server; if the firewall status is active, disable the firewall of each server; Ensure that the cluster firewall is permanently closed and edit the file to change the value of selinux to disable.

4. The method for quickly matching DNS network domain name data according to claim 1, characterized in that: The cluster is used to perform multi-node design on the name node process in the distributed file system and the resource management process in the resource manager. Specifically: ① After decompressing and installing the Hadoop software package on the server, design the corresponding parameters for the configuration files in the installation directory; ② Distribute the Hadoop installation directory with the modified configuration files to other server nodes; ③ Then configure the JDK installation path in the yarn-env.sh file; ④ Open the startup command and start the JournalNode service on all nodes; ⑤ Initialize the NameNode on the master node; ⑥ Use the command to initialize the Hadoop high availability status in Zookeeper; ⑦ Start the HDFS command and YARN command on the master node; ⑧ Start the DFSZKFailover Controller on each NameNode node; ⑨ Check whether the process service command is normal.

5. The method for quickly matching DNS network domain name data according to claim 1, characterized in that: The enhanced AC automaton is to calculate the fail() pointer using the breadth-first search algorithm for the improved double-array dictionary tree, and connect each parent and child node according to the preset rules to create a matching output function: Traverse the input string, c is the character, sn is the state, n starts from 0, s0=0; First press goto to jump: s(n+1)=base[sn]+c+1. If check[s(n+1)]=base[sn], the jump is successful and returns s(n+1). If the jump fails, check if s0==0. If so, return 0. Otherwise, continue to jump according to the fail pointer. Jump according to the fail pointer, s(n+1)=base[fail[sn]]+c+1. If check[s(n+1)]=base[sn], the jump is successful and returns s(n+1). If the jump fails, check whether s0==0. If so, return 0. Otherwise, continue to jump according to fail. Verify the output table based on the return value. If output[s(n+1)]!=null, it means the match is successful, and all matching pattern strings are returned. Continue processing with the next character.

6. A method for quickly matching DNS network domain name data according to claim 1, characterized in that: The multi-threaded parallel computing and enhanced AC automaton algorithm on the cluster are specifically as follows: 1) Split the DNS log domain name T into s subsets T = {t1, t2, ...t i }, i = 1, 2, ... s, subset t i ={tstr i1 ,tstr i2 ,...tstr ij ,...}, 1=1, 2,...s, j=12,...n,tstr ij represents the jth domain name in the i-th subset, the target domain name set P = {pstr1, pstr2, ...pstr k }, k = 1, 2, ... m, pstr k represents the kth domain name; 2) Create all working nodes in the cluster and start the parallel computing cluster; 3) Parallelize and assign the s subsets to the working nodes of the cluster; 4) Submit the AC algorithm matching task for calculation.

7. A device for quickly matching DNS network domain name data, characterized in that: The device includes: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to perform the method steps according to any one of claims 1 to 6.

Images