A large-scale DNS risk situation real-time monitoring system supporting heterogeneous probe terminals

Abstract

The application discloses a large-scale DNS risk situation real-time monitoring system supporting heterogeneous detection terminals and belongs to the field of DNS risk situation awareness, and specifically comprises a detection node cluster, a data collection module, a data cleaning and statistics module and a DNS risk situation report module; initial configuration of detection tasks of each detection node, deployment of detection nodes by using different request modes, real-time monitoring of response messages of the detection tasks for format processing, attachment of geographic positions of the detection nodes, task time stamps and DNS resolution, uploading to a Kafka cluster of the data collection module, collection of server response data corresponding to the tasks, storage and pushing to the data cleaning and statistics module, extraction of useful fields, i.e., cleaning and reduction, calculation of risk situation statistical indexes, transmission to the DNS risk situation report module, query of corresponding statistical indexes for users and intuitive reflection of DNS risk situations in regions. The application reduces the risk of identification of detection features and improves monitoring concealment.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of DNS risk situation awareness of network security, and particularly relates to a large-scale DNS risk situation real-time monitoring system supporting heterogeneous detection terminals. BACKGROUND

[0002] With the rapid development of the Internet, the domain name system (DNS) as an important infrastructure supporting the normal operation of the Internet, carries the key function of domain name resolution; however, while providing services, DNS has also become a potential target of network attacks.

[0003] In recent years, the security threats to DNS have been increasing, especially the risks of domain name hijacking and service suspension, which have seriously affected the availability of network services and data security. For websites with large daily access and important role in society, such as online financial transaction systems or well-known search engines, once they encounter domain name hijacking or service suspension attacks, it is likely to cause serious economic losses to individuals or enterprises, and the harm cannot be ignored; therefore, it is very important to timely and accurately monitor the domain name risk situation of important websites.

[0004] At present, there are many domain name risk situation monitoring systems for enterprises or individuals on the market, which generally have the following problems:

[0005] (1) The domain name risk situation detection system for personal application needs the user to manually input the domain name to be detected, and the response time is relatively long, lacking automatic monitoring and retrieval functions, which is inconvenient for users to use.

[0006] (2) Most domain name risk situation monitoring systems only use servers as detection terminals, lacking task adaptability to heterogeneous terminals.

[0007] With the development of edge computing and fog computing technology, more and more mobile terminal devices are used to perform network probing and computing tasks. Compared with server terminal nodes, mobile terminal nodes have the characteristics of low cost and easy large-scale deployment, but the computing performance is relatively weak. Therefore, considering the comprehensive cost, detection scale and performance, it is a suitable strategy to deploy multiple architecture (including server architecture, mobile terminal device architecture) detection nodes in the risk situation monitoring system.

[0008] For a cluster of heterogeneous terminal detection nodes, it is particularly important to uniformly collect and analyze the data packets from these distributed and different architecture nodes.

[0009] In big data processing and analysis applications, Kafka is a widely used message queue middleware because it has high throughput and can be distributedly deployed, and can be integrated with server terminals and embedded terminals; among all big data processing frameworks, Apache Flink is the most powerful distributed framework for real-time computing, and can be connected with Kafka to realize efficient stream data processing and analysis, and is very suitable for data analysis tasks with high real-time requirements. SUMMARY

[0010] In order to solve the problems of low automation, single terminal type and insufficient real-time of the existing DNS risk monitoring system, the application proposes a large-scale DNS risk situation real-time monitoring system supporting heterogeneous detection terminals, which realizes the automatic, large-scale and real-time monitoring of large-scale DNS risk situation by combining heterogeneous detection terminals, distributed message queues and stream processing frameworks.

[0011] The large-scale DNS risk situation real-time monitoring system supporting heterogeneous detection terminals specifically comprises the following modules: a detection node cluster, a data acquisition module, a data cleaning and statistical module, and a DNS risk situation report module.

[0012] The detection node module is responsible for executing detection tasks, the data acquisition module acquires corresponding server response data of the task, stores and transmits it to the data cleaning and statistical module, cleans and reduces the server response data to obtain real-time DNS risk situation index data, and transmits it to the DNS risk situation report module to present the risk situation index data in the form of a user-friendly visual interface, and intuitively reflect the DNS risk situation in each region.

[0013] The detection node module adopts a hierarchical storage system (HSM-like) architecture, which divides the heterogeneous detection terminals into three layers: an edge layer, a fog computing layer and a cloud computing layer.

[0014] The edge layer is composed of lightweight IoT devices and mobile devices; it can quickly access any specified location, execute periodic basic detection tasks with low computing load, and support large-scale deployment, thereby supporting large-scale detection tasks.

[0015] The fog computing layer is composed of distributed server nodes and is deployed in regional data centers or edge clouds; it is responsible for processing tasks with medium complexity and aggregating edge layer data.

[0016] The cloud computing layer is composed of a high-performance cloud server cluster and supports large-scale distributed deployment; it is responsible for global data aggregation, complex analysis and data persistent storage.

[0017] The data collection module is based on Kafka implementation, in the Kafka cluster, deploy at least 3 broker nodes, set 2 theme queue dig and ping, respectively store Dig task and Ping task response data packet, and set the number of copies and the number of partitions according to actual business requirements;

[0018] The data cleaning and statistical module takes Flink as a framework, first extracts the fields required for analysis from the original response data, then from the two dimensions of geographic location and DNS resolver, the website domain name resolution index is counted, including resolution accuracy, recall rate and average TTL, and the website domain name access connectivity from the geographic location dimension is analyzed, including the number of normal response status codes, average RTT, and the statistical result data is persisted.

[0019] The DNS risk situation report module is used to read real-time DNS risk situation statistical index data, and present it to the user through the front end, so that the user can overview the overall DNS risk situation of each region, and also can query the website domain name risk situation he cares about, including domain name resolution accuracy, recall rate, response delay index, so as to analyze the domain name hijacking risk and offline risk of the website.

[0020] The large-scale DNS risk situation real-time monitoring system supporting heterogeneous detection terminals has the following working principle:

[0021] Step one, initial configuration of the detection task of each detection node;

[0022] Including detection domain name, detection DNS resolver, detection task type (Dig / Ping) and detection frequency.

[0023] The detection task uses unstructured storage method, and uses Redis database as storage engine.

[0024] Step two, for different detection node terminal types, different request methods are used to realize the deployment of detection nodes;

[0025] The detection node terminal type includes server, IoT device and mobile device.

[0026] For server detection terminal, use Python dnslib library to realize DNS request sending, scapy library to realize Ping request sending, and use Docker containerization technology to realize large-scale server end detection node deployment;

[0027] For IoT devices, network probing is realized based on C / C++ embedded development technology, including DNS request detection and Ping request detection;

[0028] For mobile devices, a network stress testing module with migratory deployment is implemented using the Javascript language.

[0029] Step three, the detection node monitors the response packet of the detection task in real time, and performs JSON formatting processing, and attaches the geographic position of the detection node, the detection task timestamp and the detected DNS resolver, and uploads to the Kafka cluster of the data collection module.

[0030] Step four, the Kafka cluster receives the data uploaded by the detection node in real time, and pushes it to the data cleaning and statistical module.

[0031] Step five, the data cleaning and statistical module extracts useful fields from the data pushed by the Kafka cluster according to business needs, obtains cleaned data, and then aggregates the cleaned data according to the required dimensions, so as to calculate risk situation statistical indicators and save them to the statistical result database.

[0032] The useful fields include timestamp, detection domain name, detection node geographic position, accessed DNS resolver, record type, TTL, resolution result, response status code and RTT field;

[0033] The risk situation statistical indicators include precision (Precision), recall (Recall), F1 score (F1), risk coefficient (Risk),

[0034] The calculation formulas of each indicator are as follows:

[0035]

[0036] Risk=1-αPrecision-βRecall-yF1

[0037] Where TP represents the number of correct resolution results returned by the DNS resolver (i.e. the number of records in the trusted record library), FP represents the number of incorrect resolution results returned by the DNS resolver (i.e. the number of records not in the trusted record library), and FN represents the number of records in the trusted record library that have not been resolved. α, β and γ are risk assessment weighting coefficients, satisfying α+β+γ=1.

[0038] Step six, the DNS risk situation report module queries the statistical result database through SQL statements and presents the corresponding statistical indicators to the user.

[0039] The presentation forms include regional DNS risk heat map, Top 10 high-risk domain name list, domain name history hijacking rate curve, resolution accuracy and recall rate comparison chart and the like. Meanwhile, the module provides a search bar, and a user can input a website domain name of interest, and the backend sends an HTTP GET request to an ElasticSearch component, performs fuzzy matching on the query issued by the user, and returns the matched risk situation record data to the module front end for presentation.

[0040] The present application has the following advantages:

[0041] (1) Efficiency: the system uses high-throughput distributed message queue middleware Kafka as the internal communication medium, and uses high-performance stream processing distributed computing framework Flink as the data cleaning and statistical analysis tool, which fully guarantees the computing performance, availability, scalability and reliability of the system, and guarantees the real-time performance of the data statistical results, and meets the real-time DNS risk situation statistical requirements;

[0042] (2) Flexibility: the system is based on HSM-like architecture, supports multiple terminal types such as servers, mobile devices and IoT devices, and automatically allocates tasks according to terminal capabilities, so as to achieve a good trade-off between detection performance and node deployment cost, maximize resource utilization, and customize the detection scale according to actual detection requirements. In addition, the data collection module and the data processing module in the system are separated from each other, and have good maintainability. Developers can customize data processing logic and add required data index statistical logic at any time according to user detection requirements.

[0043] (3) Anti-countermeasure capability: through the layered camouflage strategy and diversified request behavior, the risk of detection features being identified is reduced, and the monitoring concealment is improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 Figure 1 is an architecture diagram of a large-scale DNS risk situation real-time monitoring system supporting heterogeneous detection terminals according to the present application. DETAILED DESCRIPTION

[0045] The specific implementation method of the present application will be further described in detail below in combination with the drawings.

[0046] The application provides a large-scale DNS risk situation real-time monitoring system supporting heterogeneous detection terminals, based on a Kafka+Flink framework, providing real-time domain name risk situation monitoring for important websites, for personal or enterprise users to real-time search the website domain name state being monitored. The application supports dynamic allocation of detection terminals according to task characteristics, including IoT cards and mobile terminals (suitable for high-frequency, low-computing tasks, with the advantages of fast deployment and easy formation of large-scale detection clusters), cloud servers (suitable for tasks requiring large-scale parallel computing, with the advantages of high performance and large data storage capacity), and fog computing hosts (capable of undertaking regional-level tasks and sharing the load of cloud servers), which can take advantage of various terminals. In addition, the application introduces a detection traffic camouflage mechanism, thereby avoiding detection behavior from being identified and suppressed, and ensuring the normal operation of the detection task.

[0047] The system described in the application realizes efficient and real-time monitoring and analysis of DNS risk situation by combining heterogeneous detection terminals, a distributed message queue (Kafka) and a stream processing framework (Flink). Specifically, it includes the following modules: a detection node cluster, a data collection module, a data cleaning and statistical module, and a DNS risk situation report module.

[0048] The detection node cluster is responsible for executing detection tasks, the data collection module collects corresponding server response data, provides data storage space and data transmission mechanism, stores and transmits the data to the data cleaning and statistical module, cleans and reduces the latest received server response data, obtains real-time DNS risk situation index data, transmits the data to the DNS risk situation report module, presents the risk situation index data in the form of a user-friendly visual interface, and intuitively reflects the DNS risk situation in each region.

[0049] The detection node cluster supports multiple terminal types such as servers, mobile devices and IoT devices, executes DNS detection tasks and collects server response packets; adopts a hierarchical storage system (HSM-like) architecture, divides the heterogeneous detection terminals into three layers: edge layer, fog computing layer and cloud computing layer; dynamically allocates detection tasks according to performance, deployment flexibility and task complexity.

[0050] The edge layer is composed of lightweight IoT devices (such as network probing cards) and mobile devices (such as smart phones); such terminals are flexible to deploy, can quickly access any specified location (such as enterprise intranet, home gateway), execute low-computing-load periodic basic detection tasks (such as high-frequency Ping detection, DNS resolution state monitoring), and have relatively low cost, supporting large-scale deployment and thus supporting large-scale detection tasks.

[0051] The fog computing layer is composed of distributed server nodes, deployed in regional data centers or edge clouds; responsible for processing tasks of medium complexity (such as multi-dimensional DNS resolution result comparison, regional risk situation pre-analysis), and aggregating edge layer data.

[0052] The cloud computing layer is composed of high-performance cloud server clusters, supporting large-scale deployment across different regions; responsible for global data aggregation, complex analysis, and data persistent storage.

[0053] The data collection module is based on Kafka implementation, using Kafka message queue middleware as the data transmission medium, receiving and storing data packets from heterogeneous detection terminals, ensuring efficient transmission and reliable storage of data. The internal communication model uses the "producer-consumer" model. The detection node acts as a "producer" and uploads the received DNS resolution response messages and Ping response messages to the Kafka message queue. The Flink cluster acts as a "consumer" and listens to the Kafka message queue in real time to obtain the latest messages (i.e., real-time DNS resolution response messages and Ping response messages), and processes the responses.

[0054] The data cleaning and statistical module uses the Apache Flink distributed data processing framework, a relational database, and ElasticSearch as the data storage engine to extract data from the Kafka cluster, perform real-time cleaning, analysis, and calculation, and generate DNS risk situation statistical results. First, extract the fields needed for analysis from the original response data, including timestamp, detection domain name, detection node geographic location, accessed DNS resolver, record type, TTL, resolution result, response status code, and RTT field. Then, from the geographic location and DNS resolver dimensions, statistical website domain name resolution indicators are generated, including resolution accuracy, recall rate, and average TTL. From the geographic location dimension, website domain name access connectivity is analyzed, including the number of normal response status codes and average RTT. The statistical result data is persistently stored.

[0055] The DNS risk situation report module is used to query and read real-time DNS risk situation statistical indicator data, and present a user-friendly data large-screen interface through BI tools, Vue, EChart, and other front-end visualization rendering frameworks, making it easy for users to overview the overall DNS risk situation in various regions, and also query the DNS risk situation of their own concerned website domain names, including domain name resolution accuracy, recall rate, and response delay indicators, to analyze the domain name hijacking risk and offline risk faced by the website.

[0056] The large-scale DNS risk situation real-time monitoring system supporting heterogeneous detection terminals works as follows:

[0057] Step one, initially configure the detection tasks of each detection node;

[0058] The detection domain name, the detection DNS resolver, the detection task type (Dig / Ping), and the detection frequency are included.

[0059] The detection task adopts an unstructured storage mode, and uses a Redis database as a storage engine.

[0060] Step two, for different detection node terminal types, different request modes are used to realize the deployment of the detection node;

[0061] The detection node terminal type includes a server, an IoT device, and a mobile device.

[0062] For a server detection terminal, a Python dnslib library is used to realize DNS request sending, a scapy library is used to realize Ping request sending, and a Docker container technology is used to realize large-scale server-side detection node deployment;

[0063] For an IoT device, a C / C++ embedded development technology is used to realize network probing, including DNS request detection and Ping request detection;

[0064] For a mobile terminal device, a Javascript language is used to realize a network probing module that can be migrated and deployed; the realized network probing module can be used as an extension of a browser or other application software, and works in a background running mode.

[0065] Step three, the detection node monitors the response message of the detection task in real time, and performs JSON format processing, and attaches the geographical position of the detection node, the detection task timestamp, and the detected DNS resolver, and uploads to the Kafka cluster of the data collection module.

[0066] Step four, the Kafka cluster serves as an intermediate communication medium, receives the data uploaded by the detection node in real time, and pushes to the data cleaning and statistical module.

[0067] Step five, the data cleaning and statistical module extracts useful fields from the data pushed by the Kafka cluster according to business requirements, obtains cleaned data, and then aggregates the cleaned data according to the required dimensions, so as to calculate risk situation statistical indicators, and saves to a statistical result database.

[0068] The useful fields include a timestamp, a detection domain name, a detection node geographical position, an accessed DNS resolver, a record type, a TTL, a resolution result, a response status code, and an RTT field;

[0069] The specific cleaning process is as follows:

[0070] Firstly, Flink cluster is built, and then data cleaning and statistical analysis program code is written in Java language, and the code is compiled, packaged and submitted to Flink cluster to start data processing and analysis task.

[0071] The risk situation statistical indicators include precision, recall, F1 score and risk coefficient.

[0072] Step six, the DNS risk situation report module sends a query to the statistical result database through a SQL statement, and presents the corresponding statistical indicators to the user.

[0073] The presentation forms include regional DNS risk heat map, Top 10 high-risk domain name list, domain name historical hijacking rate curve, resolution precision and recall rate comparison chart, etc. At the same time, the module provides a search bar, and the user can input the website domain name of interest. The backend sends an HTTP GET request to the ElasticSearch component, performs fuzzy matching on the user's query, and returns the matching risk situation record data to the module front end for presentation.

[0074] Embodiment:

[0075] The specific implementation of each module in this example is as follows:

[0076] (1) Deployment of detection node

[0077] In this system, the types of detection node terminals include servers, IoT devices and mobile devices.

[0078] Before performing the detection task, the detection node reads the detection task configured in advance (including detection domain name, detection DNS resolver, detection task type (Dig / Ping), detection frequency). The detection task uses an unstructured storage method, and uses Redis database as the storage engine.

[0079] After receiving the detection task response message, the detection node formats the response message in JSON format, and attaches the geographical position of the detection node, the detection task timestamp, and the detected DNS resolver (only add this field in Dig task), and uploads it to the Kafka cluster.

[0080] In order to avoid detection behavior from being identified, the system simulates real user behavior through the following means:

[0081] ① Device fingerprint camouflage

[0082] In the IoT card, the MAC address of the common smart hardware (such as router, camera) and the HTTP request header are simulated to pretend to be a normal IoT device; in the mobile terminal, the device ID is randomly generated, and the User-Agent (simulating different brand mobile browser) is dynamically switched; in the cloud, the IP pool rotation technology is adopted to simulate the distributed real user IP distribution.

[0083] ②Request behavior diversification

[0084] After DNS resolution, the subsequent HTTP request (GET / POST) is initiated according to the probability, so as to simulate the complete access link, and the detection frequency is dynamically adjusted to avoid fixed time interval triggering anti-crawling mechanism.

[0085] ③Traffic feature confusion

[0086] The edge layer device injects noise data (such as random DNS query) according to the region, so as to hide the core detection target.

[0087] (2) Data acquisition module deployment

[0088] The data acquisition module of the system is realized based on Kafka. In the Kafka cluster, at least 3 broker nodes are deployed, 2 topic queues dig and ping are set, which store Dig task and Ping task response data packets respectively, and distributed replica mechanism (according to actual business demand to set the number of replicas and partitions) is adopted to guarantee the system data reliability and access performance. The server-side detection node transmits data packets to Kafka through Kafka Producer API, and the IoT terminal and mobile terminal adopt lightweight MQTT protocol to bridge with Kafka to reduce resource occupation.

[0089] (3) Data cleaning and statistical module

[0090] The data cleaning and statistical module of the system is realized based on Flink, and YARN component is used for data calculation task scheduling. The developer first builds Flink cluster, then uses Java language to write data cleaning and statistical analysis program code, compiles and packages the code, and submits it to Flink cluster to start data processing and analysis task.

[0091] The Java code implementation of data cleaning part is as follows:

[0092]

[0093] The data statistical analysis code is as follows:

[0094]

[0095]

[0096] The data processing logic in this module is implemented based on Flink DataStream API, which is divided into two links: data cleaning and reduction statistics.

[0097] Data cleaning is to extract useful fields from the original data packet for analysis according to business needs, and reduction statistics is to aggregate the cleaned data according to the dimensions of the analysis needed, so as to calculate the risk situation statistical indicators. In order to facilitate traceability analysis, the data packet after data cleaning and the reduction statistics result are stored in a structured manner, and the storage engine can be MySQL or Oracle relational database.

[0098] For Dig tasks, the fields extracted by data cleaning and the value types and meanings saved in the database are shown in Table 1.

[0099] Table 1

[0100]

[0101]

[0102] After obtaining the cleaned data packet, the data analysis program takes time (timestamp), city location (city), and resolver as analysis dimensions, and counts the total DNS resolution result type (type), number (total_answers) received at the current time point, and compares it with the trusted DNS resolution record library to calculate the total trusted record number (total_creds) of the current domain name, the number of correct resolution results (true_answers). The statistical result is recorded and stored in a relational database.

[0103] The stored field information is shown in Table 2

[0104] Table 2

[0105]

[0106] For Ping task records, the data analysis program first extracts the fields, i.e. the fields retained after cleaning the Ping response data packet, as shown in Table 3:

[0107] Table 3

[0108]

[0109] After obtaining the cleaned data packet, the data analysis program performs reduction statistics on it to obtain the statistical result information as shown in Table 4:

[0110] Table 4

[0111]

[0112] (Note: The number of abnormally received messages refers to the sum of the number of requests that do not receive a response and the number of response messages with abnormal status codes; for the PING request that does not receive a response, the rtt field will be recorded as the system TIMEOUT duration)

[0113] In order to facilitate users to retrieve the domain name state they care about, the system uses high-performance distributed search engine middleware ElasticSearch to connect with the search engine module, and automatically synchronizes the statistical result data saved in the relational database to ElasticSearch.

[0114] (4) DNS risk situation report module

[0115] This module issues a query to the statistical result database through a SQL statement and presents the corresponding statistical indicators. For the domain name hijacking risk situation, this module presents the user with four indicators: precision, recall, F1 score, and risk coefficient, whose corresponding calculation formulas are as follows

[0116]

[0117] Risk = 1 - a Precision - b Recall - yF1

[0118] Where TP represents the number of correct resolution results returned by the DNS resolver (i.e., the number of records in the trusted record library), FP represents the number of incorrect resolution results returned by the DNS resolver (i.e., the number of records not in the trusted record library), and FN represents the number of records in the trusted record library that have not been resolved. a, b, and g are risk assessment weighting coefficients, satisfying a + b + g = 1.

[0119] After obtaining the corresponding indicator data, the module uses the EChart framework to render the data large-screen interface and present the queried indicator data.

Claims

1. A large-scale DNS risk posture real-time monitoring system supporting heterogeneous probe terminals, characterized in that, It comprises a cluster of probe nodes, a data collection module, a data cleaning and statistics module, and a DNS risk situation report module. The cluster of probe nodes is responsible for performing probe tasks, the data collection module collects the corresponding server response data of the tasks, stores it and transmits it to the data cleaning and statistics module, the server response data is cleaned and reduced to obtain real-time DNS risk situation index data, which is transmitted to the DNS risk situation report module, and the risk situation index data is presented in the form of a user-friendly visual interface to intuitively reflect the DNS risk situation in each region. The data collection module is implemented based on Kafka, in the Kafka cluster, at least three broker nodes are deployed, two topic queues dig and ping are set to store Dig task and Ping task response data packets respectively, and the number of replicas and the number of partitions are set according to actual business requirements. The data cleaning and statistics module uses Flink as the framework, first extracts the fields needed for analysis from the original response data, then statistically analyzes the website domain name resolution indicators from the two dimensions of geographic location and DNS resolver, including resolution accuracy, recall rate and average TTL, and analyzes the website domain name access connectivity from the geographic location dimension, including the number of normal response status codes and the average RTT. The statistical result data is persistently stored.

2. The large-scale DNS risk posture real-time monitoring system supporting heterogeneous probe terminals according to claim 1, wherein, The probe node cluster adopts a hierarchical storage architecture, and divides the heterogeneous probe terminals into three layers: an edge layer, a fog computing layer and a cloud computing layer. The edge layer is composed of lightweight IoT devices and mobile devices; it can quickly access any specified location, perform periodic basic probe tasks with low computing load, and support large-scale deployment, thereby supporting large-scale probe tasks. The fog computing layer is composed of distributed server nodes and is deployed in regional data centers or edge clouds; it is responsible for processing tasks with medium complexity and aggregating edge layer data. The cloud computing layer is composed of a high-performance cloud server cluster and supports large-scale deployment; it is responsible for global data aggregation, complex analysis and data persistent storage.

3. The large-scale DNS risk posture real-time monitoring system supporting heterogeneous probe terminals according to claim 1, wherein, The DNS risk situation report module is used to read real-time DNS risk situation statistical index data and present it to users through the front end, making it easy for users to overview the overall DNS risk situation in each region, and also allowing users to query the risk situation of their concerned website domain names, including domain name resolution accuracy, recall rate and response delay indicators, so as to analyze the domain name hijacking risk and offline server risk faced by the website.

4. The large-scale DNS risk posture real-time monitoring system supporting heterogeneous probe terminals according to claim 1, wherein, The working principle of the system is as follows: Step one, initially configure the probe tasks of each probe node; It includes probe domain name, probe DNS resolver, probe task type Dig / Ping and probe frequency; Step two, for different types of probe node terminals, different request methods are used to implement the deployment of probe nodes; The probe node terminal types include servers, IoT devices and mobile devices; Step three, the probe node real-time monitors the response message of the probe task and performs JSON formatting processing, and attaches the geographic location of the probe node, the probe task timestamp and the probe DNS resolver, and uploads it to the Kafka cluster of the data collection module. Step four, the Kafka cluster receives the data uploaded by the probe node in real time and pushes it to the data cleaning and statistical module; Step five, the data cleaning and statistical module extracts useful fields from the data pushed by the Kafka cluster according to business requirements, obtains cleaned data, and then aggregates the cleaned data according to the required dimensions, so as to calculate risk situation statistical indicators and save them to the statistical result database; The risk situation statistical indicators include precision, recall, F1 score, and risk coefficient. Step six, the DNS risk situation report module queries the statistical result database through SQL statements and presents the corresponding statistical indicators to the user.

5. The large-scale DNS risk posture real-time monitoring system supporting heterogeneous probe terminals according to claim 4, wherein, In step one, the probe task uses an unstructured storage method and uses Redis database as the storage engine.

6. The large-scale DNS risk posture real-time monitoring system supporting heterogeneous probe terminals according to claim 4, wherein, In step two, for server probe terminals, the DNS request sending is implemented using the Python dnslib library, the Ping request sending is implemented using the scapy library, and the large-scale server probe node deployment is implemented using the Docker containerization technology; For IoT devices, network probing is implemented based on C / C++ embedded development technology, including DNS request probing and Ping request probing; For mobile devices, the network probing module is implemented using Javascript language for migratable deployment.

7. The large-scale DNS risk posture real-time monitoring system supporting heterogeneous probe terminals according to claim 4, wherein, In step five, the useful fields include timestamp, probe domain name, probe node geographical location, accessed DNS resolver, record type, TTL, resolution result, response status code, and RTT field.

8. The large-scale DNS risk posture real-time monitoring system supporting heterogeneous probe terminals according to claim 4, wherein, In step six, the presented statistical indicators include regional DNS risk heat map, Top 10 high-risk domain name list, domain name historical hijacking rate curve, and resolution accuracy and recall rate comparison chart. At the same time, the DNS risk situation report module provides a search bar, the user inputs the website domain name of interest, the backend sends an HTTP GET request to the ElasticSearch component, performs fuzzy matching on the user's query, returns the matched risk situation record data to the module front end for presentation.

Images