An Adaptive and Efficient Scheduling Method for Distributed Network Probing Tasks Based on Process Decoupling

Abstract

This invention discloses an adaptive and efficient scheduling method for distributed network probing tasks based on process decoupling, belonging to the field of cyberspace mapping technology. In this method, the control terminal obtains a specified target IP range and port list, then performs task planning, pushing the generated subtasks to the task queue of a message server. Each node continuously listens to the task queue of the message server, actively pulling subtasks from the message server when idle. The probing nodes generate a total list of probing tasks locally and perform probing, while simultaneously updating and recording the current task load and performance load. After the probing results are returned, the probing nodes push multiple return messages back to the result queue of the message server. The control terminal allocates tasks to local message queues according to task type. Locally, data is read and processed from different message queues and stored in an Elasticsearch database to support downstream applications. This invention addresses the problems of low efficiency in single-point probing and uneven distribution and load in distributed probing tasks.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network space mapping, and particularly relates to a distributed network detection task adaptive and efficient scheduling method based on process decoupling. BACKGROUND

[0002] With the rapid development of new generation information technologies such as 5G, Internet of Things, cloud computing, artificial intelligence and the popularization and application of IPv6 addresses, the scale of Internet assets presents an exponential growth trend. As a key solution to the above challenges, the strategic value of network space mapping technology is increasingly prominent, and the active detection technology has shown irreplaceable unique advantages and has become a standard measurement technology for understanding various behaviors of the Internet. Existing works have established many modern scanning tools, which have significantly improved the speed of network scanning. NMap is a general network scanning tool that supports multiple types of scanning tasks. However, NMap maintains the connection state of each probe, resulting in relatively slow scanning speed. In order to accelerate the scanning process, ZMap and Masscan abandon the maintenance of state information of each connection, asynchronously send scanning probes, and with the assistance of optimizers such as PF_RING, Packetshader, netmap, etc., the time-consuming of a single measurement node traversing IPv4 address space single-port is reduced from weeks to minutes.

[0003] However, the above network detection tools all use single-point centralized scanning, which has the following two disadvantages: first, the field of view of a single node is obviously limited, and the coverage of network space assets is insufficient; second, the detection efficiency is obviously insufficient, and it is difficult to meet the needs of large-scale target (such as full address space full port) fast scanning tasks, and the speed is set too fast, and continuous detection of the same network segment target in a short time is easy to be detected and lead to ban, which sacrifices the safety, sustainability and actual detection effect of detection.

[0004] In order to further improve the detection efficiency, distributed network detection technology is continuously developed. The existing distributed detection system usually adopts master-slave node deployment and task allocation and scheduling strategy, the master node and the detection node directly communicate and are closely coupled, and the master node polls and issues tasks one by one according to the node state, which has the following disadvantages: first, the master node is required to be high, the master node is easy to become the efficiency bottleneck, the communication overhead is large, and the scheduling delay is high; second, there is lack of adaptive load balancing mechanism, which may lead to uneven task allocation and low node utilization; third, the master node and the detection node directly interact and are closely coupled, which is not conducive to the expansion and reduction of the detection node, and is not conducive to flexible task management (such as not conducive to adding emergency tasks on the spot), and is difficult to adapt to the mapping scene in large-scale, multi-user and multi-type task scenarios; fourth, there is lack of evaluation of the field of view of the detection node, and the task allocation is blind and random. SUMMARY

[0005] The application proposes a process decoupling-based distributed network detection task adaptive and efficient scheduling method, aiming to solve the technical problems of insufficient single-point detection efficiency and uneven distribution of distributed detection tasks in allocation and load.

[0006] The process decoupling-based distributed network detection task adaptive and efficient scheduling method proposed by the application comprises:

[0007] Step S1, the control end receives a user detection task and a detection mode, obtains a specified target IP address range and a port list therefrom, carries out task planning, divides the target IP address into blocks and slices, nests traverses each slice and port, and generates a plurality of subtasks, and pushes the generated all subtasks to a task queue of a message server;

[0008] Step S2, each detection node in the distributed detection node cluster continuously listens to the task queue of the message server, and monitors the state of the task load and the performance load of itself in real time, and actively pulls the subtask from the message server when the load is idle;

[0009] Step S3, after the subtask arrives at the detection node, the detection node generates a total detection task list based on the address set, the white list, the black list and the load weight contained in the subtask, and carries out detection according to the task slice number, and updates the state of the current task load and the performance load;

[0010] Step S4, the detection node pushes a plurality of detection result return messages to the result queue of the message server; the control end pulls the detection result return message from the result queue of the message server, and distributes the detection result to different local result queues according to the task type;

[0011] Step S5, the detection result of different task types is read from different local result queues, and after data processing, it is stored in an ES database for use by downstream applications.

[0012] According to the above method proposed by the application, in step S1, the control end receives a user detection task and a detection mode, obtains a specified target IP address range and a port list therefrom, the target IP address range is a specified IP segment or a specified region; wherein:

[0013] If the detection mode is accurate detection, the blocking process is performed first, the target IP address is blocked by querying the GeoIP database, and a regional label Geo-Tag is added to each block, and then each block is sliced; wherein, the accurate detection refers to a region-first detection mode;

[0014] If the detection mode is random detection, the slicing process is directly performed;

[0015] In the execution of the fragmentation, the total number of IP addresses to be probed Total_IPs is calculated according to the target IP address range, and the number of slices Num_AddrSlices is determined by using an adaptive strategy; wherein:

[0016] Num_AddrSlices = ceil(sqrt(Total_IPs)) or

[0017] Num_AddrSlices = k ceil(sqrt(Total_IPs))

[0018] wherein ceil() represents rounding up, sqrt() represents square root, and k is an adjustment coefficient.

[0019] According to the above method of the present application, in step S1, each slice and port is nested and traversed, and a plurality of subtasks are generated; wherein:

[0020] Num_SubTasks = Num_AddrSlices x Num_Ports

[0021] Num_SubTasks represents the number of subtasks, and Num_Ports represents the number of ports; one subtask at least includes one slice and a corresponding port;

[0022] The generated all subtasks are pushed to the task queue of the message server, and each probe node in the distributed probe node cluster asynchronously executes continuous monitoring on the task queue.

[0023] According to the above method of the present application, in step S2:

[0024] If the probe mode is accurate probe, only part of the probe nodes are allowed to pull the subtask with a geographical label Geo-Tag from the message server, and the part of the probe nodes refer to the n probe nodes whose view evaluation result of the region corresponding to the Geo-Tag is in the front;

[0025] If the probe mode is random probe, any idle probe node is allowed to pull the subtask without a geographical label Geo-Tag.

[0026] According to the above method of the present application, in step S2: the threshold values of the task load and performance load of the probe node are preset, so as to control the resource utilization rate of the probe node and the execution of different priority tasks; each subtask has a load weight, and the prerequisite for the probe node to pull the subtask is that the current task load and performance load are less than the preset threshold value.

[0027] According to the method, in step S3, the detection node implements detection specifically including:

[0028] According to the white list and the black list, the target IP addresses in the address set are filtered to generate a target IP address list TargetList that needs to be detected.

[0029] Based on the index number of TargetList, the number of fragments is taken modulo, and if the value of the modulo is equal to the task fragment number pulled by the current detection node, the address corresponding to the index number is detected.

[0030] According to the method, in step S4, the detection node first stores the detection result in a buffer, and when a predetermined number of detection results exist in the buffer, the detection results are packaged and compressed to generate a detection result return message.

[0031] According to the method, in step S4, the control end continuously pulls the detection result return message from the result queue of the message server and performs decompression processing to parse the original detection result; the control end distributes the detection result to different local result queues according to the task type, and the local result queue at least includes a TCP result queue, a UDP result queue, a service identification queue, and a vulnerability scanning queue.

[0032] To sum up, the application reconstructs the communication interaction process of the master node and the detection node, optimizes the detection task allocation and scheduling strategy, improves the load balancing mechanism, and enhances the evasion capability of detection, to meet the demand of large-scale, multi-user, multi-task, high-frequency, and continuous active network detection scenarios, and aims to improve the detection efficiency, resource utilization, and network asset coverage effect, and effectively evade the blocking mechanism of the target network, and maintain the flexibility, high expansibility, and sustainability of the surveying and mapping system. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0034] Figure 1 The flowchart of the distributed network detection task adaptive and efficient scheduling method based on process decoupling according to the embodiment of the application. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention proposes an adaptive and efficient scheduling method for distributed network probing tasks based on process decoupling. Its core objectives mainly include: (1) optimizing task allocation to improve efficiency and resource utilization. Based on the task scale and node perspective, the dynamic and adaptive division and allocation of tasks are realized, making full use of distributed node resources and shortening the overall probing time; (2) improving task scheduling strategy to enhance system scalability and robustness. By adding a third-party intermediary server (message queue), the direct interaction between the master node and the probe node is decoupled, and the probe tasks and results are executed asynchronously. A flexible task distribution and result collection mechanism is provided to reduce system bottlenecks and enhance the adaptability to dynamic addition and removal of nodes and changes in task scale; (3) optimizing the load balancing mechanism. By customizing the full load threshold of the probe node, the node resource occupancy rate can be flexibly adjusted to realize the differentiated management of urgent and routine tasks, meeting the needs of large-scale, multi-user, multi-task, and continuous mapping scenarios; (4) enhancing the probe avoidance capability. By allocating continuous addresses to different nodes, the time interval between the same node hitting the same target network is lengthened, reducing the risk of the probe behavior being identified and blocked by the target network, and improving the success rate and sustainability of the probe.

[0037] like Figure 1 As shown, the adaptive and efficient scheduling method for distributed network probing tasks based on process decoupling proposed in this invention includes:

[0038] Step S1: The control terminal receives the user's probe task and probe mode, and obtains the specified target IP address range and port list from it; it performs task planning, divides the target IP address into blocks and fragments, traverses each fragment and port in a nested manner, generates multiple sub-tasks, and pushes all generated sub-tasks to the task queue of the message server.

[0039] Step S2: Each probe node in the distributed probe node cluster continuously listens to the task queue of the message server and monitors its own task load and performance load status in real time. When there is idle load, it actively pulls sub-tasks from the message server.

[0040] Step S3: After the subtask arrives at the probe node, the probe node generates a total probe task list locally based on the address set, whitelist, blacklist and load weight contained in the subtask, and performs probe according to the task shard number, while updating the current task load and performance load status.

[0041] Step S4: The probe node pushes multiple probe result return messages to the result queue of the message server; the control terminal pulls the probe result return messages from the result queue of the message server and distributes the probe results to different local result queues according to the task type.

[0042] Step S5: Read the probe results of different task types from different local result queues, process the data and store it in the ES database for downstream applications to use the relevant data.

[0043] In some embodiments, the control terminal first receives the user's probe task and probe mode, obtains the specified target IP range and port list, and then performs task planning, dividing and fragmenting the target address, nesting and traversing the number of ports and fragments, and pushing all generated subtasks to the task queue of the message server. Each node continuously listens to the task queue of the message server and monitors its own task load and performance load status in real time. If there is idle time, it actively pulls one or more subtasks from the message server. After the subtasks arrive at the probe node, the node generates a total probe task list locally based on the address set, whitelist, blacklist, load weight and other parameters contained in the issued task, and then performs probes according to the task fragment number it has received, while updating the current task load and performance load. After the probe results are returned, the probe node concatenates multiple return packets into a relatively appropriate size, compresses them and pushes them back to the result queue of the message server. The control terminal pulls the probe results from the message server and allocates them to the local result queue according to the task type. The local terminal reads and processes data from different result queues and stores it in the ES database to support downstream applications.

[0044] According to the method proposed by the present invention, in step S1, the control terminal receives the user's probe task and probe mode, and obtains the specified target IP address range and port list from them, wherein the target IP address range is a specified IP segment or a specified region; wherein:

[0045] If the detection mode is precise detection, the segmentation process is performed first. The target IP address is segmented by querying the GeoIP database, and a geo-tag is added to each segment. Then, each segment is fragmented. Precise detection refers to the detection mode that prioritizes the field of view.

[0046] If the detection mode is random detection, the fragmentation process is executed directly;

[0047] During fragmentation, the total number of IP addresses to be probed (Total_IPs) is calculated based on the target IP address range, and an adaptive strategy is used to determine the number of fragments (Num_AddrSlices); where:

[0048] Num_AddrSlices = ceil(sqrt(Total_IPs)) or

[0049] Num_AddrSlices = k ceil(sqrt(Total_IPs))

[0050] Here, ceil() represents rounding up, sqrt() represents calculating the square root, and k is the adjustment coefficient.

[0051] According to the method proposed by the present invention, in step S1, nested traversal is performed on each slice and port, and multiple subtasks are generated; wherein:

[0052] Num_SubTasks = Num_AddrSlices×Num_Ports

[0053] Num_SubTasks represents the number of subtasks, and Num_Ports represents the number of ports; a subtask includes at least one shard and a corresponding port.

[0054] All generated subtasks are pushed to the task queue of the message server, and the various probe nodes in the distributed probe node cluster asynchronously perform continuous listening to the task queue.

[0055] In some embodiments, the control unit is responsible for execution: First, it receives the user's probe task and probe mode, obtains the specified target IP range (which can be an IP segment or a directly specified region) and port list. If the probe mode is precise mode, it first enters the segmentation module, queries the GeoIP database, segments the IP addresses into blocks, and then fragments each block of addresses. If the probe mode is random mode, it directly enters the fragmentation module. The fragmentation module calculates the total number of IP addresses to be probed (Total_IPs) based on the target range, and uses an adaptive strategy to determine the number of address space fragments (Num_AddrSlices). By default, the calculation of Num_AddrSlices is based on the square root function of Total_IPs, i.e., Num_AddrSlices = ceil(sqrt(Total_IPs)) or k `ceil(sqrt(Total_IPs))`, where `k` is an adjustable coefficient that can be flexibly adjusted according to the number of probe nodes or specific needs, and `ceil()` is the floor function. This strategy can automatically adjust the sharding granularity according to the task size and the number of nodes, avoiding shards that are too large or too small.

[0056] The system iterates through Num_AddrSlices address slices and a given list of ports in a nested manner (each address slice + each port = one basic subtask), ultimately generating Num_SubTasks = Num_AddrSlices. Each subtask is divided into Num_Ports subtasks and pushed to the task queue of the message server for asynchronous execution by the probe nodes. The structure of each subtask is shown in Table 1.

[0057] Table 1: Data Structure Design for the Probe Subtask

[0058]

[0059] According to the method proposed by the present invention, in step S2:

[0060] If the detection mode is precise detection, for subtasks with geo-tags, only some detection nodes are allowed to pull the subtask from the message server. The "some detection nodes" refer to the n detection nodes that perform field of view evaluation on the geo-tag and whose evaluation results are the highest.

[0061] If the detection mode is random detection, for subtasks without a geo-tag, any idle detection node is allowed to pull the subtask.

[0062] According to the method proposed by the present invention, in step S2: a threshold for the task load and performance load of the probe node is preset, so as to regulate the resource utilization of the probe node and the execution of tasks with different priorities; each subtask has a load weight, and the premise for the probe node to pull a subtask is that the current task load and performance load are both less than the preset threshold.

[0063] In some embodiments, each probe node continuously listens to the task queue of the message server and monitors its own task load and performance load status in real time. If it has idle time, it actively pulls one or more subtasks from the message server to ensure load balancing among the probe nodes. This variable master control terminal centrally controls each probe node to actively request tasks based on its own load, solving the performance bottleneck of the master control node and making full use of the resources of each node.

[0064] In some embodiments, if a subtask is pushed in precise mode and includes a geo-tag, only nodes with good view tracing performance for that region can claim it. View tracing performance is measured based on historical detection data or evaluations of each node's view tracing. The system maintains a viewshed matrix for each detection node across different geographic addresses, using this matrix to determine which nodes can execute a subtask containing a specific geo-tag, thereby improving asset discovery. If the detection mode is random and the subtask's geo-tag is empty, any idle node can claim the corresponding subtask. In this case, the subtask claimed by each node is completely random, suitable for scenarios with low asset coverage requirements but high overall execution efficiency requirements.

[0065] In some embodiments, the task load and performance load thresholds of the probe node can be flexibly set to regulate the resource utilization of the probe node and the execution of tasks with different priorities. Each type of subtask has a load weight, and a node can only accept a new subtask if both the current task load and performance load are less than a preset threshold. For example, if each node program has a task load weight limit of 100, and assuming that the load weight of a certain type of task is 25, then a node can accept a maximum of 4 such subtasks. If the performance load limit is, for example, 60% CPU utilization (ensuring that there are redundant resources available for temporary emergency calls), a node can only accept a new task if and only if the CPU utilization is ≤60%. For high-priority emergency tasks, the task load and resource load thresholds of the node can be increased as appropriate, such as set to 200 and 90%, so that high-priority emergency tasks can be added at any time until the task load reaches 200 or the CPU utilization reaches 90%. At this point, ordinary tasks will no longer be accepted until the two types of loads decrease and the probe node has idle resources, at which point it will resume accepting ordinary tasks.

[0066] According to the method proposed by the present invention, in step S3, the detection node performs detection specifically including:

[0067] The target IP addresses within the address set are filtered based on the whitelist and blacklist to generate a target IP address list TargetList that needs to be detected;

[0068] Based on the modulo operation of the TargetList index number on the number of shards, if the modulo value equals the task shard number pulled by the current probe node, then the address corresponding to that index number is probed.

[0069] In some embodiments, after a probe node receives a subtask, it parses the subtask content to obtain a target address table, shard number, port, whitelist, blacklist, task load weight, etc. It then filters the IPs within the address list based on the whitelist and blacklist, generating a final target address list TargetList to be probed. The node then takes the modulo of the total number of shards with the address list index number. If the modulo value equals the shard number received by the node, the address corresponding to that index number is probed.

[0070] This design achieves two key benefits. First, it breaks the continuity of IP addresses by distributing previously consecutive IP addresses to different probe nodes. This significantly increases the probability that consecutively hit target IPs will not be in the same network segment or adjacent IPs, thus substantially lengthening the interval between hits on the same network segment and effectively circumventing blocking strategies based on consecutive IP access patterns. Second, it eliminates the need to pre-calculate each subtask. Each probe node calculates the total address list locally, employing a compute-as-you-go approach. This avoids loading massive amounts of addresses into memory, greatly reducing memory and computational resource consumption.

[0071] According to the method proposed by the present invention, in step S4, the probe node first temporarily stores the probe results in a buffer. When there are a predetermined number of probe results in the buffer, the multiple probe results are packaged and compressed to generate a probe result return message.

[0072] According to the method proposed by the present invention, in step S4, the control terminal continuously pulls the probe result return messages from the result queue of the message server, decompresses them, and parses out the original probe results; the control terminal allocates the probe results to different local result queues according to the task type, and the local result queues include at least TCP result queue, UDP result queue, service identification queue, and vulnerability scanning queue.

[0073] In some embodiments, the probe node first temporarily stores the probe results in a local buffer, and then packages and compresses them. It's important to note that the probe node does not immediately send individual probe results. Instead, it waits until the buffer approaches or reaches a suitable size, such as 1MB, or reaches a set time threshold (to prevent small amounts of results from remaining in the buffer for extended periods) before packaging and compressing the batch of results and pushing it to the message server's result queue. This significantly reduces the number of transmissions and the amount of data, lowering bandwidth consumption and reducing the pressure on the message server.

[0074] In some embodiments, the control terminal continuously pulls compressed result data packets from the result queue of the message server, decompresses the pulled data packets, and parses out the original probe result records. Based on the task type ID carried in the result data, the control terminal distributes the parsed results to different local message queues (such as TCP result queues, UDP result queues, and specific business result queues). This design decouples result collection from subsequent processing, allowing downstream processes to handle different types of results in parallel as needed.

[0075] Regarding step S5, in some embodiments, the dedicated storage and processing modules (which may be multiple processes / threads) corresponding to various detection tasks read the result data from their respective local message queues, perform necessary cleaning and formatting, and then store it in batches into the Elasticsearch (ES) database, providing a query and analysis basis for downstream applications such as network asset discovery, port and service monitoring and identification, and vulnerability scanning.

[0076] In the above method, the master node and probe nodes interact directly without decoupling. By adding a third-party intermediary as a message server, which maintains separate queues for probe subtasks and results, the direct communication between the master node and probe nodes is decoupled. This transforms the master node's polling for task distribution into probe nodes actively requesting tasks, asynchronously executing probe tasks and sending back results. Other users can add new tasks at any time and increase or decrease the number of probe cluster nodes. This design provides a flexible task distribution and result collection mechanism, reduces system bottlenecks, enhances adaptability to dynamic node additions and removals, task additions and removals, and scale changes, and improves the flexibility of the scheduling engine.

[0077] The above method employs a two-tiered load balancing mechanism: task-level and performance-level. Two load metric parameters are designed: task load and performance load. Task load represents the total number of tasks currently executing. Each type of task issued by the master control terminal has a load weight, and the sum of the load weights of all currently executing tasks is the task load of that node. Performance load refers to the node's host resource utilization, such as CPU utilization, memory usage, and network bandwidth usage. Each probe node continuously monitors its own load status and only actively pulls one or more sub-tasks from the message server's task queue when its own load falls below a preset threshold. The resource utilization efficiency of the probe nodes can be adjusted by flexibly setting the node load threshold, while maintaining a certain resource margin to allow for the addition of high-priority tasks at any time. This proactive pull mechanism based on two-level load awareness ensures adaptive task allocation and load balancing adjustment. It can guarantee the resource utilization of probe nodes and improve probe efficiency, while also ensuring that nodes do not work overloaded, avoiding packet loss and system crashes. It can also ensure that high-priority emergency tasks have redundant resources available at any time, realizing hierarchical management of emergency and routine tasks, and meeting the needs of large-scale, multi-user, multi-task, and continuous mapping scenarios.

[0078] In the above method, two adaptive task allocation modes are configured: random and precise. Due to the influence of geographical and network restriction policies, the "visibility" (field of vision) of different probe nodes to IP addresses in different target regions varies significantly. Therefore, adaptive task allocation based on the node's field of vision capability is particularly important in scenarios with high asset coverage requirements. This invention designs two adaptive task allocation modes for probe implementation: an efficiency-first random mode and a field-of-vision-first precise mode, which users can choose according to their different needs. To ensure load balancing and resource utilization efficiency, the scheduling engine defaults to efficiency-first subtask sharding (see the "Technical Solution" section for specific sharding methods). Each probe node can receive a subtask shard to conduct probes based on its own load. In this case, the target task address received by each node is random. If the user has high requirements for the coverage of the task address set, they can specify a vision-first mode. Tasks are allocated based on the vision capabilities of the nodes. The scheduling engine will first divide the task address set into blocks by region and mark each block with a region tag (Geo-Tag). Then, it will perform the same subtask sharding. In this case, only nodes with good vision of the target region can receive the corresponding subtask.

[0079] The following example illustrates the adaptive and efficient scheduling method for distributed network probing tasks based on process decoupling disclosed in this invention. Taking a service scan with an address range of 1.0.0.0-10.0.0.0 and a TCP_80 port as an example, the blacklist is set to 1.0.0.0-2.0.0.0, the scanning mode is a vision-first precision mode, the priority is normal, and the default load of the service scan task is 25. The implementation steps are as follows.

[0080] (1) The control terminal accepts user probe commands, such as “scan --IP 1.0.0.0-10.0.0.0 --port 80 --ExcludeIP 1.0.0.0-2.0.0.0 --model viewshed --privilege 0”, where the port parameter uses TCP protocol scanning by default. If you want to scan UDP protocol, you need to specify “U:80”; the ExcludeIP field is the blacklist, which can also be imported from a file through the ExcludeFile parameter. If you want to specify a whitelist, use IncludeFile; model represents the scanning mode, viewshed represents the precise mode with view priority, and random mode can also be specified as random; the privilege field specifies the priority, which is 0 by default, and 1 can be specified for emergency tasks.

[0081] (2) Address segmentation. Query GeoIP databases such as MaxMind and IP2Location to segment IP addresses:

[0082] AU: 1.0.0.0 / 24, 1.0.4.0 / 22, 1.1.1.0 / 24,…;

[0083] CN: 1.0.1.0 / 24, 1.0.2.0 / 23, 1.1.0.0 / 24,…;

[0084] JP: 1.0.16.0 / 20, 1.0.64.0 / 18, 1.1.64.0 / 18,….

[0085] (3) Address Fragmentation. Calculate the total number of IP addresses (Total_IPs) for each address block in the AU, CN, and JP regions, and determine the number of fragments (Num_AddrSlices) by default using the square root method. If the probe mode is random mode, directly enter the fragmentation module, calculate the total number of IPs for the 1.0.0.0-10.0.0.0 address range (150,994,994), take the square root and round up to 12289, then the task fragmentation number is 12289.

[0086] (4) Push the subtasks corresponding to each region to the task queue of the message server, and set the region tag (Geo-Tag) to AU, CN, JP, etc. Taking the AU address block as an example, the main structure of its subtasks is as follows:

[0087] The IP range for the reconnaissance mission is: 1.0.0.0 / 24, 1.0.4.0 / 22, 1.1.1.0 / 24, ...;

[0088] Port number: TCP_80;

[0089] Segment number: 1;

[0090] Task priority: 0;

[0091] Blacklist: 1.0.0.0-2.0.0.0;

[0092] Task type: service;

[0093] Load weight: 25;

[0094] Detection mode: Viewshed;

[0095] Geo-Tag: AU.

[0096] (5) Each probe node listens to the task queue and checks whether its own load is less than the preset threshold, i.e., task load ≤ 100, CPU usage ≤ 60%, and pulls sub-tasks from the task queue for areas with good visibility. For example, a probe node in the AU region may receive a sub-task with Geo-Tag AU and shard number 1.

[0097] (6) Local task list generation and probe sending. The AU node parses the subtask content it has received, excludes the blacklist, and generates the final target address list TargetList locally. The address list index number is modulo the total number of shards. If the modulo value is equal to the shard number received by the node (i.e., 1), then the address corresponding to that index number is probed.

[0098] (7) The AU node caches the received probe results and waits for the buffered data to reach a threshold such as 1MB, or when it reaches a time threshold, to package and compress the batch of results data and push it to the result queue of the message server.

[0099] (8) Results collection and downstream application. The control end continuously pulls compressed result data packets from the result queue of the message server, parses them, and distributes them to the local service to scan the corresponding message queue. The storage processing module reads the result data from the queue, performs necessary cleaning and formatting, and stores it in batches into the Elasticsearch (ES) database to provide a basis for querying and analysis for downstream applications.

[0100] In summary, this invention addresses the needs of large-scale, multi-user, multi-task, high-frequency, and continuous proactive network probing scenarios by reconstructing the communication interaction process between the master control node and the probe node, optimizing the probe task allocation and scheduling strategy, improving the load balancing mechanism, and enhancing the probe's circumvention capabilities. The aim is to improve probe efficiency, resource utilization, and network asset coverage, while effectively circumventing the target network's blocking mechanisms and maintaining the flexibility, high scalability, and sustainability of the mapping system.

[0101] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for adaptive and efficient scheduling of distributed network probing tasks based on process decoupling, characterized in that, The method comprises: Step S1, the control terminal receives a user detection task and a detection mode, from which a specified target IP address range and a port list are obtained; task planning is carried out, the target IP address is divided into blocks and slices, each slice and port is nested and traversed, and a plurality of subtasks are generated; and all the generated subtasks are pushed to a task queue of a message server; In step S1, the control terminal receives a user detection task and a detection mode, from which a specified target IP address range and a port list are obtained; the target IP address range is a specified IP segment or a specified region; wherein: If the detection mode is accurate detection, a blocking process is performed first, the target IP address is blocked by querying a GeoIP database, and a regional label Geo-Tag is added to each block; then each block is sliced; wherein the accurate detection refers to a view priority detection mode; If the detection mode is random detection, a slicing process is directly performed; When the slicing process is performed, the total number of IP addresses to be detected Total_IPs is calculated according to the target IP address range, and an adaptive strategy is used to determine the number of slices Num_AddrSlices; wherein: Num_AddrSlices = ceil(sqrt(Total_IPs)) or Num_AddrSlices = k ceil(sqrt(Total_IPs)) Wherein, ceil() represents rounding up, sqrt() represents square root, and k is an adjustment coefficient; Step S2, each detection node in the distributed detection node cluster continuously listens to the task queue of the message server, and monitors the state of its own task load and performance load in real time; when the load is idle, it actively pulls a subtask from the message server; In step S2: If the detection mode is accurate detection, only part of the detection nodes are allowed to pull the subtask with the regional label Geo-Tag from the message server, the part of the detection nodes refers to the n detection nodes whose view evaluation result of the region corresponding to the Geo-Tag is in the front; If the detection mode is random detection, any idle detection node is allowed to pull the subtask without the regional label Geo-Tag; Step S3, after the subtask reaches the detection node, the detection node generates a total detection task list based on the address set, the white list, the black list and the load weight contained in the subtask, and implements detection according to the task slice number, and updates the state of the current task load and performance load; In step S3, the detection node implements detection, which specifically includes: Filtering the target IP addresses in the address set according to the white list and the black list to generate a target IP address list TargetList that needs to be detected; Modulo the number of slices based on the index number of TargetList, if the value of the modulo is equal to the task slice number pulled by the current detection node, the address corresponding to the index number is detected. Step S4, the detection node pushes the multiple detection result return messages to the result queue of the message server; the control end pulls the detection result return messages from the result queue of the message server, and distributes the detection results to different local result queues according to the task types; Step S5, the detection results of different task types are read from the different local result queues, and after data processing, the detection results are stored in the ES database for use by downstream applications.

2. The method of claim 1, wherein, In step S1, each slice and port is nested and traversed, and multiple subtasks are generated; wherein: Num_SubTasks = Num_AddrSlices × Num_Ports Num_SubTasks represents the number of subtasks, and Num_Ports represents the number of ports; one subtask at least includes one slice and a corresponding port; The generated all subtasks are pushed to the task queue of the message server, and each detection node in the distributed detection node cluster asynchronously executes continuous monitoring on the task queue.

3. The method of claim 2, wherein, In step S2, the threshold values of the task load and performance load of the preset detection node are set to control the resource utilization of the detection node and the execution of different priority tasks; each subtask has a load weight, and the premise of the detection node pulling the subtask is that the current task load and performance load are less than the preset threshold values.

4. The method of claim 3, wherein, In step S4, the detection node temporarily stores the detection results in the buffer area, and when there are a predetermined number of multiple detection results in the buffer area, the multiple detection results are packaged and compressed to generate the detection result return message.

5. The method of claim 4, wherein, In step S4, the control end continuously pulls the detection result return messages from the result queue of the message server, and performs decompression processing on the detection result return messages to parse the original detection results; the control end distributes the detection results to different local result queues according to the task types, and the local result queues at least include a TCP result queue, a UDP result queue, a service identification queue, and a vulnerability scanning queue.

Images