A machine learning-based asset detection task state perception and dynamic allocation method

Abstract

The application discloses an asset detection task state perception and dynamic allocation method based on machine learning, belongs to the technical field of network security, and aims at the problems of unable to perceive task state, low resource utilization, and slow abnormal response in the allocation mode. The application firstly constructs an asset detection task dataset and extracts a task and node feature vector. A neural network training task state perception model is adopted by fusing LSTM and an attention mechanism to realize real-time prediction of task completion progress, residual time and abnormal probability. Real-time running state data of distributed detection nodes are collected. Based on the predicted task state and node state, a dynamic allocation algorithm is constructed to minimize the total completion time and node load variance weighted sum as an objective, and an improved genetic algorithm is adopted to solve the optimal allocation scheme. Tasks are allocated according to the scheme and continuous monitoring is carried out, and iterative optimization is realized when the redistribution condition is triggered. Finally, the model precision is continuously optimized through online incremental learning.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, specifically relating to a machine learning-based method for asset detection task state perception and dynamic allocation. Background Technology

[0002] Asset detection is a fundamental component of a network security protection system. By proactively scanning IP addresses, ports, and services in the network, it discovers and identifies various assets in the network, providing data support for subsequent vulnerability scanning, risk assessment, and security protection. With the continuous expansion of network scale and the rapid increase in the number of assets, traditional single-machine asset detection methods can no longer meet the detection needs of large-scale networks, and distributed asset detection systems have emerged to address this need.

[0003] However, most existing distributed asset detection systems employ a static task allocation method, where all detection tasks are evenly distributed among the detection nodes before execution, and once allocated, they are not adjusted. This approach has the following drawbacks:

[0004] 1. Unable to perceive state changes during task execution, and unable to detect and handle abnormal or excessively time-consuming task execution in a timely manner;

[0005] 2. The inability to dynamically adjust task allocation based on the real-time running status of nodes can easily lead to some nodes being overloaded while other nodes remain idle, resulting in low resource utilization.

[0006] 3. Using the same allocation strategy for tasks of different complexities fails to fully leverage the performance advantages of each node;

[0007] 4. When a new high-priority task is added, the low-priority task cannot be interrupted in time, which affects the execution efficiency of important tasks.

[0008] Therefore, a machine learning-based method for asset detection task status awareness and dynamic allocation is needed to solve the problems of existing allocation methods that cannot perceive task status, have low resource utilization, and slow anomaly response. Summary of the Invention

[0009] The purpose of this invention is to provide a machine learning-based method for asset detection task state perception and dynamic allocation to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a method for state perception and dynamic allocation of asset exploration tasks based on machine learning, comprising the following steps:

[0011] Step S1: Construct an asset detection task dataset, preprocess the original task data, and extract task feature vectors and node feature vectors;

[0012] Step S2: Train the task state awareness model, which is a neural network based on the fusion of long short-term memory network (LSTM) and attention mechanism, to predict the completion progress, remaining time and anomaly probability of each detection task in real time.

[0013] Step S3: Collect real-time operating status data of distributed probe nodes, including CPU utilization, memory usage, network bandwidth utilization, and task queue length;

[0014] Step S4: Based on the predicted task status and node running status, calculate the optimal task allocation scheme using a dynamic allocation algorithm;

[0015] Step S5: According to the optimal allocation scheme, the tasks to be executed and the unfinished tasks are reassigned to the corresponding probe nodes;

[0016] Step S6: Continuously monitor the task execution process and node status changes. When the preset redistribution conditions are triggered, return to step S2 for iterative optimization.

[0017] It should be noted in the scheme that in step S1, the task feature vector includes task type, target IP range, number of port scans, protocol type, task priority, and historical execution time; the node feature vector includes node hardware configuration, geographical location, network latency, historical task success rate, and average processing speed.

[0018] It is further worth noting that the training process of the task state awareness model in step S2 includes:

[0019] Step S21: Divide the historical task execution data into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0020] Step S22: Construct an LSTM layer to extract temporal features during task execution, and assign different weights to features at different time steps through an attention mechanism;

[0021] Step S23: Use the fully connected layer to output three predicted values: task completion progress, remaining time, and anomaly probability;

[0022] Step S24: Train the model by using the weighted sum of the mean squared error loss function and the cross-entropy loss function as the total loss function.

[0023] Furthermore, it should be noted that in step S3, the node running status data is collected every 5 seconds. When the change in node status exceeds a preset threshold, a data collection and status update is immediately triggered.

[0024] In a preferred embodiment, the preset redistribution conditions in step S4 include:

[0025] The CPU utilization of any node remained above 85% for more than 30 seconds.

[0026] The predicted remaining time for any task exceeds twice the initial estimated time;

[0027] The probability of an anomaly in any task exceeds 0.7;

[0028] A new high-priority task has been added to the task queue.

[0029] In a preferred implementation, during task reallocation in step S5, high-priority tasks are preferentially assigned to nodes with lower load and higher historical success rates, while ensuring that related tasks of the same target IP segment are assigned to the same node for execution as much as possible.

[0030] As a preferred implementation, step S7 is also included: feeding back the result data and allocation effect data of this task execution to the task status perception model for online incremental learning and continuously optimizing the prediction accuracy of the model.

[0031] In a preferred implementation, the calculation of the anomaly probability includes four anomaly types: task timeout, node failure, network interruption, and target asset unreachable. The model outputs the probability value for each anomaly type.

[0032] As a preferred embodiment, the dynamic allocation algorithm in step S4 is specifically as follows:

[0033] The objective function is defined as minimizing the weighted sum of the total completion time of all tasks and the variance of node load, as shown in the following formula:

[0034]

[0035] in, The objective function value, and These are the weighting coefficients, and , For the total number of tasks, For the first The predicted completion time for each task. To detect the total number of nodes, For the first The overall load value of each node. This represents the average combined load value across all nodes.

[0036] Node overall load value The calculation formula is:

[0037]

[0038] in, For the first CPU utilization of each node, For memory usage, Network bandwidth utilization This is the normalized length of the task queue;

[0039] An improved genetic algorithm was used to solve the above objective function. The population size was initialized to 50, the crossover probability was 0.8, the mutation probability was 0.05, and the number of iterations was 100. The optimal task assignment matrix was finally obtained. ,in Indicates the first The task is assigned to the first 1 node This indicates that no allocation will be made.

[0040] Compared with existing technologies, the present invention provides a machine learning-based method for asset detection task state awareness and dynamic allocation, which has at least the following beneficial effects:

[0041] (1) This invention adopts a neural network model based on the fusion of LSTM and attention mechanism, which can predict the completion progress, remaining time and anomaly probability of asset detection tasks in real time, realize accurate perception of task status, and provide a reliable basis for dynamic task allocation; In addition, this invention comprehensively considers the predicted status of tasks and the real-time running status of nodes, and constructs a dynamic allocation algorithm with the goal of minimizing the total completion time and node load variance, which can realize intelligent scheduling of tasks and improve the overall efficiency and resource utilization of the system.

[0042] (2) The present invention sets up a variety of redistribution trigger conditions, which can respond to changes in system status in a timely manner and redistribute tasks, avoiding system performance degradation caused by excessive node load or abnormal task execution; at the same time, the present invention introduces an online incremental learning mechanism, which can feed back the result data of each task execution to the model, continuously optimize the prediction accuracy of the model, and enable the system to have adaptive capabilities, so as to adapt to different network environments and task scenarios. Attached Figure Description

[0043] Figure 1 This is a flowchart of a machine learning-based asset detection task state perception and dynamic allocation method according to the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to embodiments.

[0045] Reference Figure 1 As shown, this invention provides a machine learning-based method for asset detection task state awareness and dynamic allocation, comprising the following steps:

[0046] Step S1: Construct an asset detection task dataset, preprocess the original task data, and extract task feature vectors and node feature vectors.

[0047] Specifically, firstly, historical asset exploration mission data and node operation data are collected to construct an initial dataset; the raw data includes basic mission information, execution process data, execution result data, as well as node hardware configuration information and operating status information.

[0048] The raw data is preprocessed, including data cleaning, missing value imputation, outlier handling, and feature encoding. For numerical features, normalization is performed to map them to the [0,1] interval. For categorical features, one-hot encoding is used.

[0049] Extract task feature vectors and node feature vectors. Task feature vectors ,in For task types (port scanning, service identification, operating system identification, etc.). For the target IP range, Number of port scans For protocol type (TCP, UDP, ICMP, etc.). Task priority (levels 1-5, with higher values ​​indicating higher priority). This refers to the historical execution duration.

[0050] Node feature vectors ,in Configure the node hardware (number of CPU cores, memory size, etc.). For geographical location, Due to network latency, For the success rate of historical missions, This represents the average processing speed.

[0051] Furthermore, during the data cleaning phase, outliers are removed using the 3σ principle, meaning that for numerical features, samples exceeding the mean ± 3 times the standard deviation are deleted. Missing values ​​are filled using the mean-based filling method for tasks of the same type / nodes with the same configuration. For missing values ​​related to the historical execution time of a task, the average execution time of tasks of the same type and target IP segment is used to fill the missing values.

[0052] In the feature encoding stage, the task type uses 4-bit one-hot encoding (port scanning: 1000, service identification: 0100, operating system identification: 0010, vulnerability detection: 0001); the protocol type uses 3-bit one-hot encoding (TCP: 100, UDP: 010, ICMP: 001); and the geographical location uses latitude and longitude normalized encoding, mapping the global latitude and longitude range to the [0,1] interval.

[0053] The dataset is divided chronologically, with the first 70% used as the training set, the middle 20% as the validation set, and the last 10% as the test set to avoid data leakage.

[0054] Step S2: Train the task state awareness model. Based on a neural network that integrates Long Short-Term Memory (LSTM) network and attention mechanism, predict the completion progress, remaining time and anomaly probability of each detection task in real time. The calculation of the anomaly probability includes four types of anomalies: task timeout, node failure, network interruption and target asset unreachable. The model outputs the probability value of each anomaly type.

[0055] Specifically, the training process of the task state-aware model includes:

[0056] Step S21: Divide the historical task execution data into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0057] Step S22: Construct an LSTM layer to extract temporal features during task execution, and assign different weights to features at different time steps through an attention mechanism;

[0058] Step S23: Use the fully connected layer to output three predicted values: task completion progress, remaining time, and anomaly probability;

[0059] Step S24: Train the model by using the weighted sum of the mean squared error loss function and the cross-entropy loss function as the total loss function.

[0060] Step S3: Collect real-time operating status data of distributed probe nodes, including CPU utilization, memory usage, network bandwidth utilization, and task queue length.

[0061] Specifically, the node running status data is collected every 5 seconds. When the change in node status exceeds the preset threshold, a data collection and status update is immediately triggered.

[0062] Furthermore, a lightweight status acquisition agent is deployed on each probe node to obtain runtime status data using the system's native API: CPU utilization is calculated by reading the / proc / stat file (Linux system) or calling the GetSystemTimes function (Windows system); memory utilization is calculated by reading the / proc / meminfo file or calling the GlobalMemoryStatusEx function; network bandwidth utilization is calculated by reading the / proc / net / dev file or calling the GetIfTable function; and the task queue length is maintained by the node's local task scheduler, directly reading the queue length value.

[0063] The preset status change thresholds are: CPU utilization change exceeding 20%, memory utilization change exceeding 15%, network bandwidth utilization change exceeding 30%, and task queue length change exceeding 5 tasks. When any of these indicators changes by more than the corresponding threshold, the data acquisition agent immediately sends a status update data packet to the central scheduling server.

[0064] Data transmission uses the TLS1.3 encryption protocol. The data packet contains the node ID, timestamp, and various status indicator values. The data packet size does not exceed 1KB to ensure transmission efficiency.

[0065] Step S4: Based on the predicted task status and node running status, calculate the optimal task allocation scheme through a dynamic allocation algorithm.

[0066] Specifically, the preset redistribution conditions include:

[0067] The CPU utilization of any node remained above 85% for more than 30 seconds.

[0068] The predicted remaining time for any task exceeds twice the initial estimated time;

[0069] The probability of an anomaly in any task exceeds 0.7;

[0070] A new high-priority task has been added to the task queue.

[0071] Furthermore, the dynamic allocation algorithm is as follows:

[0072] The objective function is defined as minimizing the weighted sum of the total completion time of all tasks and the variance of node load, as shown in the following formula:

[0073]

[0074] in, The objective function value, and These are the weighting coefficients, and , For the total number of tasks, For the first The predicted completion time for each task. To detect the total number of nodes, For the first The overall load value of each node. This represents the average combined load value across all nodes.

[0075] Node overall load value The calculation formula is:

[0076]

[0077] in, For the first CPU utilization of each node, For memory usage, Network bandwidth utilization This is the normalized length of the task queue;

[0078] An improved genetic algorithm was used to solve the above objective function. The population size was initialized to 50, the crossover probability was 0.8, the mutation probability was 0.05, and the number of iterations was 100. The optimal task assignment matrix was finally obtained. ,in Indicates the first The task is assigned to the first 1 node This indicates that no allocation will be made.

[0079] Step S5: Based on the optimal allocation scheme, reassign the tasks to be executed and the tasks not yet completed to the corresponding probe nodes.

[0080] Specifically, when redistributing tasks, high-priority tasks are prioritized to nodes with lower load and higher historical success rates, while ensuring that related tasks within the same target IP segment are assigned to the same node for execution as much as possible.

[0081] Furthermore, the specific implementation methods for task breakpoint continuation and priority preemption are as follows:

[0082] Resuming execution from breakpoints:

[0083] When an unfinished task needs to be transferred from the original node to a new node, the original node first pauses task execution and saves a snapshot of the task status. The status snapshot includes: a list of scanned IP addresses, a list of scanned ports, the current scan progress (number of scanned IPs / total number of IPs), scan parameters (scan rate, timeout, number of retries), and asset information acquired.

[0084] The state snapshot is saved in JSON format, with a file size not exceeding 10MB, and is transmitted to the new node via a TLS encrypted channel. After receiving the state snapshot, the new node parses and restores the task execution environment, continuing the scan from the breakpoint without needing to rescan the completed parts.

[0085] High-priority task preemption mechanism:

[0086] When a high-priority task (priority ≥ 4) is added, the system immediately pauses all low-priority tasks (priority ≤ 2), saves their state snapshots, and prioritizes the execution of the high-priority task. After the high-priority task completes its execution, the system automatically resumes the paused low-priority tasks, continuing execution from the breakpoint.

[0087] Medium-priority tasks with a priority of 3 do not participate in preemption and are executed in the normal order.

[0088] Step S6: Continuously monitor the task execution process and node status changes. When the preset redistribution conditions are triggered, return to step S2 for iterative optimization.

[0089] Furthermore, when the probability of a task exception exceeds 0.7, the system executes different handling procedures based on the exception type:

[0090] Task timeout exception (probability > 0.7): Split the task into multiple subtasks and redistribute them to multiple nodes with lower load for parallel execution;

[0091] Node failure (probability > 0.7): Immediately transfer all unfinished tasks on the node to other normal nodes, mark the node as a failure node, and stop assigning new tasks to it until the node recovers.

[0092] Network interruption anomaly (probability > 0.7): Pause all tasks on this node and wait for the network to be restored. If the network interruption lasts for more than 5 minutes, the tasks will be transferred to other nodes.

[0093] Target asset unreachable anomaly (probability > 0.7): Terminate the task, record the unreachable information of the target asset, and no reallocation is required.

[0094] In addition, the method of the present invention also includes step S7: feeding back the result data and allocation effect data of this task execution to the task status perception model for online incremental learning and continuously optimizing the prediction accuracy of the model.

[0095] Specifically, the trigger conditions for online incremental learning are: every 100 tasks completed, or when the system is idle at 2:00 AM every day.

[0096] Incremental learning employs a sliding window data sampling method, retaining the execution data of the most recent 10,000 tasks as the incremental training set and discarding earlier data to ensure that the model can adapt to the latest network environment and task features.

[0097] During training, the parameters of the LSTM and attention layers of the model are frozen, and only the parameters of the last two fully connected layers are fine-tuned. The learning rate is set to 0.0001, the batch size is 16, and the number of training rounds is 5.

[0098] Meanwhile, the Elastic Weight Consolidation (EWC) algorithm is introduced to calculate the importance weights of model parameters. During incremental learning, changes in important parameters are penalized to prevent catastrophic forgetting of the model and ensure that the model's prediction accuracy on old tasks does not decrease.

[0099] In summary, this invention deeply integrates machine learning technology with distributed asset exploration task scheduling to construct a closed-loop intelligent scheduling system of "state awareness - dynamic allocation - iterative optimization". Addressing the core technical shortcomings of traditional static allocation methods, such as the inability to perceive task execution status in real time, node load imbalance, delayed handling of abnormal tasks, and low resource utilization, this invention employs a neural network that combines LSTM and attention mechanisms to accurately capture the temporal execution characteristics of asset exploration tasks. This enables real-time and forward-looking prediction of task completion progress, remaining time, and probabilities of multiple types of anomalies, providing a reliable quantitative decision-making basis for dynamic task scheduling. Furthermore, by constructing a system that considers the total exploration completion time... The system employs a multi-objective optimization function for node load balancing and utilizes an improved genetic algorithm to quickly solve for the globally optimal allocation scheme. This fully leverages the performance advantages of distributed probe nodes and enables refined and global configuration of probe resources. By setting multi-dimensional and configurable redistribution trigger conditions, the system ensures rapid response and dynamic adjustment capabilities to various emergencies such as node overload, task execution anomalies, and the insertion of high-priority urgent tasks. Simultaneously, an online incremental learning mechanism is introduced, enabling the system to continuously accumulate operational data under different network environments and task scenarios, iteratively optimizing the prediction accuracy of the state-aware model and the adaptability of the scheduling algorithm, thus exhibiting excellent self-learning and adaptive capabilities.

[0100] This invention can significantly improve the operating efficiency, resource utilization, and anti-interference capability of large-scale distributed asset detection systems. It can be widely applied to asset detection and security management in various complex network environments such as enterprise intranets, government private networks, cloud computing platforms, and industrial internet, providing a solid foundation for building a comprehensive, efficient, and intelligent network security protection system.

[0101] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for state awareness and dynamic allocation of assets in asset exploration tasks based on machine learning, characterized in that, Includes the following steps: Step S1: Construct an asset detection task dataset, preprocess the original task data, and extract task feature vectors and node feature vectors; Step S2: Train the task state awareness model, which is a neural network based on the fusion of long short-term memory network and attention mechanism, to predict the completion progress, remaining time and anomaly probability of each detection task in real time. Step S3: Collect real-time operating status data of distributed probe nodes, including CPU utilization, memory usage, network bandwidth utilization, and task queue length; Step S4: Based on the predicted task status and node running status, calculate the optimal task allocation scheme using a dynamic allocation algorithm; Step S5: According to the optimal allocation scheme, the tasks to be executed and the unfinished tasks are reassigned to the corresponding probe nodes; Step S6: Continuously monitor the task execution process and node status changes. When the preset redistribution conditions are triggered, return to step S2 for iterative optimization.

2. The method for asset detection task state awareness and dynamic allocation based on machine learning according to claim 1, characterized in that: In step S1, the task feature vector includes task type, target IP range, number of port scans, protocol type, task priority, and historical execution time; the node feature vector includes node hardware configuration, geographical location, network latency, historical task success rate, and average processing speed.

3. The method for state perception and dynamic allocation of asset exploration tasks based on machine learning according to claim 1, characterized in that: In step S2, the training process of the task state awareness model includes: Step S21: Divide the historical task execution data into a training set, a validation set, and a test set in a ratio of 7:2:1; Step S22: Construct an LSTM layer to extract temporal features during task execution, and assign different weights to features at different time steps through an attention mechanism; Step S23: Use the fully connected layer to output three predicted values: task completion progress, remaining time, and anomaly probability; Step S24: Train the model by using the weighted sum of the mean squared error loss function and the cross-entropy loss function as the total loss function.

4. The method for state perception and dynamic allocation of asset exploration tasks based on machine learning according to claim 1, characterized in that: In step S3, the node running status data is collected every 5 seconds. When the change in node status exceeds a preset threshold, a data collection and status update is immediately triggered.

5. The method for state perception and dynamic allocation of asset exploration tasks based on machine learning according to claim 1, characterized in that: In step S4, the preset redistribution conditions include: The CPU utilization of any node remained above 85% for more than 30 seconds. The predicted remaining time for any task exceeds twice the initial estimated time; The probability of an anomaly in any task exceeds 0.7; A new high-priority task has been added to the task queue.

6. The method for state perception and dynamic allocation of asset exploration tasks based on machine learning according to claim 1, characterized in that: In step S5, when tasks are redistributed, high-priority tasks are prioritized to nodes with lower load and higher historical success rates, while ensuring that related tasks of the same target IP segment are assigned to the same node for execution as much as possible.

7. The method for state perception and dynamic allocation of asset exploration tasks based on machine learning according to claim 1, characterized in that: It also includes step S7: feeding back the result data and allocation effect data of this task execution to the task status awareness model for online incremental learning and continuous optimization of the model's prediction accuracy.

8. The method for state perception and dynamic allocation of asset exploration tasks based on machine learning according to claim 1, characterized in that: The calculation of the anomaly probability includes four anomaly types: task timeout, node failure, network interruption, and target asset unreachable. The model outputs the probability value for each anomaly type.

9. The method for state perception and dynamic allocation of asset exploration tasks based on machine learning according to claim 1, characterized in that: The dynamic allocation algorithm in step S4 is as follows: The objective function is defined as minimizing the weighted sum of the total completion time of all tasks and the variance of node load, as shown in the following formula: in, The objective function value, and These are the weighting coefficients, and , For the total number of tasks, For the first The predicted completion time for each task. To detect the total number of nodes, For the first The overall load value of each node. This represents the average combined load value across all nodes. Node overall load value The calculation formula is: in, For the first CPU utilization of each node, For memory usage, Network bandwidth utilization This is the normalized length of the task queue; An improved genetic algorithm was used to solve the above objective function. The population size was initialized to 50, the crossover probability was 0.8, the mutation probability was 0.05, and the number of iterations was 100. The optimal task assignment matrix was finally obtained. ,in Indicates the first The task is assigned to the first 1 node This indicates that no allocation will be made.

Images