Intelligent scheduling method and system for distributed network probing task based on deep reinforcement learning

Abstract

The application discloses a kind of distributed network exploration task intelligent scheduling method and system based on deep reinforcement learning, it is related to network space exploration, software engineering and deep reinforcement learning field.User creates new network exploration task, and intelligent scheduling system collects the scheduling information of this task, and converts scheduling information into input vector by the input conversion module of intelligent scheduling center;Input vector is transmitted to deep reinforcement learning module for intelligent decision, and output vector is obtained, and further reinforcement learning action is obtained by output conversion module, and sent to decision execution module, to complete the execution of network exploration task.Finally, the task execution result is fed back to intelligent scheduling center, and the deep reinforcement learning model is updated, and the scheduling decision is optimized.The application improves the efficiency and response speed of network exploration, meets the high requirements of real-time and accuracy in network security field.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of network space exploration, software engineering and deep reinforcement learning, in particular to a distributed network exploration task intelligent scheduling method and system based on deep reinforcement learning. BACKGROUND

[0002] Deep Reinforcement Learning (DRL) is an advanced technology that combines deep learning and reinforcement learning. It uses the ability of deep learning to process high-dimensional data and combines the decision-making mechanism of reinforcement learning to optimize the action strategy under long-term goals. DRL is particularly effective in dealing with complex and dynamic environments, and can optimize the decision-making process through continuous self-learning and adaptation. This makes it have great application potential and effect in various complex application scenarios such as games, robot navigation, autonomous vehicles and resource scheduling.

[0003] However, in traditional network exploration systems, task scheduling is usually static and lacks flexibility, which is inefficient in the face of dynamic network changes and large-scale network exploration tasks. SUMMARY

[0004] The present application designs a distributed network exploration task intelligent scheduling method based on deep reinforcement learning, aiming to optimize the scheduling of network exploration tasks in a distributed computing environment and minimize task execution time. At the same time, the present application also designs an efficient distributed network exploration task intelligent scheduling system.

[0005] The distributed network exploration task intelligent scheduling method based on deep reinforcement learning comprises the following steps:

[0006] Step one, the user creates a new network exploration task through the Web console, the information collection module collects scheduling information, and transmits these scheduling information to the intelligent scheduling center;

[0007] The scheduling information includes current task information to be scheduled, task queue information and current server cluster information, specifically:

[0008] (1) The current task information to be scheduled includes: task CPU occupancy rate, task type, task affinity list, task anti-affinity list, total traffic required for task execution, task execution estimation time and task urgency.

[0009] (2) The task queue information includes task queue length and task queue urgency list.

[0010] (3) The current server cluster information is marked by a binary tuple, busy flag S and task remaining time LT:

[0011] Busy flag S: a binary identifier indicating the working status of the machine. Its value is 0 or 1, where 0 represents that the machine is currently idle, and 1 indicates that the machine is busy processing tasks.

[0012] Task remaining time LT: represents the estimated remaining time for completing the running task on the current machine.

[0013] Step two, the intelligent scheduling center preprocesses the scheduling information, transmits the obtained input vector to the deep reinforcement learning module for intelligent decision-making, and obtains the output vector as the task scheduling decision of the current state.

[0014] Preprocessing refers to converting the scheduling information into an input vector through an input conversion module.

[0015] The input vector includes n-dimensional state information of the server cluster, m-dimensional current task information to be scheduled, k-dimensional task queue information, and t-dimensional scalable dimension.

[0016] The input vector generates an output vector through the strategy learned by the deep reinforcement learning module;

[0017] The first k elements of the output vector are Task_Split_P i , i = 0, 1, 2, 3, … k, representing the probability of splitting the task into 2 i parts, taking the maximum value Task_Split_P m , which represents splitting the task into corresponding 2 m parts. The following n-dimensional vector represents the probability P j of scheduling the split 2 m parts of the task to the jth machine. Finally, according to the probability P j from large to small, take the first 2 m machines with larger probability values, and schedule the split tasks to the corresponding machines.

[0018] Step three, the output vector is converted into a reinforcement learning action through an output conversion module, and the intelligent scheduling center sends the reinforcement learning action to the decision execution module to guide the execution of network probing tasks.

[0019] The reinforcement learning action is divided into task splitting and task scheduling.

[0020] Step four, after the task is executed, each server uploads the task execution information in log format to the log center, uploads the probing results to the probing result center, and at the same time, the monitoring points deployed on the servers periodically obtain new cluster state information and upload it to the monitoring center and the intelligent scheduling center.

[0021] Step five, the intelligent scheduling center updates the deep reinforcement learning model according to the new cluster state information, and returns to step two to optimize the task scheduling decision through the reward function.

[0022] The reward function is:

[0023] Reward = a·Affinity_Bonus + b·Anti_Affinity_Penalty + g·Immediate_Reward

[0024] + d·Time_Cost + h·Flow_Cost

[0025] The design of the reward function includes:

[0026] The first part: Affinity Bonus and Anti-Affinity Penalty, which distinguishes whether the current task is scheduled to a node with affinity or anti-affinity. If the task is scheduled to a node with high affinity, a positive reward (Affinity_Bonus) is given; on the contrary, if the task is assigned to a node with anti-affinity, a high penalty term (Anti_Affinity_Penalty) will be generated.

[0027] The second part: Immediate Reward Immediate_Reward, after each task scheduling decision, the system will immediately assign rewards, alleviating the reward delay phenomenon.

[0028] The third part: Time Cost Time_Cost and Flow Cost Flow_Cost reward, used to guide the intelligent scheduling center to select the scheduling task.

[0029] The fourth part: a, b, g, d, h are hyperparameters, optimized through the parameter adjustment process, guiding the tendency of the intelligent scheduling center. Increasing the value of a will guide the intelligent scheduling center to prefer to schedule tasks to nodes with affinity. Increasing the value of b will cause the intelligent scheduling center to avoid anti-affinity nodes in task allocation. Adjusting the value of g can affect the importance of immediate reward, accelerating the learning process and providing timely feedback. The adjustment of d and h reflects the agent's preference for time and flow cost, guiding the intelligent scheduling center to reduce time and resource consumption.

[0030] Step six, the information obtained by the monitoring center, the detection result center and the log center is transmitted to the Web console, and the cluster information, task log and detection result are displayed through the display module, and these information is further used for scientific analysis.

[0031] The distributed network exploration task intelligent scheduling system comprises a Web console, a center node and an extensible server cluster; wherein the Web console comprises an information collection module and a display module, the center node comprises an intelligent scheduling center, an exploration result center, a log center and a monitoring center, and the extensible server cluster comprises a decision execution module and a monitoring point.

[0032] The advantages and positive effects of the present application are as follows:

[0033] (1) The distributed network exploration task intelligent scheduling system based on deep reinforcement learning provided by the present application realizes intelligent scheduling of network exploration tasks through the adaptive learning ability of deep reinforcement learning.

[0034] (2) The distributed network exploration task intelligent scheduling method based on deep reinforcement learning provided by the present application can more effectively manage and optimize the task queue, can process both timed tasks and non-timed tasks, and ensure that they can be processed in time, greatly improving the efficiency and response speed of network exploration, and meeting the high requirements of the network security field for real-time performance and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is the overall flowchart of the intelligent scheduling system of the present application;

[0036] Figure 2 is a schematic diagram of the data transmission process in the intelligent scheduling system of the present application;

[0037] Figure 3 is an interaction schematic diagram of the deep reinforcement learning module in the present application;

[0038] Figure 4 is the distributed network exploration task intelligent scheduling system based on deep reinforcement learning;

[0039] Figure 5 is the deep reinforcement learning input vector for the exploration task;

[0040] Figure 6 is the deep reinforcement learning output vector for the exploration task. DETAILED DESCRIPTION

[0041] The present application will be further described in detail below in combination with the drawings and examples.

[0042] The distributed network exploration task intelligent scheduling system designed by the present application is as shown in Figure 4As shown, it includes a web console, a central node, and a scalable server cluster. The web console includes an information collection module and a display module, the central node includes an intelligent scheduling center, a detection result center, a log center, and a monitoring center, and the scalable server cluster includes a decision execution module (server) and monitoring points.

[0043] like Figure 4 As shown, users create new network probe tasks through the web console, specifying the task name, command format, and task characteristics. For periodic network probe tasks, cron expressions can be used to add tasks periodically, eliminating the need for manual repetition. After addition, the task is sent to the intelligent scheduling center, which splits the task and distributes the subtasks to different servers for execution.

[0044] The information collection module gathers the information required for task scheduling, namely the current server cluster information, the current task queue information, and the current scheduled task information. The server cluster information is automatically collected by each monitoring point deployed on the server, and the collected information is transmitted to both the intelligent scheduling center and the monitoring center.

[0045] like Figure 2 and Figure 3 As shown, the intelligent scheduling center includes an input transformation module, a deep reinforcement learning module, and an output transformation module. The intelligent scheduling center converts the collected scheduling information into input vectors through the input transformation module for use by the deep reinforcement learning module. After processing by the deep reinforcement learning module, the input vectors produce output vectors. These output vectors are then converted into reinforcement learning actions through the output transformation module. Reinforcement learning actions are divided into task decomposition and task scheduling. After tasks are scheduled on the server, they affect the current scheduling environment, and the changed environment information is used for scheduling subsequent tasks. By continuously performing this process, the intelligent scheduling center will continuously learn its task decision-making capabilities in different environments, thereby adapting to the ever-changing cyberspace.

[0046] Intelligent scheduling method for distributed network probing tasks based on deep reinforcement learning, such as Figure 1 and Figure 4 As shown, the specific workflow is as follows:

[0047] Step 1: Users create new network probing tasks through the web console. The information collection module collects scheduling information and transmits this scheduling information to the intelligent scheduling center.

[0048] The scheduling information includes the current status information of the server cluster, the current task queue information, and the task information to be scheduled.

[0049] Step two, the intelligent scheduling center preprocesses the scheduling information and transmits the obtained input vector to the deep reinforcement learning module for intelligent decision-making. The deep reinforcement learning module uses the learned strategy to generate an output vector, which represents the optimal task scheduling decision for the current state, such as task allocation, resource allocation priority, etc.

[0050] Preprocessing refers to converting scheduling information into an input vector that can be understood and processed by the deep reinforcement learning algorithm through the input conversion module.

[0051] As shown in Figure 5 , the specific format of the input vector is: n dimensions of server cluster state information, m dimensions of current scheduling task information, k dimensions of task queue information, and t dimensions of expandable dimensions.

[0052] 1: Current scheduling task information includes:

[0053] (1) Task CPU occupancy rate

[0054] (2) Task type

[0055] (3) Task affinity list

[0056] (4) Task anti-affinity list

[0057] (5) Total traffic required for task execution

[0058] (6) Task execution estimation time

[0059] (7) Task urgency

[0060] 2: Task queue information includes

[0061] (1) Task queue length

[0062] (2) Task queue urgency list

[0063] 3: Current server state: includes a binary flag, busy flag S and task remaining time LT.

[0064] Busy flag S: This is a binary identifier indicating the working state of the machine. Its value is 0 or 1, where 0 represents the machine is currently idle, and 1 indicates the machine is busy processing tasks.

[0065] Task remaining time LT: This measure represents how much time the task running on the current machine is expected to take to complete. This time is estimated based on the complexity and resource requirements of the task and is measured in specific time units such as minutes or seconds. In order to avoid excessive dispersion of task remaining time, which makes it difficult to train, a logarithmic mapping form is used, mapping to 1-r, a total of r levels.

[0066] In the reinforcement learning framework, the output space focuses on the subdivision and scheduling of tasks. Specifically, it involves the subtask splitting scheme of the current task and the corresponding scheduling decision.

[0067] Whether to split the subtask depends on whether the current subtask is the core of the probe task. If it is not the core of the probe task, but some preparation work, or a write-back operation after the probe ends, since the number of concurrent connections does not need to be considered, it does not need to be split; if it is the core of the probe task, it needs to be split to improve efficiency.

[0068] As shown in Figure 6 , the first k dimensions of the output vector represent the task splitting situation, which is converted into the corresponding reinforcement learning action by the output conversion module. The first k elements of the output vector are Task_Split_P i , i = 0, 1, 2, 3, … k, representing the probability that the task should be split into 2 i , the maximum value of which is Task_Split_P m , which represents splitting the task into corresponding 2 m , the latter n-dimensional vector represents the probability P m that the 2 j split tasks are scheduled to the jth machine. Finally, the probability P j is sorted from large to small, and the first 2 m machines with larger probability values are selected, and the split tasks are scheduled to the corresponding machines.

[0069] Step three, the output conversion module converts the output vector into a reinforcement learning action, and the intelligent task scheduling center sends the reinforcement learning action to the decision execution module to guide the execution of the network probe task.

[0070] Step four, after the task is executed, each server uploads the task execution information in log format to the log center, uploads the probe results to the probe result center, and the monitoring points deployed on the servers periodically upload the new cluster state information to the monitoring center and the intelligent scheduling center.

[0071] Step five, the intelligent scheduling center updates the deep reinforcement learning model according to the new cluster state information, returns to step two, and continues the above process. Through continuous learning and adaptation to update the reward function, the decision strategy is optimized to quickly respond to dynamic network environments and task demands. With the passage of time and the accumulation of more data, the decision-making ability of the scheduling algorithm will continuously improve, making the scheduling of network probe tasks more efficient and intelligent.

[0072] The reward function is:

[0073] Reward = a*Affinity_Bonus + b*Anti_Affinity_Penalty + g*Immediate_Reward

[0074] + d*Time_Cost + h*Flow_Cost

[0075] The reward function is divided into three parts, specifically:

[0076] The first part: Affinity reward and Anti-affinity penalty, which distinguishes whether the current task is scheduled to a node with affinity or Anti-affinity. For example, if the task is successfully scheduled to a node with high affinity, a positive reward (Affinity_Bonus) is given; on the contrary, if the task is assigned to a node with Anti-affinity, a high penalty term (Anti_Affinity_Penalty) will be generated. Among them, Affinity_Bonus and Anti_Affinity_Penalty are determined by the characteristics of the task, and the values here are 1500 and -5000 respectively, because in this scenario, the total duration of task execution is longer, and it is not desirable to have a task that cannot be scheduled.

[0077] The second part: Timeliness reward, in order to deal with the common reward delay problem in reinforcement learning, this design introduces a timely reward mechanism. This means that after each task scheduling decision, the system will immediately allocate rewards, rather than waiting until the task is completely finished. This is aimed at promoting the acceleration of the learning process and improving the responsiveness of the algorithm to immediate feedback. In this invention, it is recorded as Immediate_Reward, which is also an extremely important hyperparameter.

[0078] The third part: Time cost Time_Cost and flow cost Flow_Cost reward, which is used to guide the intelligent scheduling center when scheduling tasks. According to the actual application situation, the shortest time consumption and the least flow consumption can be selected as much as possible, because the detection task will consume a large amount of billing traffic, which must be considered.

[0079] Finally, a, b, g, d, h are hyperparameters that can be optimized through a tuning process to guide the agent's tendencies. Increasing the value of a will guide the agent to preferentially schedule tasks on nodes with affinity, which can increase the overall performance of the system and the efficiency of task processing. Increasing the value of b will cause the agent to give more consideration to avoiding anti-affinity nodes in task allocation, in order to reduce the high penalty cost due to incompatibility. At the same time, adjusting the value of g can affect the importance of timeliness rewards, making the algorithm pay more attention to fast response to speed up the learning process and provide timely feedback. The adjustment of d and h reflects the agent's preference for time and traffic costs, guiding the agent to meet task requirements while minimizing time and resource consumption. This flexible adjustment strategy is crucial for constantly changing tasks and network environments, as it ensures that the algorithm can adapt to various situations while maximizing performance and efficiency.

[0080] Step six, the information obtained by the monitoring center, the detection result center and the log center is transmitted to the Web console, and the cluster information, task log and detection result are displayed through the display module, and these information can be further used for scientific research analysis.

Claims

1. A distributed network probing task intelligent scheduling method based on deep reinforcement learning, characterized in that, The method comprises the following steps: Step one, the user creates a new network detection task through the Web console, the information collection module collects scheduling information, and transmits the scheduling information to the intelligent scheduling center; Step two, the intelligent scheduling center pre-processes the scheduling information, transmits the obtained input vector to the deep reinforcement learning module for intelligent decision-making, and the obtained output vector is the task scheduling decision of the current state; Pre-processing refers to converting the scheduling information into an input vector through an input conversion module; The input vector includes n-dimensional state information of the server cluster, m-dimensional current task information to be scheduled, k-dimensional task queue information, and t-dimensional scalable dimension; The input vector generates an output vector through the policy learned by the deep reinforcement learning module; The first k elements of the output vector are Task_Split_P i , i = 0, 1, 2, 3, … k, representing the probability of the task being split into 2 i parts, taking the maximum value Task_Split_P m , which represents splitting the task into corresponding 2 m parts; the following n-dimensional vector represents the probability P m of scheduling the 2 j parts of the task split in the front to the jth machine, respectively; finally, according to the probability P j from large to small, take the first 2 m machines with larger probability values, and schedule the split tasks to the corresponding machines; Step three, the output vector is converted into a reinforcement learning action through an output conversion module, and the intelligent scheduling center sends the reinforcement learning action to the decision execution module to guide the execution of the network detection task; Step four, after the task is executed, each server uploads the task execution information in log format to the log center, uploads the detection results to the detection result center, and the monitoring points deployed on the servers periodically obtain new cluster state information and upload it to the monitoring center and the intelligent scheduling center; Step five, the intelligent scheduling center updates the deep reinforcement learning model according to the new cluster state information, returns to step two, and optimizes the task scheduling decision through the reward function; The reward function is: Reward = α·Affinity_Bonus + β·Anti_Affinity_Penalty + γ·Immediate_Reward + δ·Time_Cost + η·Flow_Cost Wherein, α, β, γ, δ, η are hyperparameters, Affinity_Bonus is a positive reward, Anti_Affinity_Penalty is a penalty term, Immediate_Reward is a timeliness reward, Time_Cost is a time cost reward, and Flow_Cost is a flow cost reward; Step six, the information obtained by the monitoring center, the detection result center and the log center is transmitted to the Web console, and the cluster information, task log and detection result are displayed through the display module, and the information is further used for scientific research analysis.

2. The method of claim 1, wherein, The scheduling information includes current task information to be scheduled, task queue information and current server cluster information.

3. The method of claim 2, wherein, The current task information to be scheduled includes task CPU occupancy, task type, task affinity list, task anti-affinity list, total task execution traffic, task execution estimation time and task urgency.

4. The method of claim 2, wherein, The task queue information includes task queue length and task queue urgency list.

5. The method of claim 2, wherein, The current server cluster information includes busy flag S and task remaining time LT, wherein the busy flag S is a binary identifier indicating the working state of the machine; its value is 0 or 1, wherein 0 represents that the machine is currently idle, and 1 indicates that the machine is busy processing tasks; the task remaining time LT represents the estimated remaining time of the running task on the current machine.

6. The method of claim 1, wherein, The design of the reward function includes: The first part: affinity reward and anti-affinity penalty, which distinguishes whether the current task is scheduled to a node with affinity or anti-affinity; if the task is scheduled to a node with high affinity, a positive reward Affinity_Bonus is given; on the contrary, if the task is assigned to a node with anti-affinity, a penalty term Anti_Affinity_Penalty will be generated; The second part: the timeliness reward Immediate_Reward, which will be immediately assigned after each task scheduling decision; The third part: the reward of time cost Time_Cost and flow cost Flow_Cost, which is used to guide the intelligent scheduling center to select the scheduling task; The fourth part: α, β, γ, δ, η are hyperparameters, which are optimized through the parameter adjustment process to guide the tendency of the intelligent scheduling center.

7. The method of claim 6, wherein the method further comprises: The optimization of the hyperparameters α, β, γ, δ, η is as follows: Increasing the value of α will guide the intelligent scheduling center to prefer to schedule tasks to nodes with affinity; Increasing the value of β will cause the intelligent scheduling center to avoid anti-affinity nodes in task allocation; Adjusting the value of γ can affect the importance of timeliness reward, accelerate the learning process and feedback in time; Adjusting δ and η reflects the preference of the intelligent scheduling center in terms of time and flow cost, guiding the intelligent scheduling center to reduce time and resource consumption.

8. An intelligent scheduling system based on the distributed network probing task intelligent scheduling method based on deep reinforcement learning of claim 1, 2 or 6, characterized in that, It includes a Web console, a center node and a scalable server cluster; wherein the Web console includes an information collection module and a display module, the center node includes an intelligent scheduling center, a probe result center, a log center and a monitoring center, and the scalable server cluster includes a decision execution module and a monitoring point.

9. The intelligent dispatch system of claim 8, wherein, The intelligent scheduling center includes an input conversion module, a deep reinforcement learning module and an output conversion module.

Images