Iot collection platform computing resource scheduling method based on reinforcement learning

Abstract

This invention relates to the field of IoT resource scheduling technology, specifically a computational resource scheduling method for IoT data acquisition platforms based on reinforcement learning. The invention constructs an IoT data acquisition platform, determines its state space, action space, reward function, and objective equation; introduces the Double DQN algorithm to construct a reinforcement learning network; and trains the reinforcement learning network to obtain the optimal intelligent scheduling scheme for computational resources. This invention utilizes the Double DQN algorithm to improve the task scheduling method of building IoT data acquisition platforms, significantly improving task scheduling efficiency, reducing processing latency, and effectively lowering the operating costs of building IoT data acquisition platforms.

Description

Technical Field

[0001] This invention relates to the field of IoT resource scheduling technology, specifically a method for scheduling computational resources of an IoT data acquisition platform based on reinforcement learning. Background Technology

[0002] With the rapid growth in the number of industrial IoT field nodes, the increasing frequency of data collection, and the expanding range of data collection attributes, the amount of data connected to IoT data collection platforms is growing rapidly, placing more stringent demands on the platform's processing speed, resource utilization efficiency, and other performance aspects. IoT data collection platforms contain numerous computing resources, including GPUs, CPUs, and memory; how these resources are scheduled directly impacts the platform's performance.

[0003] To address the aforementioned technical issues, patent application CN107819866A discloses an IoT data acquisition and control platform. The platform's structure includes a field data acquisition unit, a network transmission unit, a data processing middleware, and a data service center, all connected in series. The field data acquisition unit collects various types of field data and uploads it to the data processing middleware for processing via the network transmission unit. The data service center then implements business management based on the data processing middleware's processing. These existing technologies provide customers with a flexible, integrated IoT information acquisition and control system that combines comprehensive business systems. While these existing technologies can improve the level of information acquisition, the scheduling of computing resources such as GPU, CPU, and memory in the control platform typically prioritizes maximizing information acquisition without employing a suitable computing resource scheduling scheme. This leads to excessive consumption of control platform resources while reducing computational efficiency, thus requiring a solution. Summary of the Invention

[0004] To avoid and overcome the technical problems existing in the prior art, this invention provides a computational resource scheduling method for an IoT data acquisition platform based on reinforcement learning. This invention can allocate reasonable computational resource scheduling schemes to computational tasks, effectively improving computational efficiency while reducing computational resource consumption.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A computational resource scheduling method for IoT data acquisition platforms based on reinforcement learning includes the following steps: S1. Construct an IoT data acquisition platform consisting of multiple computing resources, including CPU and GPU resources; use the IoT data acquisition platform as a training environment for a reinforcement learning model to determine the state space of the training environment. S2. The scheduling algorithm is used as the agent of the reinforcement learning model. By utilizing the CPU and GPU resources in the IoT acquisition platform, the execution time of each task and the maximum acceptable execution delay time of each task are initially estimated. The scheduling range of CPU and GPU resources in the time calculation process is used as the action space of the agent.

[0006] S3. With the goal of minimizing the execution time of each task and minimizing the maximum acceptable execution delay of each task, determine the reward function of the reinforcement learning model and obtain the scheduling scheme of CPU and GPU resources corresponding to this goal. S4. Introduce a Double DQN network and use it as an agent. Simultaneously train the Double DQN network to obtain the optimal reinforcement learning model. S5. Input each task into the optimal reinforcement learning model, and obtain the CPU and GPU resource scheduling scheme corresponding to the minimum execution time and maximum acceptable execution delay time of each task by minimizing the reward function.

[0007] As a further embodiment of the present invention: the Internet of Things (IoT) data acquisition platform is a computing cluster consisting of an IoT data access and parsing server based on CPU and GPU resources, a data forwarding and big data stream processing server based on CPU and GPU resources, and a data storage and visualization server based on CPU and GPU resources. The computing cluster uses the start and end times of the computing tasks, as well as the resource status of the data access and parsing servers during the execution of the computing tasks, as the state space of the reinforcement learning model. S , ;in, For one A 3D matrix Indicating that in each round of scheduling, M Each computational task N Composed of a combination of start times M OK N A matrix of columns; For one A 3D matrix Indicating that in each round of scheduling, M Each computational task N Composed of a combination of end times M OK N A matrix of columns; For one A 3D matrix This indicates that in each round of scheduling, M A computing task in KThe distribution state running on each edge node.

[0008] As a further aspect of the present invention: the action space output by the scheduling algorithm For one A 3D matrix Each element in the table represents the scheduling of the corresponding computing task to... K The probability value Q of each edge node; action space The Middle m Line 1 k Column elements Q mk The value and scheduling algorithm will compute the task m Dispatch to edge computing nodes k The probability is directly proportional to the action space. It is expressed as follows:

[0009] in, Q π ( s , a ) represents the action space The matrix name, Indicates spatial state. Indicates an action; for Q π ( s , a The element in the first row and first column of the array; for Q π ( s , a The first line of the text K Column elements; for Q π ( s , a ) M The element in the first row and first column; for Q π ( s , a ) M Line 1 K Column elements; The agent selects actions based on the following formula:

[0010] in, This indicates that the computation task will be performed. m Dispatch to edge computing nodes k The action; This indicates the operation of retrieving the maximum value; Indicates that the agent is in the first... i The actions of round-robin scheduling Indicates the spatial state.

[0011] As a further aspect of this invention: the scheduling algorithm outputs action decisions for different computational tasks, and then submits the computational tasks to different computational nodes according to the action decisions. After scheduling is completed, before the next round of scheduling begins, rewards are calculated based on the observed average task execution time and task execution delay; i The reward for round-robin scheduling is defined as:

[0012] in, Indicates the first i Rewards for round-robin scheduling; This represents the reward constant, used to adjust the value of the current reward. Indicates the first i The value function value of round-robin scheduling Indicates the first i A spatial state; Minimum value of the value function; and All are variable reward control functions; Indicates the threshold of the value function; Indicates range; The specific calculations are as follows:

[0013] in, Indicates the first i During round-robin scheduling K A matrix of the number of tasks on each edge node; Indicates to Variance calculation operation; The weight representing the average completion time of tasks on the first computing node; Indicates the first K The corresponding weights of the average task completion time on each computing node; and All are variable reward control functions; This indicates the execution time of the first computation task; Indicates the first K The execution time of each computational task.

[0014] As a further embodiment of the present invention, the specific steps of step S4 are as follows: S41. Initialize the training rounds to 0; S42, using random parameters Initialize the Double DQN network and obtain the trained network. ; S43, Usage Parameters To initialize the target network ; S44. Initialize the experience playback device; S45. Iteration: Obtaining the state space Based on the current training network Greedy strategy Select an action, execute the action, receive a report, and update the state space. S ; S46, Experience Store the experience data in the experience playback device; if the amount of experience data in the experience playback device reaches the set quantity, randomly select data from the experience playback device. N Group data ,use and Calculate the loss function and update accordingly. Parameters; S47. When performing a set algebraic C steps in the iteration, Parameters synchronized to The goal is to obtain the optimal target network, which is the corresponding optimal reinforcement learning model.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention utilizes the Double DQN algorithm to improve the task scheduling method of the IoT data acquisition platform, which greatly improves the processing efficiency of task scheduling, reduces processing latency, and effectively reduces the operating cost of the IoT data acquisition platform. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the main process structure of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 In this embodiment of the invention, the computational resource scheduling method for an IoT data acquisition platform based on reinforcement learning mainly includes the following: 1. Build an IoT data acquisition platform.

[0019] The IoT data acquisition platform includes a scheduler controller, data middleware, data storage, and computing resources. The IoT data acquisition platform is used as a reinforcement learning training environment, and the state space of the environment is defined. A GPU and CPU cluster testing environment is built, the scheduler controller and scheduler command executor programs are developed, the RabbitMQ message middleware system is deployed, a data storage and publishing system is developed, and the reinforcement learning network runtime environment is deployed.

[0020] 2. Treat the scheduling algorithm as an intelligent agent.

[0021] Based on the preliminary estimates of the execution time of each task and the maximum acceptable task execution delay by the IoT data acquisition platform, the action space of the intelligent agent is determined.

[0022] Given a data acquisition platform P and a task queue The goal of intelligent scheduling methods is to find suitable placement strategies and execution sequences to maximize the utilization of the acquisition platform's computing cluster while satisfying the Quality of Service (QoS) for each task.

[0023] The state space of the Double DQN deep network is defined by the start and end times of the computation tasks and the resource status of the data access and parsing servers. It indicates. Among them, For one A 3D matrix Indicating that in each round of scheduling, M Each computational task N Composed of a combination of start times M OK N A matrix of columns. For example, the start time of the scheduling process is... If a computational task is executed three times in one scheduling round, then the computational task's... N =3, the start time is expressed as .

[0024] For one A 3D matrix Indicating that in each round of scheduling, M Each computational task N Composed of a combination of end times M OK N A matrix of columns; For one A 3D matrix This indicates that in each round of scheduling, M A computing task in K The distribution state running on each edge node.

[0025] The action space output by the Double DQN scheduling algorithm For one A dimensional matrix, representing the... M A computing task was scheduled to K The probability value of each edge computing node Q Action space The Middle m Line 1 k Column elements Q mk The value and scheduling algorithm will compute the task m Dispatch to edge computing nodes k The probability is directly proportional to the action space. It is expressed as follows:

[0026] in, Q π ( s , a ) represents the action space The matrix name, Indicates spatial state. Indicates an action; for Q π ( s , a The element in the first row and first column of the array; for Q π ( s , a The first line of the text K Column elements; for Q π ( s , a ) M The element in the first row and first column; for Q π ( s , a ) M Line 1 K The elements of the column.

[0027] The agent selects actions based on the following formula:

[0028] in, This indicates that the computation task will be performed. m Dispatch to edge computing nodes k The action; This indicates the operation of retrieving the maximum value; Indicates that the agent is in the first... iThe actions of round-robin scheduling Indicates the spatial state.

[0029] 3. Determine the reward function with the goal of minimizing the execution time and latency of each task.

[0030] The scheduling algorithm outputs action decisions for different computational tasks, and then submits the computational tasks to different computing nodes according to the action decisions. After scheduling is completed, before the next round of scheduling begins, rewards are calculated based on the observed task execution time (JobCompletion Time, JCT) and task execution latency. i The reward for round-robin scheduling is defined as:

[0031] in, Set to 100. Set it to 10. rw Used to adjust the value of the current reward. The specific calculations are as follows:

[0032] Set it to 0.01. Set it to 20.

[0033] 4. By introducing a Double DQN network, an optimal intelligent scheduling scheme for computing resources of the IoT data acquisition platform is obtained.

[0034] The training rounds are initialized to 0.

[0035] Use random parameters Initialize the Double DQN network and obtain the trained network. .

[0036] Use parameters To initialize the target network .

[0037] Initialize the experience replayer.

[0038] Iterative process: Obtaining the state space Based on the current training network Greedy strategy Select an action, execute the action, receive a report, and update the state space. S ; Experience Store the experience data in the experience playback device; if the amount of experience data in the experience playback device reaches the set quantity, randomly select data from the experience playback device. N Group data ,use and Calculate the loss function and update accordingly. Parameters; When performing a given algebraic C steps in iteration, Parameters synchronized to The goal is to obtain the optimal target network, which is the corresponding optimal reinforcement learning model.

[0039] Since offline simulations also require estimating the average value for each task JCT To obtain the reward value for the current state. The completion time of a computation task is related to the actual computing resources used by the tasks running on the current node. When multiple machine learning tasks run on a node, the computing resource usage time for each task is relatively reduced. Based on the average computation task... JCT Comparing experimental results with execution overlap time, assuming the task is at an edge node. k running JCT With the average of this task JCT With nodes k The overlap time interval between task executions is related to the time interval between tasks, and is estimated using the following calculation method:

[0040] in, Indicates the first k The sum of the task overlap times on each edge node is calculated using the start and end times of all computational tasks on that node. Indicates the first k The number of tasks on each edge node.

[0041] Indicates when only the first m A computing task runs on an edge node. k The average JCT over time can be obtained by deploying the task to the node for actual measurement. and All of these are variable weighting coefficients.

[0042] To evaluate the proposed DQN reinforcement learning algorithm framework, it is first compared with the DDPG-DQN algorithm. For each agent, the training rounds are set to 1000, with each round containing 100 steps. In each step, the agent must continuously make decisions, interact with the environment, and update the network model based on environmental feedback. It can be seen that the average reward of the Double DQN reinforcement learning algorithm of this invention increases with the number of training rounds, eventually converging. This trend indicates that MID can still learn effective policies without any prior knowledge. Note that the proposed Double DQN reinforcement learning algorithm begins to converge after 380 rounds and gradually stabilizes after 600 rounds, demonstrating that the algorithm designed using the Double DQN network has a faster convergence speed and better stability.

[0043] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A computational resource scheduling method for an IoT data acquisition platform based on reinforcement learning, characterized in that, Includes the following steps: S1. Construct an IoT data acquisition platform consisting of multiple computing resources, including CPU and GPU resources; use the IoT data acquisition platform as a training environment for a reinforcement learning model to determine the state space of the training environment. S2. The scheduling algorithm is used as the agent of the reinforcement learning model. The execution time of each task and the maximum acceptable execution delay time of each task are estimated by utilizing the CPU and GPU resources in the IoT acquisition platform. The scheduling range of CPU and GPU resources in the time calculation process is used as the action space of the agent. S3. With the goal of minimizing the execution time of each task and minimizing the maximum acceptable execution delay of each task, determine the reward function of the reinforcement learning model and obtain the scheduling scheme of CPU and GPU resources corresponding to this goal. S4. Introduce a Double DQN network and use it as an agent. Simultaneously train the Double DQN network to obtain the optimal reinforcement learning model. S5. Input each task into the optimal reinforcement learning model, and obtain the CPU and GPU resource scheduling scheme corresponding to the minimum execution time and maximum acceptable execution delay time of each task by minimizing the reward function. The IoT data acquisition platform is a computing cluster consisting of IoT data access and parsing servers based on CPU and GPU resources, data forwarding and big data stream processing servers based on CPU and GPU resources, and data storage and visualization servers based on CPU and GPU resources. The computing cluster uses the start and end times of the computing tasks, as well as the resource status of the data access and parsing servers during the execution of the computing tasks, as the state space of the reinforcement learning model. S , ;in, For one A 3D matrix Indicating that in each round of scheduling, M Each computational task N Composed of a combination of start times M OK N A matrix of columns; For one A 3D matrix Indicating that in each round of scheduling, M Each computational task N Composed of a combination of end times M OK N A matrix of columns; For one A 3D matrix This indicates that in each round of scheduling, M A computing task in K The distribution state running on each edge node; The scheduling algorithm outputs action decisions for different computational tasks, and then submits the computational tasks to different computing nodes according to the action decisions. After scheduling is completed, rewards are calculated based on the observed average task execution time and task execution latency before the next round of scheduling. i The reward for round-robin scheduling is defined as: in, Indicates the first i Rewards for round-robin scheduling; This represents the reward constant, used to adjust the value of the current reward. Indicates the first i The value function value of round-robin scheduling Indicates the first i A spatial state; Minimum value of the value function; and All are variable reward control functions; Indicates the threshold of the value function; Indicates range; The specific calculations are as follows: in, Indicates the first i During round-robin scheduling K A matrix of the number of tasks on each edge node; Indicates to Variance calculation operation; The weight representing the average completion time of tasks on the first computing node; Indicates the first K The corresponding weights of the average task completion time on each computing node; and All are variable reward control functions; This indicates the execution time of the first computation task; Indicates the first K The execution time of each computational task; The specific steps of step S4 are as follows: S41. Initialize the training rounds to 0; S42, using random parameters Initialize the Double DQN network and obtain the trained network. ; S43, Usage Parameters To initialize the target network ; S44. Initialize the experience playback device; S45. Iteration: Obtaining the state space Based on the current training network Greedy strategy Select an action, execute the action, receive a report, and update the state space. S ; S46, Experience Store the experience data in the experience playback device; if the amount of experience data in the experience playback device reaches the set quantity, randomly select data from the experience playback device. N Group data ,use and Calculate the loss function and update accordingly. Parameters; S47. When performing a set algebraic C steps in the iteration, Parameters synchronized to The goal is to obtain the optimal target network, which is the corresponding optimal reinforcement learning model.

2. The method for scheduling computing resources for an IoT data acquisition platform based on reinforcement learning according to claim 1, characterized in that, Action space output by the scheduling algorithm For one A 3D matrix Each element in the table represents the scheduling of the corresponding computing task to... K The probability value Q of each edge node; action space The Middle m Line 1 k Column elements Q mk The value and scheduling algorithm will compute the task m Dispatch to edge computing nodes k The probability is directly proportional to the action space. It is expressed as follows: in, Q π ( s , a ) represents the action space The matrix name, Indicates spatial state. Indicates an action; for Q π ( s , a The element in the first row and first column of the array; for Q π ( s , a The first line of the text K Column elements; for Q π ( s , a ) M The element in the first row and first column; for Q π ( s , a ) M Line 1 K Column elements; The agent selects actions based on the following formula: in, This indicates that the computation task will be performed. m Dispatch to edge computing nodes k The action; This indicates the operation of retrieving the maximum value; Indicates that the agent is in the first... i The actions of round-robin scheduling Indicates the spatial state.

Images