Green space-time task scheduling and hybrid energy collaborative optimization method for computing power network

Abstract

The application discloses a kind of computing power network green space-time task scheduling and mixed energy collaborative optimization method.The steps are as follows: one, establish the computing power network system model of fusing computing resources, network routing resources and mixed energy, construct multi-objective joint optimization problem;Two, model the problem as a Markov decision process, define the state space containing task, network and energy state, and the action space composed of computing power node selection, routing path selection and forwarding time;Three, use the integrated deep reinforcement learning framework, configure multiple intelligent agents with different preference weights for parallel training, to achieve the best balance between quality of service and carbon emissions;Four, according to the integrated strategy of complete training, combined with the real-time input task flow and carbon emission intensity, output the optimal space-time scheduling decision and energy storage charging and discharging strategy.

Description

Technical Field

[0002] This invention relates to the fields of Computing Power Networks (CPNs) and green computing technology, and more specifically, to a green spatiotemporal task scheduling and hybrid energy collaborative optimization method for computing power networks based on deep reinforcement learning. Background Technology

[0004] In recent years, with the rapid development of information and communication technologies, especially the popularization of AI-generated content, machine learning, and large-scale concurrent computing, global data centers and computing infrastructure are facing unprecedented computing pressure and energy consumption challenges. The traditional "cloud-edge-device" architecture mainly focuses on resource allocation within a single domain, which easily leads to resource silos. Computing Power Networks have emerged to address this issue, achieving deep integration of computing power and network resources by perceiving and scheduling global heterogeneous computing power through the network control plane.

[0005] However, the high-load operation of computing networks is accompanied by massive energy consumption and carbon emissions. Introducing renewable energy sources (green energy) such as wind and solar power to power computing networks has become an inevitable trend. However, renewable energy has strong spatiotemporal randomness and intermittency; simultaneously, the carbon emission intensity of electricity supplied by the power grid fluctuates significantly across different time periods. Existing task scheduling methods are often limited to the spatial dimension (node ​​selection) or the temporal dimension (delayed execution), and typically employ a single-objective optimization model that simply linearly weights energy consumption and latency. This method cannot simultaneously consider the coupling characteristics of computing network resources, the uncertainty of renewable energy, or provide a flexible Pareto optimal strategy set based on the decision-maker's risk preferences. Therefore, how to design a task scheduling mechanism that integrates spatiotemporal dimensions and collaboratively manages hybrid energy to achieve truly green computing is a pressing technical challenge that needs to be addressed. Summary of the Invention

[0007] This invention discloses a green spatiotemporal task scheduling and hybrid energy collaborative optimization method for computing networks. It primarily addresses how, in a hybrid energy power supply environment, computing networks can minimize system carbon emissions and maximize Quality of Service (QoS) by jointly allocating computing resources, network routing, and decision execution time when processing heterogeneous tasks. The steps of the method are as follows:

[0008] Step 1: Establish a system model based on the computing network architecture. This model consists of three parts: a task model, a latency model, and an energy / carbon emission model. After a task arrives at the edge router, its total latency is defined as the sum of transmission latency, node queue waiting latency, and computation execution latency. Transmission latency is limited by the real-time bandwidth of the network link, waiting latency depends on the FIFO mechanism of the node buffer queue, and computation latency depends on the computational load of the task and the node CPU frequency. The system's energy is provided jointly by renewable energy sources, energy storage batteries, and the brown grid. Carbon emissions... The carbon intensity is determined by multiplying the consumed brown energy by the real-time grid carbon intensity. This forms the basis of a multi-objective optimization problem.

[0009] Step 2: Define a Markov Decision Process (MDP) framework for multi-objective optimization and construct a multi-dimensional joint state space. This space senses in real time the characteristics of tasks to be scheduled, the status of computing node buffers and computing queues, the available bandwidth of network links, the state of charge of energy storage systems, and the real-time carbon intensity of the power grid; the design encompasses a hybrid discrete action space with three-dimensional characteristics of "space-time-network". It performs a “triple selection”: selecting the target computing node, selecting the routing path from the edge to the node, and determining the actual forwarding time of the task within the tolerance latency range. At the same time, it realizes the mapping and rounding from the continuous action space to the discrete action space to meet the needs of actual scheduling decisions.

[0010] Step 3: Design a Spatiotemporal Task Scheduling (STST) algorithm based on Ensemble Deep Reinforcement Learning (EDRL). This algorithm is built upon the dual-delay deep deterministic policy gradient (TD3) framework. Addressing the inherent conflict between QoS guarantees (requiring tasks to execute as quickly as possible) and carbon emission minimization (requiring waiting until carbon intensity is low or green electricity is abundant), an ensemble learning mechanism is introduced. Multiple base agents with different multi-objective preference weights are configured for parallel training. Each agent employs a two-layer fully connected neural network structure, exploring higher-order nonlinear relationships between states and actions in parallel within a shared environment. Network parameters are iteratively optimized through an experience replay mechanism, a dual-critic network to avoid Q-value overestimation, delayed updates in the actor network, and a soft update strategy in the target network. Simultaneously, delay constraints and load balancing constraints are integrated into the reward function, and a penalty term is introduced to construct a constrained reward function, ensuring that the hard constraints of task execution and system operation are met. Finally, a set of scheduling policy models covering the Pareto front is obtained.

[0011] Step 4: Online Deployment, Decision Execution, and Hybrid Energy Linkage. In actual operation, edge routers collect task requests and dynamic carbon intensity curves issued by the power grid in real time, inputting them into a pre-trained integrated policy network. Based on the operator's current operational preferences for service quality and energy conservation and emission reduction, the network selects the corresponding agent or conducts policy integration voting, outputting the optimal time-spacing command. The computing network then temporarily stores the task or immediately routes it to the designated node along the selected path according to this command. Simultaneously, the energy management module, based on the real-time power load generated during the execution of computing tasks, coordinates and controls the charging and discharging behavior of energy storage units according to the principle of green electricity priority, energy storage second, and power grid last. Peak shaving energy storage is implemented when renewable energy is abundant or the grid carbon intensity is low, and valley filling power supply is implemented when the load is high or the carbon intensity is high. The battery state of charge is dynamically updated, thereby maximizing the utilization rate of green energy and reducing the overall network carbon footprint while ensuring the continuity of computing services and QoS indicators. Attached Figure Description

[0013] To more clearly illustrate the technical methods in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below.

[0014] Figure 1 The graph shows the convergence performance curve and energy consumption trend curve of the method of the present invention as the number of training rounds increases.

[0015] Figure 2 The task delay constraint violation rate curve for the method of the present invention when the number of training rounds increases.

[0016] Figure 3 The carbon emission curve of the computing power network when the training rounds of the method of the present invention are increased.

[0017] Figure 4 A comparison chart of the STST algorithm based on TD3 and four benchmark algorithms. Detailed Implementation

[0019] This invention proposes a green spatiotemporal task scheduling and hybrid energy collaborative optimization method for computing power networks. The following detailed description of the implementation is provided with reference to the accompanying drawings and simulation experiments. The specific implementation scenario simulates a network topology consisting of 20 computing power nodes, 5 edge routers, 10 ordinary routers, and 20 computing power network routers. The system power supply comprises the power grid, a photovoltaic / wind power generation array, and an energy storage battery array. The data volume of the tasks is randomly distributed between [10, 40] MB, and the computational load is between [1, 5] Gcycles. The computing power of the computing nodes ranges from [8, 20] Gcycles / s. Based on real power grid data, the carbon intensity of the power grid is set to dynamically fluctuate over time within a day, and the charging and discharging efficiency of the energy storage batteries is... In the EDRL algorithm proposed in this invention, the hyperparameters of the spatiotemporal task scheduling (STST) algorithm based on dual-delay deep deterministic policy gradient (TD3) are selected as follows: the learning rate is set to 5×10. −5 Discount factor The target network soft update coefficient is 0.95. The initial value is 0.01, the experience replay pool capacity is 100,000, and the mini-batch size for each sampling is 64. A Gaussian distribution strategy is used for exploration noise, with a noise clipping value of 0.2, an initial exploration noise value of 0.4, a strategy noise of 1, and a delayed update step size. Set to 2, delay penalty weight =500, load balancing penalty weight =5000, the algorithm trained for a total of 600 learning rounds.

[0020] The specific implementation steps are as follows:

[0021] 1) Edge routers in time slots Upon receiving a user request, a request is sent to the centralized scheduler in the control plane. The scheduler collects multi-dimensional status information across the entire network, including: real-time available bandwidth of each link. Buffer queues for each computing node With computation queue Real-time estimated value of wind and solar power generation Battery state of charge And the real-time carbon intensity of the power grid. All state variables are normalized to the [0,1] interval to eliminate the gradient vanishing problem caused by dimensional differences, and finally a high-dimensional state vector is constructed. .

[0022] 2) Parallel evaluation of multi-preference strategies: The states are evaluated in parallel. Simultaneously, the input is fed into three predefined base agents with predefined preferences. Each agent's Actor network, based on the TD3 framework, outputs continuous action values ​​(range).

[0023] [-1,1]), for the three types of decisions—computing node selection, routing path selection, and forwarding time selection—continuous action space is discretized through mapping and rounding. The mapping formula is as follows:

[0024]

[0025]

[0026]

[0027] Where round(⋅) is the floor function. This represents the number of computing nodes. Number of routing paths The maximum time slot is defined. Each agent performs forward propagation through a two-layer fully connected neural network to calculate the Q-value of all selectable discrete actions.

[0028] 3) Execution of time-based scheduling actions with constraints and penalties: When using online applications, select the corresponding action combination based on the currently set operational goals of the system. This action clearly indicates the task. We need to wait in the edge buffer until time Z. ∗ Then through the gth ∗ The route is transmitted to the computing node m. ∗ .

[0029] Simultaneously, latency constraints and load balancing constraints are integrated into the reward function, and a penalty term is introduced to construct a constrained reward function, ensuring that the task execution time does not exceed the maximum tolerable latency and the system load does not exceed the upper limit. The reward function after penalty is as follows:

[0030]

[0031] in, This represents the number of tasks in the current time slot. The actual execution time of the task. The maximum tolerable delay for the task.

[0032] 4) Hybrid Energy Response: During the corresponding time slot of task execution, the energy management unit determines the energy consumption requirements of the computing node. Responding to the current green electricity surplus, the principle of prioritizing green electricity, followed by energy storage, and lastly, the power grid should be followed. > The excess electricity is calculated using the formula Store the battery; if insufficient, use the battery as the primary energy source. The system efficiently releases battery power, and any shortfall is converted into brown electricity from the grid, with carbon emissions calculated cumulatively.

[0033] 5) Results Analysis: Figure 1 The changes in reward value and network energy consumption with the number of training rounds are described. As the number of training rounds increases, the algorithm reward rises rapidly and converges steadily, while the system energy consumption continues to decrease. This is because the model gradually finds the optimal scheduling strategy through learning, improving resource utilization and reducing energy consumption. These results verify that the method of this invention has the advantages of fast convergence speed, stable optimization effect, and significantly reduced energy consumption.

[0034] Figure 2The study describes the change in the task latency constraint violation rate with the number of training epochs. As the number of training epochs increases, the task latency constraint violation rate continuously decreases and remains at a low level. This is because the scheduling strategy can rationally plan the task migration path and execution time, accurately matching differentiated latency requirements. These results demonstrate that the present invention can effectively schedule massive tasks in a computing network, ensuring the quality of computing services.

[0035] Figure 3 The paper describes the change in carbon emissions of a computing network with the number of training epochs. As the number of training epochs increases, carbon emissions decrease significantly and tend to plateau. This is because the algorithm learns to prioritize the use of green electricity and avoid performing tasks during periods of high carbon intensity. These results validate that the present invention, through spatiotemporal joint scheduling and hybrid energy synergy, can significantly reduce carbon emissions from computing networks.

[0036] Figure 4 This paper describes a performance comparison between the STST algorithm of this invention and four benchmark methods: RANDSS, STSD, STTSWS, and STS. RANDSS stands for Random Scheduling Algorithm, which allocates computing nodes, routing paths, and forwarding times to tasks completely randomly without any optimization logic. STSD is a DDPG-based spatiotemporal scheduling algorithm that uses traditional DDPG reinforcement learning methods to simultaneously make scheduling decisions regarding node selection, path selection, and forwarding time selection. STTSWS is a spatiotemporal scheduling algorithm without energy storage; although it performs spatiotemporal joint scheduling, the system does not have energy storage batteries to balance or suppress fluctuations in renewable energy. STS is a spatial-only scheduling algorithm that only selects nodes and paths without optimizing the forwarding time of tasks. The proposed STST spatiotemporal task scheduling method based on integrated TD3 achieves a convergence speed improvement of approximately 30% compared to the DDPG-based benchmark algorithm and significantly outperforms the traditional RANDSS, STTSWS, and STS schemes. The reason is that this method adopts a two-layer fully connected neural network structure, which avoids training oscillations caused by overestimation of value through a dual critic network. At the same time, it adopts a delayed policy update to make the training more stable. In addition, it optimizes the three decisions of node, path, and forwarding time, and incorporates latency, load balancing and carbon emissions into the reward to guide the update of the neural network. Thus, it integrates multi-agent training to quickly find the optimal policy. Therefore, its overall learning efficiency and decision accuracy are far higher than other benchmark algorithms.

[0037] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept.

Claims

1. A green spatiotemporal task scheduling and hybrid energy collaborative optimization method for computing power networks, which addresses the joint optimization of task scheduling and energy management when a computing power network powered by hybrid energy provides computing services to users. The method comprises the following steps: Step 1: Establish a computing power network model comprising M computing power nodes, R routers, and a hybrid energy supply system (renewable energy, energy storage batteries, brown grid), derive the delay formula for task transmission and execution, and derive the carbon emission formula by combining the dynamic grid carbon intensity. The spatiotemporal task scheduling problem of joint optimization of carbon emissions and delay is described as follows: s.t. in, For scheduling strategy; The system reward function includes carbon emission penalties and QoS rewards; and These represent the actual execution time of the task and the maximum tolerable delay, respectively. The decision-making and forwarding time for the task; For the overall load balancing of the system, Maximum load; Indicates whether the task has been scheduled to a computing node. ; ∈{0,1} represents whether to select the first... One routing path; The solution steps for the multi-objective joint optimization problem are as follows: First, model the problem as a Markov decision process (MDP) and define the state space of the system. These include task attributes, node queue status, link status, and hybrid energy status (including current renewable energy generation). and battery state of charge Define the action space It covers the joint selection of nodes, paths, and times; To address the Pareto front search problem caused by multi-objective conflicts, an ensemble deep reinforcement learning algorithm based on dual-delay deep deterministic policy gradient (TD3) is employed to construct N algorithms with different preference weights. The intelligent agent calculates the weighted reward function. The evaluation network and target network of each agent are parameterized by minimizing the following mean squared error loss function. Update: Among them, the target value , For micro-batch samples in the experience replay pool, Discount factor; Finally, in the online scheduling phase, the real-time task status and grid carbon intensity characteristics are input into the trained integrated policy network, and multiple agents output the final optimal time and air scheduling decision based on majority voting or weighted aggregation mechanisms. It guides task transmission and directs the charging and discharging of energy storage systems, thereby achieving synergistic optimization of carbon emissions and QoS in computing networks.

Images