A battery scheduling method and device for a multi-edge service provider scenario

Abstract

The present application relates to a kind of battery scheduling method and device for multi-edge service provider scene.The method includes: in the edge computing region of interest consisting of multiple edge service providers and multiple batteries, establish an independent agent for each edge service provider, and redefine: 1) the action executed by agent to reduce the dimension of action space, and 2) the reward after executing action, to align optimization goal, i.e. reduce carbon emission and battery degradation cost.Distributed training and distributed execution strategy are used, and Q value money disturbance strategy is added, so as to isolate the training and inference of each agent, realize information protection, in addition, action conversion mechanism is designed, the defined action is converted into the actual charge and discharge operation of battery in agent, improve the overall training and execution efficiency.The present application has the ability to effectively coordinate multiple edge service providers, can realize that carbon emission is reduced by up to 60%, while maintaining privacy and controlling overhead.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing, and in particular to a battery scheduling method and device for a multi-edge service provider scenario. BACKGROUND

[0002] In the post-5G era, the number of Internet of Things (IoT) terminal devices, such as smart phones and wearable devices, has grown significantly. These devices have placed higher demands on network quality of service, including lower latency and improved resource provisioning conditions, especially for emerging applications such as virtual reality / augmented reality and autonomous driving. In response, edge computing is being adopted more and more. Edge computing shifts task execution from remote data centers (DCs) to numerous small computing nodes located at the edge of the network, thereby providing users with lower latency and more reliable network and computing services. Generally speaking, these edge computing nodes are managed and operated by various edge service providers (ESPs) in the form of edge server (ES) clusters or micro-DCs. Edge computing is expected to handle more than 70% of tasks from cloud computing and terminal devices in the future.

[0003] Compared with cloud computing, edge computing has a significant advantage in utilizing renewable energy (RE) to build low-carbon or even zero-carbon computing systems (such as wind and solar energy). Geographically dispersed ESs can naturally access nearby renewable energy generators (REGs) as their primary power source. Figure 1 A brief schematic diagram is provided: numerous ESs and surrounding REGs together form an energy supply and demand unit, defined as an off-site RE supply model. In addition, widely used energy storage technologies can alleviate the instability and intermittency of RE supply, for example, solar panels cannot generate electricity at night, and wind power production varies significantly with wind speed. The configuration and planning of batteries are crucial. In Figure 1 In the case of ES node ES1, the battery can store energy when there is excess RE power, so that it can be used during shortages. This ensures that RE-powered ESs have a smooth power supply and network service quality. The battery of ES1 performs charge and discharge scheduling to overcome the intermittency and instability of renewable energy. Figure 1 In the case of renewable energy and energy demand data from CAISO and NORDUnet. (T1, T3) and (T2) color blocks represent battery charging and discharging operations, respectively. The solid line and the short dashed line represent the energy demand of ES1 and renewable energy generation, respectively; the long dashed line represents the energy demand of the newly added ES2.

[0004] Nonetheless, research on integrating REGs into edge computing systems is currently still scarce and the potential of REs in edge computing systems has not been fully explored. Specifically, previous research works usually assume a one-to-one correspondence between ESs and REGs, which is essentially equivalent to a centralized RE-supporting data center, defining an on-site RE provisioning model. In contrast, edge systems adopting off-site RE sharing model are technically more challenging but have not been fully explored.

[0005] To illustrate the off-site RE sharing model, we further consider another computing node ES2 with a battery. In this case, ES2 and ES1 jointly consume the nearby REs while they also constitute potential competition as they both want to charge (discharge) their batteries when the REs are sufficient (insufficient), which can lead to another RE shortage (surplus) situation, resulting in low RE utilization efficiency. Therefore, coordinating the battery charging and discharging capabilities among ESs is crucial under the off-site RE provisioning model. However, this is not easy as ESs, REG configurations, and their energy supply and demand are dynamically changing and heterogeneous (Challenge 1). Moreover, the unique business of edge computing and the RE provisioning model exacerbate the above challenges.

[0006] Limited information sharing (Challenge 2): In the real world, edge computing usually involves multiple ESPs, each of which operates its own ES. For business reasons, ESPs do not share the configurations and operating states of their ESs and batteries with each other. This leads to a lack of global scheduling. Each ESP can only make battery charging and discharging decisions based on local observations, leading to difficulties in coordination among ESPs and potential RE competition.

[0007] Online scheduling requirement (Challenge 3): In the off-site RE provisioning scenario, it is very difficult to obtain the specific configurations of nearby REGs and site-specific meteorological data. This limitation prevents ESPs from having the availability of long-term predictions of REs (e.g., days or even hours). Therefore, each ESP must rely on current and historical data and schedule batteries online for the next short-term time slot. This time-slot-by-time-slot scheduling makes long-term planning and optimization difficult. In particular, the increase in the number of batteries leads to an increasing dimension of decisions, which is crucial for online algorithms. SUMMARY

[0008] Based on this, it is necessary to provide a battery scheduling method and device for a multi-edge service provider scenario in view of the above technical problems.

[0009] A battery scheduling method for a multi-edge service provider scenario, the method comprising:

[0010] determining an edge computing region of interest; in the edge computing region, a plurality of edge service providers and a plurality of batteries are contained; each edge service provider has a set of edge servers and a set of batteries; each edge service provider runs as an independent agent;

[0011] defining the definition of the action of the agent and the reward after the agent performs the action; wherein the action is defined according to the total sum of the charging states of all the batteries possessed by the agent, and the reward after the action is performed is defined according to the carbon emission and the battery degradation cost;

[0012] obtaining a tuple sample of the agent; wherein a tuple is composed of the system state of the current time slot and the next time slot, the action of the agent and the reward after the action is performed; the system state of the current and next time slot is obtained by merging the local observation of all the agents in the time slot;

[0013] inputting the local observation of each agent into the corresponding deep recursive Q network to obtain the corresponding first Q value and perturbing it, then aggregating the first perturbed Q values of all the agents to obtain the first joint Q value;

[0014] inputting the local observation after each agent performs the action into the corresponding target Q network to obtain the corresponding second Q value and perturbing it, then aggregating the second perturbed Q values of all the agents to obtain the second joint Q value, and obtaining the target value according to the second joint Q value and the reward after all the agents perform the action; wherein the network structure of the target Q network and the deep recursive Q network is the same;

[0015] constructing a loss function according to the first joint Q value and the target value, and obtaining the trained deep recursive Q network when the loss function converges;

[0016] inputting the actual local observation of the agent into the trained deep recursive Q network to output the Q value of the agent, and obtaining the action and guiding the charging and discharging action of the battery through greedy selection.

[0017] A battery scheduling device for a multi-edge service provider scenario, the device comprising:

[0018] a region determining module for determining an edge computing region of interest; in the edge computing region, a plurality of edge service providers and a plurality of batteries are contained; each edge service provider has a set of edge servers; each edge service provider runs as an independent agent;

[0019] a definition determining module for determining the definition of the action of the agent and the reward after the agent performs the action; wherein the action is defined according to the total sum of the charging states of all the batteries possessed by the agent, and the reward after the action is performed is defined according to the carbon emission and the battery degradation cost;

[0020] a sample obtaining module configured to obtain tuple samples of the agents; wherein a tuple is composed of the system state of the current and next time slots, the action of the agent, and the reward after the action is performed; the system state of the current and next time slots is obtained by merging the local observations of all agents in the time slots;

[0021] a joint Q value calculation module configured to input the local observation of each agent into a corresponding deep recurrent Q network, to obtain a corresponding first Q value and perturb the first Q value, and then aggregate the first perturbed Q values of all agents to obtain a first joint Q value;

[0022] a target value calculation module configured to input the local observation after each agent performs an action into a corresponding target Q network, to obtain a corresponding second Q value and perturb the second Q value, and then aggregate the second perturbed Q values of all agents to obtain a second joint Q value, and obtain a target value according to the second joint Q value and the reward after all agents perform the action; wherein the network structure of the target Q network is the same as that of the deep recurrent Q network;

[0023] a network training module configured to construct a loss function according to the first joint Q value and the target value, and obtain a trained deep recurrent Q network when the loss function converges;

[0024] a battery scheduling module configured to input the actual local observation of the agent into the trained deep recurrent Q network, to output the Q value of the agent, and to obtain an action by greedy selection and guide the charging and discharging action of the battery.

[0025] The above battery scheduling method and device for the multi-edge service provider scenario include: in an edge computing area of interest composed of multiple edge service providers and multiple batteries, an independent agent is established for each edge service provider, and 1) the action performed by the agent is redefined to reduce the dimension of the action space, and 2) the reward after the action is performed is redefined to align the optimization goal, i.e., to reduce carbon emissions and battery degradation cost. Then, a distributed training and distributed execution strategy is adopted, and a Q value perturbation strategy is added to isolate the training and inference of each agent, to achieve information protection. In addition, an action conversion mechanism is designed to convert the defined action into the actual charging and discharging operation of the battery in the agent, to improve the overall training and execution efficiency. In summary, the method has the ability to effectively coordinate multiple edge service providers, can achieve a reduction in carbon emissions of up to 60%, while maintaining privacy and controlling overhead. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 Schematic diagram of off-site renewable energy sharing multi-ESP edge computing scenario;

[0027] Figure 2This is a flowchart illustrating a battery scheduling method for a multi-edge service provider scenario in one embodiment.

[0028] Figure 3 This is a typical diagram illustrating the relationship between the maximum number of battery cycles and the DoD level.

[0029] Figure 4 This is a schematic diagram of a distributed battery scheduling method framework based on Value Decomposition Network (VDN).

[0030] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] In one embodiment, such as Figure 2 As shown, a battery scheduling method for multi-edge service provider scenarios is provided, including the following steps:

[0033] Step 202: Determine the edge computing region of interest.

[0034] The edge computing region contains multiple edge service providers and multiple batteries. Each edge service provider has a set of edge servers. Each edge service provider operates as an independent intelligent agent.

[0035] refer to Figure 1 We begin by providing a formal description of the scenario under study. The edge computing region of interest in this application consists of multiple edge servers and renewable energy generators. Edge servers from different edge service providers are connected to nearby renewable energy generators via the power grid to obtain renewable energy support. Geographically proximate edge servers and renewable energy generators form a supply and demand relationship, constituting a renewable energy unit. The scope of this application is limited to a single renewable energy unit. Each renewable energy unit contains multiple edge service providers, and each set of multiple edge service providers owns a set of edge servers. The set of edge servers in the target renewable energy unit is denoted as S:={S1,S2,…,S…} n ,...,S N} T S n It is the set of edge servers of the nth edge service provider, denoted as

[0036] In real-world systems, with Figure 1A simplified description of each edge server is that it usually owns one battery. Different from this, multiple edge servers of the same edge service provider can share one battery (especially for high-density, low-capacity edge servers). The set of batteries is denoted as B:={B1, B2,..., B n ,...,B N} T The set of batteries of the nth edge service provider is denoted as Bn:={B1, B2,..., B There are K n ≤M n . The binding relationship between edge servers and batteries is represented by a numerical value χ, where χ n,m,k =1 indicates that the battery b n,k serves the edge server s n,m , otherwise χ n,m,k takes the value of zero.

[0037] As a decision entity, the agent learns the scheduling strategy of the distributed battery by interacting with the environment. In our scheme, each edge service provider runs as an independent agent and cooperates with other agents to achieve the carbon emission reduction target.

[0038] Step 204, determine the action of the agent and the definition of the reward after executing the action.

[0039] Specifically, it includes defining the action according to the change of the charging state of all the batteries owned by the agent, and defining the reward after executing the action according to the carbon emission and battery degradation cost.

[0040] For agent n, its tentative action at time slot t is π n,k (t),k∈[1,K n ]. At the beginning of each time slot, each battery decides whether to charge / discharged to cope with renewable energy fluctuations and edge server energy demand. The guiding principle is to maximize battery charging when renewable energy is sufficient and edge server energy demand is low, and to release stored energy as early as possible when energy demand is high, so as to achieve carbon emission reduction and cost transfer in advance. For battery b n,k , its decision in period T is formalized as a series of power values:

[0041] Π n,k ={π n,k (1),π n,k (2),…,π n,k (t),…,π n,k (T)} T ;

[0042] Where π n,k (t) represents the real value of the three situations:

[0043] i) positive value: indicates to operate at power p n,k (t) to charge the battery;

[0044] ii) negative value: indicates to operate at power -p n,k (t) to discharge the battery, energy is delivered to the edge server serving;

[0045] iii) zero value: indicates to do nothing.

[0046] As the number of batteries increases, the action space dimension also increases accordingly, exponentially increasing the learning difficulty of deep reinforcement learning and multi-agent deep reinforcement learning. To solve this problem, the total effect of the charging and discharging power applied by the agent battery is considered, which is defined as the state of charge change of all batteries.

[0047] The application implements a timely reward policy, and the agent obtains feedback from the environment after performing the action, as a reward for effectiveness. In the context of the present application, the reward of each time slot is aligned with the optimization goal (minimizing the total carbon emissions of the edge server provider of the target renewable energy unit and the battery degradation cost), and is consistent among the agents, defined as: where r(t) is the reward of agent n after performing the action at time slot t, is the carbon emissions generated by agent n after performing the action at time slot t, is the degradation cost of the battery owned by agent n after performing the action at time slot t, and β is a hyperparameter used to adjust the trade-off between carbon emission reduction and battery degradation cost. For example, considering the impact of cost, β can be set as the ratio of carbon emission cost to battery cost.

[0048] In the model of the present application, β in the above definition remains unchanged for all edge server providers. However, considering that some edge server providers may prioritize carbon emission reduction or battery degradation differently, the present method allows adjusting the β value by establishing a unique aggregation layer for each edge server provider. This flexibility introduces a practical challenge: a higher β value can reduce the battery degradation cost of some edge server providers, but may also increase overall carbon emissions. This requires the introduction of a normalization mechanism in order to target individual rewards or a fair mechanism to prevent agents from focusing too much on battery cost degradation.

[0049] Step 206, obtaining a tuple sample of the agent.

[0050] where one tuple is composed of the system state of the current and next time slot, the agent action and the reward after performing the action; the system state of the current and next time slot is obtained by merging the local observation of all agents in the time slot, i.e., (s(t), a(t), r(t), s(t+1)). where o n (t) is the local state observation of agent n at the beginning of time slot t.

[0051] As a basic component of off-policy reinforcement learning, the experience replay buffer (RB) stores the tuples of agent historical interactions for training data sampling and enhancing the training process. In order to realize distributed training, the present application provides a separate RB for each agent, and consistent sampling of samples in the RB is performed during the training process. The agents maintain consistent sampling indexes to ensure the stability of the training.

[0052] Step 208, input the local observation of each agent into the corresponding deep recurrent Q network to obtain the corresponding first Q value and perturb it, then aggregate the first perturbed Q values of all agents to obtain the first joint Q value.

[0053] Step 210, input the local observation of each agent after performing the action into the corresponding target Q network to obtain the corresponding second Q value and perturb it, then aggregate the second perturbed Q values of all agents to obtain the second joint Q value, and obtain the target value according to the second joint Q value and the reward after performing the action of all agents.

[0054] where the network structure of the target Q network and the deep recurrent Q network is the same.

[0055] In order to avoid the instability of the agent in the training and learn the policy based on the global perspective, an aggregation layer is created to aggregate the Q values of all agents to represent the set Q value. In the inference process, the global Q value is decomposed into the Q value of each agent.

[0056] In the scenario of the present application, the aggregation layer can be deployed on one or more edge service providers. In order to prevent information leakage, the agents for training and inference are completely independent, and only the perturbed Q value is sent to the hidden layer, which is effective in protecting the information of the agent. The joint Q value is calculated as follows:

[0057]

[0058] where θ represents the deep recurrent Q network parameters of all agents, Q value of the disturbance after invalid work shielding of the action performed by the agent n at time slot t. The invalid shielding value is denoted by α n (t) represents. The disturbed Q value is obtained by adding a Gaussian variance to the original Q value:

[0059]

[0060] In the framework of the present application, by submitting the disturbed Q value in each training round, the agent receives feedback based on the loss value from the aggregation layer, which is used to update the parameters of its neural network. In addition to the aforementioned network (i.e., the deep recurrent Q network as the main network), the agent also defines another network with the same structure during the training process, called the target Q network.

[0061] The target value is obtained according to the second joint Q value and the reward after all agents perform actions, and the target value is the maximum Q value obtained by the following way:

[0062]

[0063] Where y tot is the target value, θ - is the target Q network parameter, α is the learning rate, is the gradient of the loss function of the deep recurrent Q network with respect to θ.

[0064] Step 212, construct a loss function according to the first joint Q value and the target value, and when the loss function converges, obtain the trained deep recurrent Q network and target Q network.

[0065] The loss function can be represented as follows:

[0066]

[0067] Where N b is the batch size of the training data tuple sample.

[0068] Step 214, input the actual local observation of the agent into the trained deep recurrent Q network, output the Q value of the agent, and obtain the action through greedy selection and guide the charging and discharging action of the battery.

[0069] Since the action is converted from the battery charging and discharging to the state change of the charging, the actual executed action needs to be converted from the action generated during training and inference.

[0070] The battery scheduling method for the multi-edge service provider scenario redefines 1) the actions performed by the agent to reduce the dimensionality of the action space, and 2) the reward after performing the action, to align the optimization goal, i.e., to reduce carbon emissions and battery degradation costs. Then, a distributed training and distributed execution strategy is adopted, and a Q-value perturbation strategy is added to isolate the training and inference of each agent, thereby achieving information protection. In addition, an action conversion mechanism is designed to convert the defined actions into actual charging and discharging operations of the battery in the agent, thereby improving the overall training and execution efficiency. In summary, the method has the ability to effectively coordinate multiple edge service providers and can achieve a reduction in carbon emissions of up to 60% while maintaining privacy and controlling overhead.

[0071] It should be understood that, although Figure 1 The steps in the flowchart of the method can be displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 1 At least some of the steps in the method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or sub-steps or stages of other steps.

[0072] In one embodiment, the local observation of an agent can be represented as:

[0073]

[0074] o n (t) is divided into three parts:

[0075] 1) the state of the battery owned by the current time slot.

[0076] The state of the battery is represented by the state of charge, the depth of discharge, and the state of health of the battery.

[0077] Suppose battery b n,k has a capacity c n,k in ampere-hours (Ah). For example, a 10 Ah battery can be continuously discharged at a current of 2 A for 5 hours, providing a total energy storage of 10 Ah x 2 V = 1.1 kWh. During the charging and discharging cycle, the state indicators of the battery change constantly. At time slot t, the state of battery b n,k is represented as a three-tuple:

[0078] S n,k (t):={SoC n,k (t),DoD n,k (t),SoH n,k (t)};

[0079] where SoC n,k (t) is the state of charge (SoC), i.e., the percentage of the remaining capacity over the rated capacity. DoD n,k (t) is the depth of discharge (DoD), which represents the proportion of energy that has been extracted from the battery. SoC and DoD values are complementary, satisfying SoC n,k (t) + DoD n,k (t) = 1. SoH n,k (t) is used to monitor the state of health (SoH) of the battery, whose value fluctuates between 0 and 1. The battery inevitably ages and degrades gradually after experiencing multiple charge-discharge cycles. If SoH n,k falls below a certain critical threshold, e.g., then the battery needs to be replaced to ensure system efficiency and safety.

[0080] 2) the total energy demand of the edge server served by each battery in the current time slot.

[0081] is denoted as is calculated as follows:

[0082]

[0083] p n,m (t) is the power demand of edge server s n,m at time slot t.

[0084] This application divides the continuous time period T into multiple time slots of length Δt (e.g., 15 minutes), denoted as T = {1, 2, …, t, …, T}. Within the time period T, the average power demand distribution of edge server s n,m is denoted as:

[0085] p n,m :={p n,m (1),p n,m (2),...,p n,m (t),…,p n,m (T)} T

[0086] The power of ES is limited by its rated power, i.e., In practical deployment, the power consumption of edge servers can be easily obtained by reading external power meters or accessing bypass operating systems. Since the battery system is introduced, the power of edge servers is not the final value obtained from the grid.

[0087] 3) Energy generation of renewable energy generator in the previous time slot.

[0088] The total energy generation of the renewable energy generator REG in the time period T is denoted as:

[0089] G = {g(l), g(2),..., g(t),..., g(T)}T

[0090] Conventional regular energy (e.g. dominated by fossil fuels) will be used as a supplement when the renewable energy source cannot provide enough energy. However, this will result in additional carbon emissions (or higher price expenditure due to local policies encouraging carbon reduction).

[0091] In one embodiment, the action is defined according to the change of the state of charge of all the batteries owned by the agent, including:

[0092] The change of the state of charge of all the batteries owned by the agent is calculated by the target state of charge value of each battery owned by the agent and the upper and lower limits of the state of charge, so as to obtain the action of the agent.

[0093] The total is expressed as:

[0094]

[0095] wherein SoC n,k (t) represents the target state of charge value of the battery b n,k ; SoC n (t) max and SoC n (t) min are the SoC aggregation upper and lower limits of the edge service provider n, respectively calculated as and Then, a n (t) is used to guide the redistribution of each battery specific charging and discharging action. In addition, the action can be discretized into percentage increments from 0% to 100% to simplify the decision-making process. Such discretization can help battery operation (e.g. less than 1% granularity), reduce battery usage frequency and prolong battery life.

[0096] In one embodiment, the reward after performing the action is defined according to the carbon emission and the battery degradation cost, including:

[0097] 1) Calculate the carbon emission by the total energy demand, the total energy supply of the edge server served by each battery owned by the agent in the current time slot, and the energy generation of the renewable energy generator in the current time slot; the total energy supply is calculated by the action performed by the agent.

[0098] Where renewable energy is not enough to meet the energy demand of numerous edge servers, traditional energy is integrated into the grid to maintain system operation. Edge servers of edge service providers collectively produce carbon emissions. First, focus on the energy demand of a single edge server at time slot t:

[0099] d n,m (t) = p n,m × Δt;

[0100] The total energy demand of the edge server is:

[0101]

[0102] At the same time, the total energy demand or supply of the battery to the edge server is calculated as:

[0103]

[0104] where, is the actual power obtained by the edge server. Due to factors such as AC / DC conversion efficiency and energy leakage, the battery charging and discharging efficiency cannot be ignored, which is represented as and The actual power obtained from the grid to the ES is represented as:

[0105]

[0106] The change in SoC of the battery after charging and discharging is calculated as follows:

[0107]

[0108] The c n,k (t) in the formula is different from the rated capacity c n,k of the battery. It is the actual capacity of the battery within time slot t, and is closely related to SoH n,k (t). This relationship can be represented by the following formula:

[0109] c n,k (t) = c n,k × SoH n,k (t);

[0110] After implementing the decision π n,k (t) of the battery b n,k , the battery state will be updated as follows:

[0111]

[0112] Within T time slots, the total carbon emissions generated within the target carbon emission limit are calculated as:

[0113]

[0114] The carbon emission of a single time slot can also be calculated as follows. Where λ is the carbon intensity parameter, which quantifies the carbon emission per unit of grid energy production, in g / KWh.

[0115] 2) Calculate the battery degradation cost by the change of the state of health of each battery owned by the agent in the current time slot.

[0116] Due to the cumulative effects of chemical and physical processes, such as the degradation of active materials and changes in the structure of electrode materials, the ability of a battery to withstand consecutive charge and discharge cycles gradually weakens. The main factor affecting this degradation is the cumulative depth of discharge (DoD) experienced by the battery. Figure 3 The relationship between the typical maximum number of cycles of a battery and the DoD level is shown. Where each point (x, y) represents that the battery can undergo up to y full discharge cycles at a DoD of x before reaching a threshold of SoH n,k (t) for battery b n,k The decrease in the number of cycles of battery b n,k is quantified by the change in SoH n,k (t) and is represented as:

[0117]

[0118] Assuming the wear cost of battery b n,k is ω n,k , the battery degradation cost of all edge service provider batteries within T time slots is:

[0119]

[0120] The battery degradation cost of a single time slot can also be calculated as follows.

[0121] Finally, we can establish an optimization model to minimize the total carbon emission and battery cost of the edge service providers of the target renewable energy unit:

[0122]

[0123] By solving the problem of this optimization model, the optimal battery charging and discharging scheduling sequence can be obtained. However, during the problem solving process, we will face the following challenges:

[0124] High intrinsic complexity: Nonlinear optimization problem, very difficult to solve directly. For a single battery scheduling, the size of the solution space is where N p is the size of the battery's candidate actions (i.e., charging and discharging power). In distributed scenarios involving multiple batteries, the solution space grows exponentially to It can be imagined that the cost of solving the problem is huge, and when facing long time and multiple battery scheduling, it will be unacceptable.

[0125] The need for online decision-making: The energy demand of edge servers and renewable energy generation has significant volatility and difficulty in long-term prediction. Especially for off-site scenarios, it is difficult to obtain accurate on-site fine weather data, and it is impractical to predict renewable energy. This means that model-based online algorithms will be useless because there is no input data. At this point, online algorithms are necessary. In addition, it can be observed that the battery state at time slot t+1 (and thereafter) is largely constrained by the scheduling results at time slot t. Online algorithms must have a long-term perspective and be able to make strategic plans based on existing knowledge.

[0126] Information barriers between ESPs: Due to commercial operation reasons, information related to battery and edge server configuration series and real-time energy demand of edge servers may be prohibited between edge service providers to share scheduling information because the scheduling information is highly sensitive. Edge service providers rely only on the information they own to schedule batteries, inevitably leading to suboptimal results. This lack of coordination can lead to all edge service providers charging when renewable energy is abundant and discharging when it is scarce, a phenomenon known as renewable energy competition.

[0127] Classical online algorithms are used to solve timing decision problems, including Lyapunov and DRL algorithms. However, Lyapunov algorithms are difficult to deal with distributed scenarios. DRL models the problem as a Markov process and empirically derives decisions from historical data. However, DRL has poor scalability and is difficult to deal with larger decision-making problems. More critically, both Lyapunov and DRL algorithms require all data to be input into the decision-making process, which contradicts the principle of not sharing information between edge service providers, so further exploration is needed.

[0128] In one embodiment, the network structure of the deep recursive Q network comprises in sequence: a gated recurrent unit, an invalid action mask layer, a perturbation layer, and an aggregation layer;

[0129] First interaction:

[0130] The local observation value of each agent in the time slot is input into the deep recursive Q network to obtain a first initial Q value; the first initial Q value is invalid shielded to obtain a first Q value, and a set of actions corresponding to the first Q value is determined by greedy selection; the set of actions includes an agent action performed by each agent; each agent performs the action to obtain a reward set, and a tuple is constructed according to the system state corresponding to the local observation value, the set of actions and the reward set, and the tuple is stored in a buffer; the tuples in the buffer are marked by serial numbers;

[0131] The second interaction:

[0132] Each agent randomly reads a preset number of tuples from its buffer, and inputs the preset number of tuples into the deep recursive Q network and the target Q network to obtain a first joint Q value and a second joint Q value.

[0133] As shown in Figure 4 , a distributed battery scheduling method framework based on a value decomposition network (VDN) of the application is provided. As shown in Figure 4 , each agent n is trained using a deep recursive Q network (DRQN) equipped with a weight parameter θ n for evaluating the Q value of future actions. The DRQN uses a gated recurrent unit (GRU) to memorize historical information. In each time slot t, in addition to the local observation o n (t), the DRQN also inputs the hidden state h n (t-1) generated by the Q network, thereby implicitly capturing the observation information of the historical time slots.

[0134] As shown in Table 1, the training and inference process of the distributed battery scheduling algorithm based on the value decomposition network (VDN) of the application is provided.

[0135] Table 1 Training and inference process of the distributed battery scheduling algorithm based on VDN

[0136]

[0137]

[0138] In one embodiment, the invalid action shielding parameter in the invalid action shielding layer is calculated as follows:

[0139] The upper limit of the charging state of the agent is calculated, and the first action boundary is calculated according to the upper limit of the charging state;

[0140] The lower limit of the charging state of the agent is calculated, and the second action boundary is calculated according to the lower limit of the charging state;

[0141] The action boundary is obtained from the first action boundary and the second action boundary.

[0142] The action mask parameter is negative infinity when the action performed by the agent exceeds the action bound, otherwise it is zero.

[0143] To improve efficiency in both training and inference, we introduce an invalid action mask (IAM) mechanism to filter actions to satisfy the constraints 1) and 2):

[0144] 1) Battery capacity constraint:

[0145] The state of charge SoC of the battery at any given time period is bounded by its upper and lower bounds, denoted as and These constraints are specified by the edge service provider to avoid overcharging / discharging the battery and are crucial for the battery safety measures. Therefore, there is:

[0146]

[0147] In general,

[0148] 2) Battery charging / discharging power constraint:

[0149] The maximum charging and discharging power of the battery b n,k are denoted as and They differ significantly. For example, in the case of lithium batteries, can be several times higher than . The battery configuration is usually consistent with the energy efficiency consumption level of its service, for example:

[0150]

[0151] where ∈ c and ∈ dc are parameters quantifying the battery and edge server configuration, determined by the edge service provider during deployment. Reference can be made to common battery energy storage system (BESS) configurations, where ∈ c = 5 and ∈ dc = 1. As a result, we have:

[0152] In addition, we must ensure that the amount of electricity discharged from the battery can be completely consumed by the relevant server:

[0153]

[0154] Specifically, for agent n, there is an upper limit on the charging power of each battery. Accordingly, there is an upper limit on the state of charge during its charging process, denoted as: ;

[0155] We thus have:

[0156] a n (t) max = min{SoC n (t) max , 100%};

[0157] Similarly, since the battery cannot be over-discharged, the lower bound of the state of charge of the battery is determined by the state of charge of the power supply provided by the edge server and its load. We take the minimum value to be:

[0158]

[0159] We have:

[0160]

[0161] a n (t) min = max{SoC n (t) min , 0};

[0162] The IAM parameter of the agent n is then represented as: We have:

[0163]

[0164] In one embodiment, the output agent’s action and guide the charging and discharging actions of the battery, including:

[0165] Given a n (t), the action actually executed is found by solving the following optimization problem:

[0166]

[0167] Due to the nonlinearity of Γ(·), it is infeasible to directly solve the above optimization problem. Our solution is to decompose Γ(·) into multiple small linear functions, e.g., with an interval of 0.001%. This granularity is practical considering the limited precision of the battery DoD. We then use the interior point method to solve this linearized optimization problem.

[0168] In summary, (i) in terms of architecture, the present application turns to the state-of-the-art multi-agent deep reinforcement learning by establishing an independent agent for each edge service provider, which helps to increase the scalability of the algorithm and avoid bloated neural networks; (ii) in terms of deployment, the DTDE strategy is adopted, and the Q-value perturbation and index exchange strategies are added, which helps to isolate the training and inference of each agent and achieve information protection; (iii) in terms of decision-making, the action conversion mechanism is designed, and the decision of the agent is used as a guide for actual execution, thereby improving the overall training and execution efficiency.

[0169] In one embodiment, a battery scheduling device for a multi-edge service provider scenario is provided, comprising: a region determination module, a definition determination module, a sample acquisition module, a joint Q-value calculation module, a target value calculation module, a network training module, and a battery scheduling module, wherein:

[0170] The region determination module is configured to determine an edge computing region of interest; in the edge computing region, a plurality of edge service providers and a plurality of batteries are contained; each edge service provider has a group of edge servers; and each edge service provider operates as an independent agent.

[0171] The definition determination module is configured to determine the definition of the action of the agent and the reward after the action is performed; specifically, the action is defined according to the change of the charging state of all the batteries owned by the agent; and the reward after the action is performed is defined according to the carbon emission and the battery degradation cost.

[0172] The sample acquisition module is configured to acquire a tuple sample of the agent; wherein a tuple is composed of the system state of the current time slot, the action of the agent, and the reward and the system state after the action is performed; the system state of the current time slot is obtained by merging the local observations of all agents in the current time slot.

[0173] The joint Q-value calculation module is configured to input the local observation of each agent into the corresponding deep recursive Q network to obtain the corresponding first Q value and perturb it, and then aggregate the first perturbed Q values of all agents to obtain the first joint Q value.

[0174] The target value calculation module inputs the local observation after the action of each agent is performed into the corresponding target Q network to obtain the corresponding second Q value and perturb it, and then aggregates the second perturbed Q values of all agents to obtain the second joint Q value, and obtains the target value according to the second joint Q value and the reward after the action of all agents is performed; wherein the network structure of the target Q network and the deep recursive Q network is the same.

[0175] a network training module, configured to construct a loss function according to the first joint Q value and the target value, and obtain a trained deep recurrent Q network and a target Q network when the loss function converges;

[0176] a battery scheduling module, configured to input an actual local observation of the agent into the trained deep recurrent Q network, and output an action of the agent and guide a charging and discharging action of the battery.

[0177] The specific definition of the battery scheduling apparatus for the multi-edge service provider scenario can be referred to the definition of the battery scheduling method for the multi-edge service provider scenario, which will not be repeated here. Each module in the battery scheduling apparatus for the multi-edge service provider scenario can be realized by software, hardware and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0178] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data such as meta tuple samples. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the battery scheduling method for the multi-edge service provider scenario.

[0179] Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0180] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in the above embodiments.

[0181] In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the steps of the method of the above embodiments.

[0182] A person of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above embodiments. Any reference to memory, storage, database or other medium in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0183] Any combination of the technical features of the above embodiments can be made, and in order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0184] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A battery scheduling method for a multi-edge service provider scenario, characterized in that, The method comprises: determining an edge computing region of interest; in the edge computing region, containing a plurality of edge service providers and a plurality of batteries; each edge service provider has a set of edge servers and a set of batteries; each edge service provider runs as an independent agent; determining the definition of the action of the agent and the reward after the action is executed; wherein the action is defined according to the total state of charge of all batteries owned by the agent, and the reward after the action is executed is defined according to carbon emissions and battery degradation cost; obtaining a tuple sample of the agent; wherein a tuple is composed of the system state of the current time slot and the next time slot, the action of the agent and the reward after the action is executed; the system state of the current and next time slot is obtained by merging the local observation of all agents in the time slot; input the local observation of each agent into the corresponding deep recursive Q network to obtain the corresponding first Q value and perturb it, then aggregate the first perturbed Q value of all agents to obtain the first joint Q value; input the local observation of each agent after the action is executed into the corresponding target Q network to obtain the corresponding second Q value and perturb it, then aggregate the second perturbed Q value of all agents to obtain the second joint Q value, and obtain the target value according to the second joint Q value and the reward after the action is executed by all agents; wherein the network structure of the target Q network and the deep recursive Q network is the same; construct a loss function according to the first joint Q value and the target value, and obtain the trained deep recursive Q network when the loss function converges; input the actual local observation of the agent into the trained deep recursive Q network to output the Q value of the agent, and obtain the action and guide the charging and discharging action of the battery through greedy selection; The local observation of an agent includes: 1) the state of the battery owned by the current time slot; the state of the battery is represented by the state of charge, the depth of discharge and the state of health of the battery; 2) the total energy demand of the edge server owned; 3) the energy generation of the renewable energy generator in the previous time slot.

2. The method of claim 1, wherein, The action is defined according to the total state of charge of all batteries owned by the agent, including: by adding the total capacity of all batteries owned by the agent, and multiplying the target state of charge of all batteries by a percentage obtained by probabilistic greedy selection from the Q value of the agent, construct an optimization problem that minimizes the loss of each battery health, and solve the optimization problem to obtain the optimal charging and discharging power of each battery.

3. The method of claim 1, wherein, The reward after the action is executed is defined according to the carbon emissions and the battery degradation cost, including: calculate the carbon emissions by the total energy demand of the edge server owned by all agents, the total charging and discharging amount of the battery owned, and the energy generation of the renewable energy generator in the current time slot; calculate the battery degradation cost by the change of the state of health of each battery owned by the agent in the current time slot.

4. The method of claim 1, wherein, The local observation of each agent is input into the corresponding deep recurrent Q network to obtain a corresponding first Q value and perturb the first Q value, and then the first perturbed Q values of all agents are aggregated to obtain a first joint Q value; the local observation of each agent after performing an action is input into the corresponding target Q network to obtain a corresponding second Q value and perturb the second Q value, and then the second perturbed Q values of all agents are aggregated to obtain a second joint Q value, comprising: The first interaction: The local observation value of each agent in the time slot is input into the deep recurrent Q network to obtain a first initial Q value; The first initial Q value is invalid shielded to obtain a first Q value, and a set of actions corresponding to the first Q value is determined by greedy selection; the set of actions includes an agent action performed by each agent; Each agent performs the action to obtain a reward set, and a tuple is constructed according to the system state corresponding to the local observation value, the action set and the reward set, and the tuple is stored in the buffer; the tuples in the buffer are marked by serial numbers; The second interaction: Each agent randomly reads a preset number of tuples from its buffer, and inputs the preset number of tuples into the deep recurrent Q network and the target Q network to obtain a first joint Q value and a second joint Q value.

5. The method of claim 4, wherein, The calculation method of the invalid action shielding parameter in the invalid action shielding layer is as follows: Calculate the upper limit of the state of charge of all batteries owned by the agent, and calculate the first action boundary according to the upper limit of the state of charge; Calculate the lower limit of the state of charge of the agent, and calculate the second action boundary according to the lower limit of the state of charge; The action boundary is obtained from the first action boundary and the second action boundary; When the action performed by the agent exceeds the action boundary, the action shielding parameter is negative infinity, otherwise it is zero.

6. A battery scheduling device for multi-edge service provider scenarios, characterized in that, The device comprises: A region determination module for determining an edge computing region of interest; in the edge computing region, a plurality of edge service providers and a plurality of batteries are included; each edge service provider owns a group of edge servers; each edge service provider operates as an independent agent; A definition determination module for determining the definition of the action of the agent and the reward after performing the action; wherein the action is defined according to the total state of charge of all batteries owned by the agent, and the reward after performing the action is defined according to the carbon emission and the battery degradation cost; A sample acquisition module for acquiring a tuple sample of the agent; wherein a tuple is composed of the system state of the current and next time slots, the agent action and the reward after performing the action; the system state of the current and next time slots is obtained by merging the local observations of all agents in the time slot; wherein the local observation of an agent includes: 1) the state of the battery owned by the current time slot; the state of the battery is represented by the state of charge, the depth of discharge and the battery health state; 2) the total energy demand of the edge server owned; 3) the energy generation of the renewable energy generator in the previous time slot; The joint Q value calculation module is configured to input the local observation of each agent into a corresponding deep recurrent Q network, obtain a corresponding first Q value, perturb the first Q value, aggregate the first perturbed Q values of all agents, and obtain a first joint Q value; The target value calculation module is configured to input the local observation of each agent after performing an action into a corresponding target Q network, obtain a corresponding second Q value, perturb the second Q value, aggregate the second perturbed Q values of all agents, obtain a second joint Q value, and obtain a target value according to the second joint Q value and the reward after performing the action by all agents; wherein the network structure of the target Q network is the same as that of the deep recurrent Q network; The network training module is configured to construct a loss function according to the first joint Q value and the target value, and obtain a trained deep recurrent Q network when the loss function converges. The battery scheduling module is configured to input the actual local observation of the agent into the trained deep recurrent Q network, output an agent Q value, and obtain an action through greedy selection and guide the charging and discharging action of the battery. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images