Battery scheduling method and device for multi-edge service provider scene
Through distributed deep reinforcement learning methods, the battery charging and discharging is coordinated in multi-edge service provider scenarios, and the problems of low renewable energy utilization efficiency and high carbon emissions are solved, and carbon emission reduction and battery scheduling efficiency are achieved.
Patent Information
- Application Number
- CN202411088970.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-09
AI Technical Summary
In the multi-edge service provider scenario, it is difficult for the existing technology to effectively coordinate battery charging and discharging between multiple edge service providers, resulting in low renewable energy utilization efficiency, high carbon emissions, and lack of global scheduling and information sharing, which is difficult to achieve Long-term battery scheduling optimization.
Adopting distributed deep reinforcement learning methods, by establishing independent agents at each edge service provider, defining action and reward mechanisms, using deep recursive Q networks and target Q networks for training and reasoning, realizing action coordination and battery scheduling, and reducing carbon Emissions and battery degradation costs.
Achieve a 60% reduction in carbon emissions, maintain privacy and control overhead, improve the overall efficiency and scalability of battery scheduling, and effectively coordinate battery charging and discharging of multiple edge service providers.
Smart Images

Figure CN118964025A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of edge computing technology, and in particular to a battery scheduling method and device for a multi-edge service provider scenario. Background Art
[0002] In the post-5G era, the number of IoT terminal devices (such as smartphones and wearable devices) has grown significantly. These devices have put forward higher requirements for network service quality, including reducing latency and improving resource supply conditions, especially for emerging applications such as virtual reality / augmented reality and autonomous driving. In response, edge computing is increasingly being adopted. Edge computing transfers task execution from remote data centers (DCs) to numerous small computing nodes located at the edge of the network, thereby providing users with lower latency and more reliable network and computing services. Generally speaking, these edge computing nodes are managed and operated by various edge service providers (ESPs) and organized in the form of edge server (ES) clusters or micro data centers (micro-DCs). The edge computing model is expected to handle more than 70% of tasks from cloud computing and terminal devices in the future.
[0003] Compared to cloud computing, edge computing has significant advantages in building low-carbon or even zero-carbon computing systems using renewable energy (RE) (e.g. wind and solar). Geographically dispersed ES can naturally access nearby renewable energy generators (REGs) as their primary power source. Figure 1 A simple schematic diagram is provided: numerous ES and surrounding REGs together constitute an energy supply and demand unit, which is defined as the off-site RE supply model. In addition, widely used energy storage technology can alleviate the instability and intermittency of RE supply. For example, solar panels cannot generate electricity at night, and wind energy production varies significantly with wind speed. The configuration and planning of batteries are crucial. Figure 1 In the example, when ES node ES1 has excess RE power, its battery can store energy for use during shortages. This ensures that the ES powered by REs have a smooth power supply and network service quality. ES1's battery performs charge and discharge scheduling to overcome the intermittency and instability of renewable energy. Figure 1 Renewable energy and energy demand data are from CAISO and NORDUnet. The (T1, T3) and (T2) color blocks represent battery charging and discharging operations, respectively. The solid and short dashed lines represent the energy demand of ES1 and renewable energy generation, respectively; the long dashed line represents the energy demand of the newly added ES2.
[0004] Nevertheless, research on integrating REG into edge computing systems is still scarce and the potential of RE in edge computing systems has not been fully explored. Specifically, previous research works usually assume a one-to-one correspondence between ES and REG, which is essentially equivalent to a centralized RE-supported data center, which is defined as an on-site RE provisioning model. In contrast, edge systems adopting an off-site RE sharing model are technically more challenging and have not been fully explored.
[0005] To illustrate the off-site RE sharing model, we further consider another computing node ES2 with a battery. In this case, ES2 and ES1 jointly consume nearby REs, and they also constitute potential competition because they both want to charge (discharge) their batteries when REs are sufficient (insufficient), and this competition may lead to another RE shortage (surplus) situation, resulting in inefficient RE utilization. Therefore, under the off-site RE supply model, it is crucial to coordinate the battery charging and discharging capabilities between ESs. However, this is not easy because ES, REG configurations, and their energy supply and demand are dynamically changing and heterogeneous (Challenge 1). In addition, the unique business and RE supply models of edge computing exacerbate the above challenges.
[0006] Limited information sharing (Challenge 2): In the real world, edge computing usually involves multiple ESPs, each operating its own ES. For commercial considerations, ESPs do not share the configuration and operating status of their ES and batteries with each other. This leads to a lack of global scheduling. Each ESP can only make battery charging and discharging decisions based on local observations, resulting in coordination difficulties between ESPs and potential RE competition.
[0007] Online scheduling requirements (Challenge 3): In the off-site RE supply scenario, it is difficult to obtain the specific configuration of nearby REGs and site-specific meteorological data. This limitation prevents the availability of long-term forecasts for RE by ESPs (e.g., days or even hours). Therefore, each ESP must rely on current and historical data and schedule batteries online for the next short-term time slot. This time slot-by-time slot scheduling makes long-term planning and optimization difficult. In particular, the increase in the number of batteries leads to an increasing number of decision dimensions, which is critical for online algorithms. Summary of the invention
[0008] Based on this, it is necessary to provide a battery scheduling method and device for multi-edge service provider scenarios to address the above technical issues.
[0009] A battery scheduling method for a multi-edge service provider scenario, the method comprising:
[0010] Determine an edge computing area of interest; include multiple edge service providers and multiple batteries in the edge computing area; each edge service provider has a group of edge servers and a group of batteries; each edge service provider operates as an independent intelligent entity;
[0011] Determine the definition of the agent's actions and the rewards after executing the actions; wherein the actions are defined according to the sum of the charge states of all batteries owned by the agent, and the rewards after executing the actions are defined according to the carbon emissions and the battery degradation cost;
[0012] Get a tuple sample of agents; a tuple consists of the system state of the current time slot and the next time slot, the agent action, and the reward after the action is performed; the system state of the current and next time slots is obtained by merging the local observations of all agents in the time slot;
[0013] Input the local observation of each agent into the corresponding deep recursive Q network, obtain the corresponding first Q value and perturb it, then aggregate the first perturbed Q values of all agents to obtain the first joint Q value;
[0014] The local observation after each agent performs an action is input into the corresponding target Q network to obtain the corresponding second Q value and perturb it, then the second perturbation Q values of all agents are aggregated to obtain the second joint Q value, and the target value is obtained according to the second joint Q value and the rewards after all agents perform actions; wherein, the network structure of the target Q network and the deep recursive Q network is the same;
[0015] A loss function is constructed according to the first joint Q value and the target value. When the loss function converges, a trained deep recursive Q network is obtained.
[0016] The actual local observation of the agent is input into the trained deep recursive Q network, the agent Q value is output, and the action is obtained through greedy selection to guide the charging and discharging actions of the battery.
[0017] A battery scheduling device for a multi-edge service provider scenario, the device comprising:
[0018] A region determination module is used to determine an edge computing region of interest; within the edge computing region, multiple edge service providers and multiple batteries are included; each edge service provider has a group of edge servers; each edge service provider operates as an independent intelligent entity;
[0019] A definition determination module is used to determine the definition of the action of the intelligent agent and the reward after the action is executed; wherein the action is defined according to the sum of the charging states of all batteries owned by the intelligent agent, and the reward after the action is executed is defined according to the carbon emission and the battery degradation cost;
[0020] The sample acquisition module is used to obtain tuple samples of the agent; a tuple consists of the system state of the current and next time slots, the agent action, and the reward after the action is performed; the system state of the current and next time slots is obtained by merging the local observations of all agents in the time slot;
[0021] A joint Q-value calculation module is used to input the local observation of each agent into the corresponding deep recursive Q network, obtain the corresponding first Q-value and perturb it, and then aggregate the first perturbed Q-values of all agents to obtain the first joint Q-value;
[0022] The target value calculation module inputs the local observations after each agent performs an action into the corresponding target Q network, obtains the corresponding second Q value and perturbs it, then aggregates the second perturbed Q values of all agents to obtain the second joint Q value, and obtains the target value according to the second joint Q value and the rewards after all agents perform actions; wherein, the network structure of the target Q network and the deep recursive Q network is the same;
[0023] A network training module is used to construct a loss function according to the first joint Q value and the target value, and when the loss function converges, a trained deep recursive Q network is obtained;
[0024] The battery scheduling module is used to input the actual local observations of the agent into the trained deep recursive Q network, output the agent Q value, and obtain actions through greedy selection to guide the charging and discharging actions of the battery.
[0025] The above-mentioned battery scheduling method and device for a multi-edge service provider scenario includes: in an edge computing area of interest consisting of multiple edge service providers and multiple batteries, an independent agent is established for each edge service provider, and: 1) the actions performed by the agent to reduce the dimension of the action space, and 2) the rewards after the actions are performed to align the optimization goals, that is, to reduce carbon emissions and battery degradation costs. Then, a distributed training and distributed execution strategy is adopted, and a Q-value perturbation strategy is added to isolate the training and reasoning of each agent to achieve information protection. In addition, an action conversion mechanism is designed to convert the defined actions into actual battery charging and discharging operations in the agent, thereby improving the overall training and execution efficiency. In summary, this method has the ability to effectively coordinate multiple edge service providers and can achieve a carbon emission reduction of up to 60% while maintaining privacy and controlling overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 Schematic diagram of the edge computing scenario of off-site renewable energy sharing with multiple ESPs;
[0027] Figure 2A flowchart of a battery scheduling method for a multi-edge service provider scenario in an embodiment;
[0028] Figure 3 Schematic diagram of the relationship between the maximum number of battery cycles and DoD level;
[0029] Figure 4 The schematic diagram of the framework of the distributed battery scheduling method based on the value decomposition network (VDN);
[0030] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0032] In one embodiment, Figure 2 As shown, a battery scheduling method for a multi-edge service provider scenario is provided, including the following steps:
[0033] Step 202: determine an edge computing area of interest.
[0034] The edge computing area contains multiple edge service providers and multiple batteries. Each edge service provider has a group of edge servers. Each edge service provider operates as an independent intelligent entity.
[0035] refer to Figure 1 , we first provide a formal description of the studied scenario. The edge computing area of interest in this application consists of multiple edge servers and renewable energy generators. Edge servers from different edge service providers are connected to nearby renewable energy generators through the power grid to obtain renewable energy support. Geographically close edge servers and renewable energy generators form a supply and demand relationship and constitute a renewable energy unit. The research scope of this application is limited to a single renewable energy unit. Each renewable energy unit contains multiple edge service providers, and each of the multiple edge service providers has a set of edge servers. The set of edge servers in the target renewable energy unit is represented as S:={S1,S2,…,S n ,...,S N} T , where S n is the set of edge servers of the nth edge service provider, denoted as
[0036] In real systems, Figure 1Different from the simplified description that each edge server usually has one battery, multiple edge servers of the same edge service provider may share one battery (especially for high-density, low-capacity edge servers). This application represents the battery set as B:={B1,B2,...,B n ,...,B N} T The battery set of the nth edge service provider is represented as Yes K n ≤M n The binding relationship between the edge server and the battery is represented by a value χ, where χ n,m,k =1 means battery b n,k Serving edge servers n,m , otherwise χ n,m,k Takes zero value.
[0037] As a decision-making entity, the agent learns the scheduling strategy of distributed batteries by interacting with the environment. In our scheme, each edge service provider operates as an independent agent and cooperates with other agents to achieve carbon reduction targets.
[0038] Step 204, determine the definition of the agent's action and the reward after executing the action.
[0039] Specifically, it includes defining actions based on the changes in the charge status of all batteries owned by the agent, and defining the rewards after executing actions based on carbon emissions and battery degradation costs.
[0040] For agent n, its tentative action at time slot t is π n,k (t),k∈[1,K n ]. At the beginning of each time slot, each battery decides whether to charge / discharge to cope with the fluctuation of renewable energy and edge server energy demand. The guiding principle is to maximize battery charging when renewable energy is abundant and edge server energy demand is low, and conversely release the stored energy as early as possible when energy demand is high, thereby achieving carbon emission reduction and cost shifting in advance. For battery b n,k , whose decision within period T is formalized as a series of power values:
[0041] Π n,k ={π n,k (1),π n,k (2),…,π n,k (t),…,π n,k (T)} T ;
[0042] Among them, π n,k (t) represents the real value of three cases:
[0043] i) Positive value: indicates power π n,k (t) charging the battery;
[0044] ii) Negative value: indicates power -π n,k (t) Discharge the battery and transfer the energy to the edge server of the service;
[0045] iii) Zero value: indicates no operation.
[0046] As the number of batteries increases, the dimension of the action space also increases accordingly, exponentially increasing the learning difficulty of deep reinforcement learning and multi-agent deep reinforcement learning. To address this issue, the total impact of the charge and discharge power exerted by the agent's batteries is considered, defined as the change in the charge state of all batteries.
[0047] This application implements a timely reward strategy, where the agent gets feedback from the environment after performing the action as a reward for effectiveness. In the scenario of this application, the reward for each time slot is aligned with the optimization goal (minimizing the total carbon emissions of the edge service provider of the target renewable energy unit and the battery degradation cost) and is consistent across agents, defined as: Among them, r(t) is the reward after agent n performs the action in time slot t, is the carbon emissions generated by agent n after performing actions in time slot t, is the degradation cost of the battery owned by agent n after it performs an action at time slot t, and β is a hyperparameter used to adjust the trade-off between carbon emission reduction and battery degradation cost. For example, considering the impact of cost, β can be set to the ratio of carbon emission cost to battery cost.
[0048] In the model of this application, β in the above definition remains constant for all edge service providers. However, considering that some edge service providers may prioritize carbon reduction or battery degradation differently, this method allows the β value to be adjusted by establishing a unique aggregation layer for each edge service provider. This flexibility introduces a practical challenge: a higher β value can reduce the battery degradation cost of some edge service providers, but may also increase overall carbon emissions. This requires the introduction of a normalization mechanism to target individual rewards or a fairness mechanism to prevent the agent from focusing too much on battery cost degradation.
[0049] Step 206, obtaining a tuple sample of the agent.
[0050] A tuple consists of the system state of the current time slot and the next time slot, the agent action, and the reward after the action is performed; the system state of the current and next time slots is obtained by merging the local observations of all agents in the time slot, that is, (s(t), a(t), r(t), s(t+1)). The state is a comprehensive description of the environment. In the multi-agent deep reinforcement learning model, each agent has its partial observation of the global system state. The system state of the current time slot is obtained by merging the local observations of all agents in the current time slot, that is, Among them, n (t) is the local state observation of agent n at the beginning of time slot t.
[0051] As a basic component of off-policy reinforcement learning, the experience replay buffer (RB) stores tuples of historical interactions of agents for training data sampling and enhancing the training process. In order to achieve distributed training, this application equips each agent with a separate RB and samples the samples therein consistently during the training process. Agents maintain a consistent sampling index to ensure the stability of training.
[0052] Step 208, input the local observation of each agent into the corresponding deep recursive Q network, obtain the corresponding first Q value and perturb it, and then aggregate the first perturbed Q values of all agents to obtain the first joint Q value.
[0053] In step 210, the local observations after each agent performs an action are input into the corresponding target Q network to obtain the corresponding second Q value and perturb it, and then the second perturbed Q values of all agents are aggregated to obtain the second joint Q value, and the target value is obtained according to the second joint Q value and the rewards after all agents perform the action.
[0054] Among them, the network structures of the target Q network and the deep recursive Q network are the same.
[0055] In order to avoid instability of the agent during training and learn a strategy based on a global perspective, an aggregation layer is created to aggregate the Q-values of all agents to represent the collective Q-value. During inference, the global Q-value is decomposed into the Q-value of each agent.
[0056] In the scenario of this application, the aggregation layer can be deployed on one or more edge service providers. In order to prevent information leakage, the training and reasoning agents are completely independent, and only the perturbed Q value is sent to the hidden layer, which is effective in protecting the agent information. The joint Q value is calculated as follows:
[0057]
[0058] where θ represents the deep recurrent Q-network parameters of all agents, represents the perturbation Q value after the invalid work masking of the action performed by agent n in time slot t. The invalid masking value is denoted by α n (t) represents. The perturbed Q value is obtained by adding Gaussian variance to the original Q value:
[0059]
[0060] In the framework of this application, by submitting the perturbed Q value in each training round, the agent receives feedback based on the loss value from the aggregation layer, which is used to update the parameters of its neural network. In addition to the aforementioned network (i.e., the deep recursive Q network as the main network), the agent also defines another network with the same structure during the training process, called the target Q network.
[0061] The target value is obtained based on the second joint Q value and the rewards after all agents perform actions. The target value is the maximum Q value obtained by the following method:
[0062]
[0063] Among them, y tot is the target value, θ - is the target Q network parameter, α is the learning rate, is the gradient of the loss function of the deep recurrent Q network with respect to θ.
[0064] Step 212, constructing a loss function according to the first joint Q value and the target value, and when the loss function converges, obtaining a trained deep recursive Q network and a target Q network.
[0065] The loss function can be expressed as follows:
[0066]
[0067] Among them, N b is the batch size of training data tuple samples.
[0068] Step 214, input the actual local observation of the agent into the trained deep recursive Q network, output the agent Q value, and obtain the action through greedy selection to guide the charging and discharging action of the battery.
[0069] Since the actions are converted from battery charging and discharging to charge state changes, the actions actually performed need to be converted from the actions generated during training and inference.
[0070] The above-mentioned battery scheduling method for multi-edge service provider scenarios establishes an independent agent for each edge service provider in the edge computing area of interest consisting of multiple edge service providers and multiple batteries, and redefines: 1) the actions performed by the agent to reduce the dimension of the action space, and 2) the rewards after performing the actions to align the optimization goals, namely reducing carbon emissions and battery degradation costs. Then, a distributed training and distributed execution strategy is adopted, and a Q-value perturbation strategy is added to isolate the training and reasoning of each agent to achieve information protection. In addition, an action conversion mechanism is designed to convert the defined actions into actual battery charging and discharging operations in the agent, thereby improving the overall training and execution efficiency. In summary, this method has the ability to effectively coordinate multiple edge service providers and can achieve up to 60% reduction in carbon emissions while maintaining privacy and controlling overhead.
[0071] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0072] In one embodiment, a local observation of an agent can be expressed as:
[0073]
[0074] o n (t) is divided into three parts:
[0075] 1) The status of the battery in the current time slot.
[0076] The status of the battery is indicated by the state of charge, depth of discharge, and battery health.
[0077] Assume that battery b n,k Has a capacity in ampere hours (Ah) c n,k For example, a 10Ah battery can be discharged continuously at 2A for 5 hours, providing a total energy storage of 10Ah×2V=1.1kWh. During the charge and discharge cycle, the battery status indicators change continuously. At time slot t, battery b n,k The state is represented as a triple:
[0078] S n,k (t): = {SoC n,k (t),DoD n,k (t),SoH n,k (t)};
[0079] Among them, SoC n,k (t) is the state of charge (SoC), which is the percentage of remaining capacity to rated capacity. DoD n,k (t) is the depth of discharge (DoD), which represents the proportion of energy that has been extracted from the battery. SoC and DoD values are complementary, and satisfying SoC n,k (t)+DoD n,k (t) = 1. SoH n,k (t) is used to monitor the battery's state of health (SoH), and its value fluctuates between 0 and 1. Batteries inevitably age and gradually degrade after undergoing multiple charge and discharge cycles. If SoH n,k falls below a critical threshold, e.g. The battery needs to be replaced to ensure system efficiency and safety.
[0080] 2) The total energy demand of the edge servers served by each owned battery in the current time slot.
[0081] Expressed as The calculation is as follows:
[0082]
[0083] p n,m (t) is the edge server s n,m Power demand at time slot t.
[0084] The present application divides the continuous time period T into multiple time slots of length Δt (e.g., 15 minutes), expressed as T = {1, 2, ..., t, ..., T}. In the time period T, the edge server s n,m The average power demand distribution is expressed as:
[0085] p n,m :={p n,m (1),p n,m (2),...,p n,m (t),…,p n,m (T)} T
[0086] The power of ES is limited by its rated power, i.e. In actual deployment, the power consumption of the edge server can be easily obtained by reading an external power meter or accessing the bypass operating system. Due to the introduction of the battery system, the power of the edge server is not the final value obtained from the grid.
[0087] 3) Energy generation by renewable energy generators in the previous time slot.
[0088] The total energy generation of the renewable energy generator REG in the time period T is expressed as:
[0089] G={g(1),g(2),…,g(t),…,g(T)}T
[0090] Conventional energy (e.g. dominated by fossil fuels) will be used as a supplement when renewable energy sources cannot provide sufficient energy. However, this will lead to additional carbon emissions (or higher price expenditures due to local policies encouraging carbon reduction).
[0091] In one embodiment, actions are defined based on changes in the charge status of all batteries owned by the agent, including:
[0092] The change in the charging state of all batteries owned by the agent is calculated by the target charging state value of each battery owned by the agent and the upper and lower limits of the charging state, thereby obtaining the action of the agent.
[0093] The overall expression is:
[0094]
[0095] Among them, SoC' n,k (t) represents battery b n,k Target charge state value; SoC n (t) max and SoC n (t) min are the upper and lower limits of the SoC aggregation of edge service provider n, which are calculated as and Then, a n (t) is used to guide the redistribution of specific charging and discharging actions for each battery. In addition, the actions can be discretized into percentage increments from 0% to 100% to simplify the decision-making process. Such discretization may help battery operation (e.g., granularity of less than 1%), reduce battery usage frequency and extend battery life.
[0096] In one embodiment, the reward after performing an action is defined based on carbon emissions and battery degradation costs, including:
[0097] 1) Calculate carbon emissions through the total energy demand of the edge servers served by each battery owned by the agent in the current time slot, the total energy supply, and the energy generation of renewable energy generators in the current time slot; the total energy supply is calculated through the actions performed by the agent.
[0098] Among them, when renewable energy is not enough to meet the energy needs of many edge servers, traditional energy is integrated into the grid to maintain system operation. The edge servers of edge service providers jointly generate carbon emissions. First, focus on the energy demand of a single edge server at time slot t:
[0099] d n,m (t) = p n,m ×Δt;
[0100] The overall energy requirement of the edge server is:
[0101]
[0102] Meanwhile, the total energy demanded by the battery or supplied to the edge server is calculated as:
[0103]
[0104] in, is the actual power obtained by the edge server. Due to factors such as AC / DC conversion efficiency and energy leakage, the battery charging and discharging efficiency cannot be ignored. These factors are expressed as and The actual power obtained from the grid to the ES is expressed as:
[0105]
[0106] The SoC change of the battery after the charge and discharge process is calculated as follows:
[0107]
[0108] The c in the formula n,k (t) Different from the rated capacity of the battery c n,k It is the actual capacity of the battery during time slot t and is related to SoH n,k (t) is closely related. This relationship can be expressed by the following formula:
[0109] c n,k (t) = c n,k ×SoH n,k (t);
[0110] In implementing battery b n,k The decision π n,k (t) After that, the battery status will be updated as follows:
[0111]
[0112] In T time slots, the total carbon emissions generated within the target carbon emission limit are calculated as:
[0113]
[0114] The carbon emissions of a single time slot can also be calculated in this way. Where λ is the carbon intensity parameter, which is used to quantify the carbon emissions generated per unit of grid energy, in units of g / KWh.
[0115] 2) The battery degradation cost is calculated by the change in the battery health status of each battery owned by the agent in the current time slot.
[0116] The ability of a battery to withstand continuous charge and discharge cycles gradually weakens due to the cumulative effects of chemical and physical processes, such as degradation of active materials and changes in the structure of electrode materials. The main factor affecting this degradation is the cumulative depth of discharge (DoD) experienced by the battery. Figure 3 The figure shows the relationship between the maximum number of typical battery cycles and the DoD level. Each point (x, y) represents the battery reaching the threshold. Previously, up to y full discharge cycles could be performed at a DoD of x. n,k SoH n,k (t) changes to quantify the decrease in the number of battery cycles, expressed as:
[0117]
[0118] Assume that battery b n,k The loss cost is ω n,k , then the battery degradation cost of all edge service provider batteries in T time slots is:
[0119]
[0120] The battery degradation cost for a single time slot can also be calculated.
[0121] Finally, we can build an optimization model to minimize the total carbon emissions of the edge service provider for the target renewable energy unit as well as the battery cost:
[0122]
[0123] By solving the optimization model, we can obtain the optimal battery charging and discharging scheduling sequence. However, in the process of problem solving, we will face the following challenges:
[0124] High intrinsic complexity: Nonlinear optimization problems are very difficult to solve directly. For a single battery schedule, the solution space size is Where N p is the size of the battery's candidate action (i.e., charging and discharging power). In distributed scenarios involving multiple batteries, the solution space grows exponentially to As you can imagine, the cost of fixing the problem is enormous, which would be unacceptable when faced with long hours and multiple battery dispatches.
[0125] The need for online decision making: The energy demand of edge servers and renewable energy generation has significant volatility and is difficult to predict in the long term. Especially for off-site scenarios, it is difficult to obtain accurate on-site fine-grained meteorological data, and forecasting of renewable energy is impractical. This means that model-based online algorithms will be useless because there is no input data. At this point, online algorithms are necessary. In addition, it can be observed that the battery status at time slot t+1 (and thereafter) is largely constrained by the scheduling results of time slot t. Online algorithms must have a long-term perspective and be able to make strategic plans based on existing knowledge.
[0126] Information barriers between ESPs: Sharing scheduling information between edge service providers involving battery and edge server configuration families and real-time energy demand of edge servers may be prohibited for commercial operational reasons due to the high sensitivity of scheduling information. Edge service providers rely only on the information they possess for battery scheduling, inevitably leading to suboptimal results. This lack of coordination may cause all edge service providers to charge when renewable energy is abundant and discharge when it is scarce, a phenomenon known as renewable energy competition.
[0127] Classic online algorithms are used to solve sequential decision problems, including Lyapunov and DRL algorithms. However, the Lyapunov algorithm has difficulty coping with distributed scenarios. DRL models the problem as a Markov process and empirically derives decisions from historical data. However, DRL has poor scalability and has difficulty coping with problems with larger decision scales. More importantly, both Lyapunov and DRL algorithms require all data to be input into the decision process, which contradicts the principle of not sharing information between edge service providers, and therefore needs further exploration.
[0128] In one embodiment, the network structure of the deep recursive Q network includes: a gated recurrent unit, an invalid action shielding layer, a perturbation layer, and an aggregation layer in sequence;
[0129] First interaction:
[0130] Input the local observation value of each agent in the current time slot into the deep recursive Q network to obtain a first initial Q value; invalidate the first initial Q value to obtain a first Q value, and determine the action set corresponding to the first Q value through greedy selection; the action set includes the agent actions performed by each agent; each agent performs the action to obtain a reward set, and construct a tuple according to the system state, action set and reward set corresponding to the local observation value, and store the tuple in a buffer; the tuple in the buffer is marked by a sequence number;
[0131] Second interaction:
[0132] Each agent randomly reads a preset number of tuples from its buffer, inputs the preset number of tuples into the deep recursive Q network and the target Q network, and obtains a first joint Q value and a second joint Q value.
[0133] like Figure 4 As shown, a schematic diagram of the framework of the distributed battery scheduling method based on the value decomposition network (VDN) of the present application is provided. Figure 4 As shown, each agent n is trained using a deep recurrent Q-network (DRQN) equipped with weight parameters θ for evaluating the Q-values of future actions. n DRQN uses a gated recurrent unit (GRU) to memorize historical information. In each time slot t, in addition to the local observation o n In addition to (t), DRQN also inputs the hidden state h generated by the Q network n (t-1), thereby implicitly capturing the observation information of historical time slots.
[0134] As shown in Table 1, the training and reasoning process of the distributed battery scheduling algorithm based on the value decomposition network (VDN) of the present application is provided.
[0135] Table 1 Training and inference process of VDN-based distributed battery scheduling algorithm
[0136]
[0137]
[0138] In one embodiment, the invalid action mask parameter in the invalid action mask layer is calculated as follows:
[0139] Calculate the upper limit of the charging state of the agent, and calculate the first action boundary according to the upper limit of the charging state;
[0140] Calculate the lower limit of the charging state of the intelligent agent, and calculate the second action boundary according to the lower limit of the charging state;
[0141] Obtaining an action boundary from a first action boundary and a second action boundary;
[0142] The action mask parameter is negative infinity when the action performed by the agent exceeds the action boundary, and zero otherwise.
[0143] To improve efficiency during training and inference, we introduce the Invalid Action Masking (IAM) mechanism to filter actions to meet constraints 1) and 2):
[0144] 1) Battery capacity limitation:
[0145] The battery's state of charge SoC at any given time period is constrained by its upper and lower limits, expressed as and These limits are imposed by the edge service provider to avoid overcharging / discharging of the battery and are critical for battery safety measures. Therefore, there are:
[0146]
[0147] generally,
[0148] 2) Battery charging / discharging power constraints:
[0149] Battery n,k The maximum charging and discharging powers are expressed as and There are significant differences. For example, in the case of lithium batteries, Can be compared Battery configurations are usually consistent with the energy efficiency consumption level of the service they serve, for example:
[0150]
[0151] where ∈ c and ∈ dc It is a parameter that quantifies the configuration of batteries and edge servers, which is determined by the edge service provider during deployment. You can refer to the common battery energy storage system (BESS) configuration, where ∈ c =5 and∈ dc = 1. As a result, we have:
[0152] Furthermore, we must ensure that the power discharged from the battery can be fully consumed by the associated servers:
[0153]
[0154] Specifically, for agent n, each battery has an upper limit on the charging power. Accordingly, there is an upper limit on the charging state during its charging process, which is expressed as: ;
[0155] We therefore have:
[0156] a n (t) max =min{SoC' n (t) max ,100%};
[0157] Similarly, since the battery cannot be over-discharged, the lower limit of the battery's state of charge is determined by the state of charge of the power source provided by the edge server and its load. We take the minimum value as:
[0158]
[0159] We have:
[0160]
[0161] a n (t) min =max{SoC' n (t) min ,0};
[0162] Then, the IAM parameters of agent n are expressed as: We have:
[0163]
[0164] In one embodiment, outputting the actions of the intelligent agent and guiding the charging and discharging actions of the battery include:
[0165] Given a n (t), find the actual action to be performed by solving the following optimization problem:
[0166]
[0167] Due to the nonlinearity of Γ(·), it is not feasible to solve the above optimization problem directly. Our solution is to decompose Γ(·) into multiple small linear functions, for example, with an interval of 0.001%. This granularity is practical considering the limited precision of the battery DoD. We then use the interior point method to solve this linearized optimization problem.
[0168] In summary, (i) in terms of architecture, this application turns to the most advanced multi-agent deep reinforcement learning by establishing an independent agent for each edge service provider, which helps to increase the scalability of the algorithm and avoid bloated neural networks. (ii) in terms of deployment, a distributed training and distributed execution (DTDE) strategy is adopted, and Q-value perturbation and index exchange strategies are added, which helps to isolate the training and reasoning of each agent and achieve information protection. (iii) in terms of decision-making, an action conversion mechanism is designed to use the agent's decision as a guide for actual execution, thereby improving the overall training and execution efficiency.
[0169] In one embodiment, a battery scheduling device for a multi-edge service provider scenario is provided, comprising: a region determination module, a definition determination module, a sample acquisition module, a joint Q value calculation module, a target value calculation module, a network training module and a battery scheduling module, wherein:
[0170] A region determination module is used to determine an edge computing region of interest; within the edge computing region, multiple edge service providers and multiple batteries are included; each edge service provider has a group of edge servers; each edge service provider operates as an independent intelligent entity;
[0171] A definition determination module is used to determine the definition of the action of the intelligent agent and the reward after the action is executed; specifically, it includes: defining the action according to the change of the charging state of all batteries owned by the intelligent agent; defining the reward after the action is executed according to the carbon emission and battery degradation cost;
[0172] The sample acquisition module is used to obtain tuple samples of the agent; a tuple consists of the system state and agent action of the current time slot, as well as the reward and system state after the action is executed; the system state of the current time slot is obtained by merging the local observations of all agents in the current time slot;
[0173] A joint Q-value calculation module is used to input the local observation of each agent into the corresponding deep recursive Q network, obtain the corresponding first Q-value and perturb it, and then aggregate the first perturbed Q-values of all agents to obtain the first joint Q-value;
[0174] The target value calculation module inputs the local observations after each agent performs an action into the corresponding target Q network, obtains the corresponding second Q value and perturbs it, then aggregates the second perturbed Q values of all agents to obtain the second joint Q value, and obtains the target value according to the second joint Q value and the rewards after all agents perform actions; wherein, the network structure of the target Q network and the deep recursive Q network is the same;
[0175] A network training module, used for constructing a loss function according to the first joint Q value and the target value, and obtaining a trained deep recursive Q network and a target Q network when the loss function converges;
[0176] The battery scheduling module is used to input the actual local observations of the agent into the trained deep recursive Q network, output the action of the agent and guide the charging and discharging actions of the battery.
[0177] For the specific definition of a battery scheduling device for a multi-edge service provider scenario, please refer to the definition of a battery scheduling method for a multi-edge service provider scenario above, which will not be repeated here. Each module in the above-mentioned battery scheduling device for a multi-edge service provider scenario can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0178] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as tuple samples. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a battery scheduling method for a multi-edge service provider scenario is implemented.
[0179] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0180] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.
[0181] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.
[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0183] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0184] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.
Claims
1. A battery scheduling method for a multi-edge service provider scenario, characterized in that: The method comprises: Determine an edge computing area of interest; include multiple edge service providers and multiple batteries in the edge computing area; each edge service provider has a group of edge servers and a group of batteries; each edge service provider operates as an independent intelligent entity; Determine the definition of the agent's actions and the rewards after executing the actions; wherein the actions are defined according to the sum of the charge states of all batteries owned by the agent, and the rewards after executing the actions are defined according to the carbon emissions and the battery degradation cost; Get a tuple sample of agents; a tuple consists of the system state of the current time slot and the next time slot, the agent action, and the reward after the action is performed; the system state of the current and next time slots is obtained by merging the local observations of all agents in the time slot; Input the local observation of each agent into the corresponding deep recursive Q network, obtain the corresponding first Q value and perturb it, then aggregate the first perturbed Q values of all agents to obtain the first joint Q value; The local observation after each agent performs an action is input into the corresponding target Q network to obtain the corresponding second Q value and perturb it, then the second perturbation Q values of all agents are aggregated to obtain the second joint Q value, and the target value is obtained according to the second joint Q value and the rewards after all agents perform actions; wherein, the network structure of the target Q network and the deep recursive Q network is the same; A loss function is constructed according to the first joint Q value and the target value. When the loss function converges, a trained deep recursive Q network is obtained. The actual local observation of the agent is input into the trained deep recursive Q network, the agent Q value is output, and the action is obtained through greedy selection to guide the charging and discharging actions of the battery.
2. The method according to claim 1, characterized in that An agent's local observations include: 1) The status of the battery in the current time slot; the battery status is represented by the charge state, discharge depth and battery health status; 2) The total energy demand of the edge servers owned; 3) Energy generation by renewable energy generators in the previous time slot.
3. The method according to claim 1, characterized in that Actions are defined based on the sum of the charge states of all batteries owned by the agent, including: By adding up the total capacity of all batteries owned by the agent, and probabilistically greedily selecting a percentage from the agent's Q value multiplied by the total battery capacity to represent the target state of charge of all batteries, An optimization problem for minimizing the health loss of each battery is constructed, and the optimal charge and discharge power of each battery is obtained by solving the optimization problem.
4. The method according to claim 1, characterized in that: Define rewards for performing actions based on carbon emissions and battery degradation costs, including: Carbon emissions are calculated through the total energy demand of edge servers owned by all agents, the total charge and discharge of the owned batteries, and the energy generation of renewable energy generators in the current time slot; The battery degradation cost is calculated by the change in the battery health status of each battery owned by the agent at the current time slot.
5. The method according to claim 1, characterized in that Inputting the local observation of each agent into the corresponding deep recursive Q network, obtaining the corresponding first Q value and perturbing it, then aggregating the first perturbed Q values of all agents to obtain the first joint Q value; inputting the local observation after each agent performs the action into the corresponding target Q network, obtaining the corresponding second Q value and perturbing it, then aggregating the second perturbed Q values of all agents to obtain the second joint Q value, including: First interaction: Input the local observation value of each agent in this time slot into the deep recursive Q network to obtain the first initial Q value; Invalidate the first initial Q value to obtain a first Q value, and determine an action set corresponding to the first Q value by greedy selection; the action set includes agent actions executed by each agent; Each agent performs the action to obtain a reward set, constructs a tuple according to the system state, action set and reward set corresponding to the local observation value, and stores the tuple in a buffer; the tuples in the buffer are marked by serial numbers; Second interaction: Each agent randomly reads a preset number of tuples from its buffer, inputs the preset number of tuples into the deep recursive Q network and the target Q network, and obtains a first joint Q value and a second joint Q value.
6. The method according to claim 5, characterized in that The invalid action mask parameters in the invalid action mask layer are calculated as follows: Calculate the upper limit of the state of charge of all batteries owned by the agent, and calculate the first action boundary according to the upper limit of the state of charge; Calculate the lower limit of the charging state of the intelligent agent, and calculate the second action boundary according to the lower limit of the charging state; Obtaining an action boundary from a first action boundary and a second action boundary; The action mask parameter is negative infinity when the action performed by the agent exceeds the action boundary, and zero otherwise.
7. A battery scheduling device for multi-edge service provider scenarios, characterized in that: The device comprises: A region determination module is used to determine an edge computing region of interest; within the edge computing region, multiple edge service providers and multiple batteries are included; each edge service provider has a group of edge servers; each edge service provider operates as an independent intelligent entity; A definition determination module is used to determine the definition of the action of the intelligent agent and the reward after the action is executed; wherein the action is defined according to the sum of the charging states of all batteries owned by the intelligent agent, and the reward after the action is executed is defined according to the carbon emission and the battery degradation cost; The sample acquisition module is used to obtain tuple samples of the agent; a tuple consists of the system state of the current and next time slots, the agent action, and the reward after the action is performed; the system state of the current and next time slots is obtained by merging the local observations of all agents in the time slot; A joint Q-value calculation module is used to input the local observation of each agent into the corresponding deep recursive Q network, obtain the corresponding first Q-value and perturb it, and then aggregate the first perturbed Q-values of all agents to obtain the first joint Q-value; The target value calculation module inputs the local observations after each agent performs an action into the corresponding target Q network, obtains the corresponding second Q value and perturbs it, then aggregates the second perturbed Q values of all agents to obtain the second joint Q value, and obtains the target value according to the second joint Q value and the rewards after all agents perform actions; wherein, the network structure of the target Q network and the deep recursive Q network is the same; A network training module is used to construct a loss function according to the first joint Q value and the target value, and when the loss function converges, a trained deep recursive Q network is obtained; The battery scheduling module is used to input the actual local observations of the agent into the trained deep recursive Q network, output the agent Q value, and obtain actions through greedy selection to guide the charging and discharging actions of the battery.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Edge computing system and method based on electric vehicle assistance
CN116668449A
Multi-agent task decision method based on PF-MADDPG
CN116776929A
Multi-unmanned aerial vehicle wireless charging and edge calculation joint optimization method and system based on deep reinforcement learning
CN116882270A
Electric vehicle edge calculation scheduling method and device, equipment and storage medium
CN116974753A
Edge computing network optimization method and system based on task unloading
CN118118927A
Cited By
Battery replacement scheduling method, device and equipment, computer storage medium and program product
CN121766717A
