A residential aggregator optimal energy management method based on deep reinforcement learning

Abstract

This invention provides a residential aggregator-based optimal energy management method, comprising the following steps: constructing an aggregator model, building generalized energy storage models for three types of residential appliances—batteries, HVAC systems, and electric vehicles—and aggregating them into a residential aggregator for centralized management; generating scheduling instructions for the aggregator, in the first decision-making stage, formalizing the optimization control process into a Markov decision process, interacting with the environment based on the Soft Actor-Critic framework, learning the optimal scheduling strategy, and tracking the upper-level power grid scheduling while ensuring operational constraints; decomposing the aggregator's control instructions, in the second decision-making stage, the residential aggregator generates scheduling plans for each appliance through a three-step decomposition algorithm to follow the aggregated power scheduling defined in the first decision-making stage. This invention achieves automated and integrated control of residential energy, improves the integration and intelligence level of the energy system, reduces operating costs, and enhances user comfort.

Description

Technical Field

[0001] This invention belongs to the field of distributed energy control technology, and more specifically, relates to an optimal energy management method for residential aggregators based on deep reinforcement learning. Background Technology

[0002] With the rapid development of smart grid technology, the deployment of distributed energy resources (DERs) in residential energy systems is becoming increasingly widespread, including rooftop solar photovoltaics, electric vehicles, battery storage systems, and HVAC systems. These DERs not only improve the energy efficiency and reliability of the power grid but also generate additional revenue for home users through energy trading. However, the integration of large-scale DERs also brings significant energy management challenges, especially in areas containing thousands of homes. How to coordinate the dynamic operation of various types of appliances, meet the personalized needs of users, and ensure grid stability has become a core issue that urgently needs to be addressed.

[0003] Traditional home energy management (HEM) methods are mainly divided into two categories: model-based optimization methods and model-free data-driven methods. Model-based optimization methods rely on constructing accurate mathematical models and solving for optimal scheduling strategies through mathematical programming (such as mixed-integer linear programming and robust optimization). For example, some studies use multi-objective optimization models to simultaneously reduce electricity costs, improve user comfort, and temperature stability; others utilize clustering algorithms or information gap decision theory to handle the uncertainty of user behavior and electricity prices. However, these methods have significant limitations, such as high model dependence, high computational complexity, and insufficient adaptability. Model-free data-driven methods, represented by deep reinforcement learning (DRL), learn optimal strategies through the interaction between the agent and the environment, avoiding the complexity of explicit modeling. The DQN (Deep Q-Network) algorithm performs well in discrete action spaces but cannot handle continuous control problems; TD3 (Dual Delay Deep Deterministic Policy Gradient) and SAC (Flexible Action-Evaluation) algorithms support continuous action spaces and can achieve fine-grained control of home appliance power. While existing DRL-based HEM research has shown potential in dynamic environments, it still faces the following challenges: 1) High-dimensional action space challenges: When managing multiple types of devices (EVs, HVACs, energy storage, etc.) in thousands of homes, the dimension of the action space grows exponentially, leading to a surge in training sample requirements and slow convergence speed; 2) Insufficient user behavior modeling: Existing studies often simplify EV charging needs to fixed arrival / departure times and target charge levels, ignoring drivers' personalized behaviors (such as "range anxiety" and "time anxiety"), resulting in discrepancies between scheduling strategies and actual user needs; 3) Limited collaborative scheduling capabilities: Existing aggregator technologies are mostly designed for single device types (EVs or HVAC clusters), lacking comprehensive optimization for the coordinated operation of multiple types of home appliances, making it difficult to maximize global energy efficiency.

[0004] In summary, existing HEM technologies have significant shortcomings in the following aspects: 1) Curse of dimensionality: Traditional optimization methods and single DRL algorithms are difficult to effectively cope with the high-dimensional action space of large-scale residential groups, resulting in excessive consumption of computational resources or suboptimal strategies; 2) Poor dynamic adaptability: Model-based methods rely on static assumptions and cannot adapt to real-time changing environments (such as sudden changes in user behavior and fluctuations in grid demand); while model-free DRL methods have low sample efficiency and insufficient training stability in high-dimensional scenarios; 3) Neglect of user needs: Existing scheduling strategies are mostly based on economic efficiency and do not fully consider personalized user behaviors (such as EV charging anxiety and indoor temperature preferences), resulting in a decline in user experience.

[0005] 4) Lack of collaborative management: There is a lack of a systematic framework for the collaborative scheduling of various types of home appliances, making it difficult to balance the energy interaction between devices and the conflict of user needs. Summary of the Invention

[0006] To address the shortcomings and improvement needs of existing technologies, this invention provides an optimal energy management method for residential aggregators based on deep reinforcement learning. The aim is to promote the efficient and coordinated operation of various types of home appliances in residential aggregators, thereby improving energy supply quality and operational economy.

[0007] The technical solution provided by this invention is: an optimal energy management method for residential aggregators based on deep reinforcement learning, comprising the following steps:

[0008] (1) Construct an aggregator model. First, construct a unified model of generalized energy storage forms for three types of residential appliances: batteries, HVAC, and electric vehicles. Then, aggregate them into residential aggregators for centralized management based on the generalized energy storage model of residential appliances.

[0009] (2) Generate scheduling instructions for aggregators. In the first decision-making stage, the optimization control process is formalized into a Markov decision process. The deep reinforcement learning controller of the residential aggregator interacts with the environment based on the Soft Actor-Critic framework, learns the optimal scheduling strategy, and tracks the upper-level power grid scheduling under the premise of ensuring operational constraints.

[0010] (3) Decompose the control instructions of the aggregator. In the second decision stage, the residential aggregator generates the scheduling plan of each home appliance through a three-step decomposition algorithm to follow the aggregated power scheduling defined in the first decision stage. First, the decomposition algorithm collects the status information of the home appliances, including SoC, individual user behavior, and power range. Second, the algorithm generates individual power instructions for each home appliance. Finally, each control power value is sent to the corresponding home appliance unit.

[0011] The unified model for generalized energy storage in step (1) is as follows:

[0012] ,

[0013] ,

[0014] ,

[0015] ,

[0016] Where, p i Indicates the power consumption of appliance i; e i p represents the state of charge of electrical appliance i; i and e i θ represents the state parameters of the electrical appliance. iη represents the dissipation rate of appliance i; i σ represents the conversion factor of appliance i; i θ represents the influence factor of ambient temperature on appliance i; i η i and σ i For electrical equipment parameters; p i,min and p i,max δ represents the lower and upper bounds of the power consumption of device i, respectively; i,max e represents the ramp-up limit of appliance i; i,min and e i,max These represent the lower and upper bounds of the indoor temperature of appliance i, respectively.

[0017] Furthermore, in step (1), the equivalent parameters of the aggregator model used for upper-level scheduling are obtained by calculating the weighted average of the parameters of each household appliance, specifically expressed as follows:

[0018] ,

[0019] ,

[0020] ,

[0021] ,

[0022] in, and These are the power consumption and remaining energy of aggregator M, respectively; and These are the lower and upper limits of the power consumption of aggregator M; and These are the lower and upper limits of the remaining energy of the aggregator M; , and Let M represent the dissipation rate, conversion coefficient, and ambient temperature influence factor, respectively. The weighted average of the aggregator model is calculated by directly summing the power and energy boundary parameters, and weighting the dissipation rate, conversion coefficient, and temperature influence factor according to the rated power of each device.

[0023] Furthermore, the aggregator model constructed in step (1) considers the operating costs, noise costs, and power point tracking errors of residential appliances, which are expressed as follows:

[0024] ,

[0025] ,

[0026] ,

[0027] in, , and These represent the total operating cost, total noise cost, and power point tracking error of aggregator M, respectively. and This is an approximation of the operating cost coefficient of aggregator M. It is an approximation of the noise figure of the aggregator M. For electricity demand; The local power load, including rooftop photovoltaic units and other non-storable loads, operating costs, noise costs, and power point tracking errors, are used to construct the reward function for the deep reinforcement learning controller in the first decision-making stage.

[0028] Furthermore, the decision-making method in step (2) is based on a Markov decision process, which is defined by variables {S, A, P, R, γ}, where S represents the set of environmental states observed by the deep reinforcement learning controller, A represents the set of actions, P represents the transition probability from any state s∈S to any state s'∈S, R is the set of immediate rewards for any action a∈A, and γ∈[0,1] represents the discount rate used to penalize future rewards. The Markov decision process is solved using the Soft Actor-Critic algorithm, which uses the maximum entropy reinforcement learning framework. The Soft Actor-Critic framework completes the decision-making process by executing the following steps: (2-1) State observation: The controller observes the current state s of the aggregator from the environment. M,t (2-2) Action generation: Generate state s M,t Input to online action network And by the online action network Determine the action of the aggregator, i.e., the power output rate a. M,t (2-3) Environmental interaction and reward calculation: Aggregator according to action a M,t Output its power P M (t), the environment transitions to the next state s. M,t+1 Simultaneously calculate the instant reward r t (2-4) Experience storage: storing the quadruple (s M,t , a M,t , r M,t , s M,t+1 Store in the experience replay buffer D i In the middle, it is used to train the deep reinforcement learning agent; (2-5) Network update: from buffer D i Randomly sample small batches of data and update them sequentially: Soft Commentator Network q θ Online Action Network Entropy regularization parameter α, and target action network Repeat steps (2-1)-(2-5) above until the algorithm converges, thus completing the first stage of determining the optimal power at the system level for the aggregator.

[0029] Furthermore, the deep reinforcement learning controller is based on local observers. M,t Solve operational problems.

[0030] ,

[0031] in, and These are the power consumption and remaining energy of aggregator M, respectively; and These are the lower and upper limits of the power consumption of aggregator M; It is a power requirement; It is the discrepancy between power demand and actual power consumption;

[0032] Action a M,t ∈[0,1] is defined as power output rate, power consumption P M (t) is naturally restricted to [P] M,min , P M,max Within the range of , the joint action at time step t is represented as a t = (a 1,t , a 2,t ,…, a N,t ), P M The formula for calculating (t) is:

[0033] ;

[0034] Among them, P M (t) and E M (t) represents the power consumption and remaining energy of aggregator M, respectively; P M,min (t) and P M,max (t) represents the lower and upper limits of the power consumption of aggregator M; P D (t) represents the power demand; It is the discrepancy between power demand and actual power consumption.

[0035] System status can be obtained from s t Transfer to s t+1 The probability is P(s) t , s t+1 ) = Pr(s t+1 | s t , a t When the system state changes from s t Transfer to st+1 At that time, the aggregator's controller will receive a reward r t :

[0036]

[0037] The reward function consists of three parts: tracking error cost, operating cost, and acoustic discomfort cost. , and These are the weighting coefficients for tracking error cost, operating cost, and acoustic discomfort cost, respectively.

[0038] The goal of the aggregator deep reinforcement learning controller is to maximize the expected reward value at T time steps, and its scaling function J is expressed as:

[0039] ,

[0040] Where π is the generation action. Control strategy, based on state The discount rate γ determines the impact of future rewards on current rewards. It is a state The entropy of the next strategy π, where α is a temperature parameter, determines the importance of entropy relative to the reward.

[0041] Furthermore, in step (3), the three-step decomposition algorithm first allocates the initial control signal of the storable load power proportionally according to the device power range:

[0042] ,

[0043] in, It is the power control signal for home appliance i. and These are the upper and lower power limits for appliance i, respectively.

[0044] The three-step decomposition algorithm considers the individual user behavior of electric vehicles as their charging needs, and the power command allocated to the electric vehicle is expressed as:

[0045]

[0046]

[0047] in, It is the power control signal for electric vehicle i. The charging power for electric vehicles is given by α and β, which are shape parameters, and a1 and a2, which are variables related to the arrival and departure times of the vehicle.

[0048] Furthermore, to meet the charging needs of electric vehicles, aggregators require additional power. This additional power is supplemented by other storage-capable appliances within the same aggregator in proportion to their power range, as shown below:

[0049] ,

[0050] ,

[0051] in, This indicates the additional power required by aggregator M to meet the dynamic user behavior of electric vehicles. For the control signal corresponding to the additional power of the storable device i, agg M Let M be the aggregator.

[0052] Furthermore, in step (3), additional power is allocated through an iterative process, while ensuring feasibility and meeting SoC / temperature boundaries. The iteration rule is as follows: Initial state and initial additional power The initial power range is , The initial power allocated to each individual load is calculated as follows:

[0053] .

[0054] Furthermore, the iteration number is named iter, and in each iteration step, for each storable load, if p i,max >0, then:

[0055] ,

[0056] Among them, for or When the predicted load reaches the power and SoC boundary, known as saturated load, the power range of saturated load is set to 0 to avoid allocating power to them:

[0057] ,

[0058] ,

[0059] Minimum aggregator M Calculate the additional power:

[0060] ,

[0061] ,

[0062] If all aggregators satisfy If the extra power has been fully allocated, the algorithm's output is sent to each unit in step (3); otherwise, iter = iter+1, and the iteration continues.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1) High-efficiency optimization of residential energy management problem: Through the proposed two-stage aggregation control architecture, namely aggregation scheduling and dynamic decomposition, the high-dimensional action space of distributed energy and home appliances of large-scale residential users is compressed to the aggregation quotient level, which significantly reduces the computational complexity. Based on the SAC framework, an entropy regularization mechanism is introduced into deep reinforcement learning, thereby realizing fine-grained control in the continuous action space, improving the convergence speed and sample efficiency of the algorithm, and effectively solving the solution difficulties caused by the curse of dimensionality in traditional methods.

[0065] 2) User behavior-driven dynamic adaptability: Introducing a user anxiety quantification model (such as "range anxiety RA" and "time anxiety TA" for EVs) to dynamically adjust charging priority and power allocation strategies. This characterizes the psychological sensitivity of users' charging needs, ensuring that EVs reach the target state of charge (SOC≥80%) before the anxiety period, significantly improving user satisfaction and charging demand fulfillment rate.

[0066] 3) Enhanced Multi-Device Collaborative Management Capabilities: Batteries, HVAC systems, electric vehicles, and other energy-storing appliances are uniformly modeled as generalized energy storage. Through dynamic equations and their constraints, the energy state and power interaction relationships of these devices are standardized. This model supports a collaborative framework of centralized scheduling and distributed execution, enabling global energy efficiency optimization and safe operation of multiple device types (energy storage, EVs, HVAC).

[0067] 4) Balancing Economy and Comfort: The dynamic decomposition algorithm prioritizes critical loads (such as EV charging needs) and uses an iterative correction mechanism (saturation detection and power redistribution) to ensure that the power and status of each device (such as battery SOC and indoor temperature) do not exceed limits. At the same time, the indoor temperature is stabilized within a comfortable range, improving user acoustic comfort, while reducing operating costs and tracking errors.

[0068] In summary, this invention provides a specific theoretical basis and solution for optimal energy management of various types of home appliances in residential aggregators. It is of great significance for promoting the automated and integrated control of various types of storage devices and improving the integration and intelligence level of energy systems. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of a photovoltaic integrated residential building.

[0070] Figure 2This is a schematic diagram showing the relationship between the charging power and control signal of an electric vehicle.

[0071] Figure 3 This is a schematic diagram illustrating the training performance of the proposed SAC algorithm.

[0072] Figure 4 This is a diagram illustrating the energy consumption plans of various residential aggregators.

[0073] Figure 5 This is a diagram illustrating the control effect under different weighting coefficients.

[0074] Figure 6 This is a schematic diagram showing the power distribution results for individual devices.

[0075] Figure 7 This is a schematic diagram of the power distribution results for individual energy storage systems, a) the proposed method, b) the MPC method.

[0076] Figure 8 This is a diagram comparing the cumulative rewards of different methods. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0078] An optimal energy management method for residential aggregators based on deep reinforcement learning includes:

[0079] To build an aggregator model, firstly, a unified model of generalized energy storage forms for three types of residential appliances—batteries, HVAC systems, and electric vehicles—is constructed. Then, based on the generalized energy storage model of these residential appliances, they are aggregated into residential aggregators for centralized management.

[0080] The scheduling instructions for the aggregator are generated. In the first decision-making stage, the optimization control process can be formalized as a Markov decision process (MDP). The deep reinforcement learning controller of the residential aggregator interacts with the environment based on the Soft Actor-Critic (SAC) framework to learn the optimal scheduling strategy and track the upper-level power grid scheduling while ensuring operational constraints.

[0081] In the second decision-making stage, the residential aggregator decomposes the control commands of the aggregator. The aggregator generates a scheduling plan for each appliance through a three-step decomposition algorithm to follow the aggregated power scheduling defined in the first decision-making stage. First, the decomposition algorithm collects the status information of the appliances, including SoC, individual user behavior, power range, etc. Second, the algorithm generates individual power commands for each appliance. Finally, each control power value is sent to the corresponding appliance unit.

[0082] Residential appliances can be divided into three categories: on-site power generation, storable loads, and non-storable loads. This invention only considers rooftop photovoltaic units as on-site power generation for residences; the main dispatch and control objects are storable loads, including batteries, HVAC systems, and electric vehicles.

[0083] The storable home appliances in the aforementioned residential aggregator can be uniformly modeled as a generalized form of energy storage:

[0084]

[0085]

[0086]

[0087]

[0088] Where, p i Indicates the power consumption of appliance i; e i p represents the state of charge of electrical appliance i; i and e i θ represents the state parameters of the electrical appliance. i η represents the dissipation rate of appliance i; i σ represents the conversion factor of appliance i; i θ represents the influence factor of ambient temperature on appliance i; i η i and σ i For electrical equipment parameters; p i,min and p i,max δ represents the lower and upper bounds of the power consumption of device i, respectively; i,max e represents the ramp-up limit of appliance i; i,min and e i,max These represent the lower and upper bounds of the indoor temperature for appliance i, respectively. The specific parameter values ​​and their corresponding device relationships are as follows: 1) Battery: θ i 0, η i 0, σ i = 0; 2) Heating, ventilation and air conditioning: θ i 0, η i 0, σ i 0; 3) Electric vehicles: θ i 0, η i 0, σ i = 0.

[0089] By unifying the modeling of different types of storable home appliances, a dynamic mathematical model of various home appliances (such as electric vehicles, battery energy storage systems, and HVAC) in residential aggregators is characterized in the form of generalized energy storage. This facilitates the construction of subsequent residential aggregator models. The model can be adapted to various home appliance characteristics, including battery charging and discharging efficiency and air conditioning heat loss rate, providing a scalable technical foundation for the integrated management of future multi-energy systems (such as heat pumps and electric boilers).

[0090] Based on this unified form, the equivalent parameters of the aggregator model used for upper-level scheduling are obtained by calculating the weighted average of the parameters of each household appliance, specifically expressed as follows:

[0091]

[0092]

[0093]

[0094]

[0095] in, and These are the power consumption and remaining energy of aggregator M, respectively; and These are the lower and upper limits of the power consumption of aggregator M; and These are the lower and upper limits of the remaining energy of the aggregator M; , and These represent the dissipation rate, conversion coefficient, and ambient temperature influence factor of the aggregator M, respectively. The weighted average of the aggregator model is calculated by directly summing the power and energy boundary parameters, and then weighting the dynamic parameters (dissipation rate, conversion coefficient, and temperature influence factor) according to the rated power of each device.

[0096] Based on generalized energy storage modeling of storable home appliances, the approximate parameters of residential aggregators can be divided into two types: {P M,min , P M,maxE M,min E M,max , δ M,max This is related to the running boundary, and the approximation is calculated by directly summing the corresponding parameters; {θ} M , η M , σ M A 0,M A 1,M A 2,M B M , ζ M It is associated with the dynamic model, and the approximate value is calculated by the weighted average of the corresponding individual parameters. The weights can be determined by the rated active capacity of the equipment. In this way, the dynamic model of the residential aggregator participating in the scheduling can be obtained in a timely manner from the equipment parameters, thereby centrally managing the distributed energy and residential appliances of a large number of residential users and reducing the dimensionality of the optimization problem.

[0097] The aggregator model takes into account the operating costs of residential appliances, noise costs, and power point tracking errors, which are expressed as follows:

[0098]

[0099]

[0100]

[0101] in, , and These represent the total operating cost, total noise cost, and power point tracking error of aggregator M, respectively. and This is an approximation of the operating cost coefficient of aggregator M. It is an approximation of the noise figure of the aggregator M. For electricity demand; This refers to local power load, including rooftop solar PV units and other non-storable loads. The three costs mentioned above are used to construct the reward function of the deep reinforcement learning controller in the first decision-making phase, driving the agent to learn the optimal scheduling strategy that minimizes long-term operating costs, noise costs, and tracking errors.

[0102] The deep reinforcement learning controller of the aggregator utilizes the Soft Actor-Critic framework to learn the optimal scheduling policy and generate aggregated control actions through interaction with the environment. This decision-making process is first formalized as a Markov decision process, defining variables {S, A, P, R, γ}, where S represents the set of environmental states observed by the deep reinforcement learning controller, A represents the set of actions, P represents the transition probability from any state s∈S to any state s'∈S, R is the set of immediate rewards for any action a∈A, and γ∈[0,1] represents the discount rate used to penalize future rewards. Based on this, the Soft Actor-Critic algorithm is used to solve the Markov decision process. The Soft Actor-Critic algorithm, through the maximum entropy reinforcement learning framework, encourages the maximization of policy entropy while maximizing cumulative rewards, thereby improving exploration efficiency and learning stability.

[0103] Deep reinforcement learning controllers based on local observers M,t Solve operational problems.

[0104]

[0105] in, and These are the power consumption and remaining energy of aggregator M, respectively; and These are the lower and upper limits of the power consumption of aggregator M; It is a power requirement; It is the discrepancy between power demand and actual power consumption.

[0106] Action a M,t ∈[0,1] is defined as the power output rate, thus the power consumption P M (t) is naturally restricted to [P] M,min ,P M,max Within the range of ]. The joint effect at time step t can be expressed as a t = (a 1,t , a 2,t ,…, a N,t P M The formula for calculating (t) is:

[0107]

[0108] Among them, P M (t) and E M (t) represents the power consumption and remaining energy of aggregator M, respectively; P M,min (t) and P M,max (t) represents the lower and upper limits of the power consumption of aggregator M; P D (t) represents the power demand; It is the discrepancy between power demand and actual power consumption.

[0109] Based on local observer information, the aggregator's deep reinforcement learning controller utilizes a soft actor-critic framework to learn the optimal scheduling strategy through interaction with the environment. This control method has the advantages of timeliness and not requiring an accurate mathematical model. Simultaneously, the power consumption of the aggregator control action is generated according to the above method. Nature is confined to [P] M,min , P M,max Within the range of ], the equipment operation constraints are met.

[0110] System status can be obtained from s t Transfer to s t+1 The probability is P(s) t , s t+1 ) = Pr(s t+1 | s t , a t When the system state changes from s t Transfer to s t+1 At that time, the aggregator's controller will receive a reward r t :

[0111]

[0112] The reward function can be divided into three parts: tracking error cost, operating cost, and acoustic discomfort cost. , and The weighting coefficients for these three parts respectively.

[0113] The goal of the aggregator deep reinforcement learning controller is to maximize the expected reward value at T time steps, and its scaling function J is expressed as:

[0114]

[0115] Where π is the generation action. Control strategy, based on state The discount rate γ determines the impact of future rewards on current rewards. It is a state The entropy of the next strategy π, where α is a temperature parameter, determines the importance of entropy relative to the reward.

[0116] The Soft Actor-Critic algorithm is a state-of-the-art continuous control model-free reinforcement learning algorithm that can handle high sample complexity and improve the stability of model-free deep reinforcement learning methods. Compared with standard reinforcement learning methods, Soft Actor-Critic incorporates the policy entropy into the reward. This maximum entropy reinforcement learning framework encourages the exploration process, improves the learning speed, and enables the algorithm to better explore the state space and avoid getting trapped in local optima.

[0117] The Soft Actor-Critic algorithm for unified energy control of the aggregator uses a target network and an experience replay buffer to train the action and critic networks. The target network is a copy of the action or critic network used to calculate the target value during training. The experience replay buffer is used to store state-action-reward-next state information for training.

[0118] Using the target network (a copy of the action or commentator network) to compute the target value during training can reduce fluctuations during training and improve the stability of the algorithm. The experience replay buffer stores state-action-reward-next state information for training, which allows the algorithm to make full use of historical data, improve data utilization efficiency, reduce dependence on real-time data, and enhance the robustness of the algorithm.

[0119] Based on the SAC algorithm theory, this invention proposes a power dispatching strategy using a Soft Actor-Critic framework learning architecture for unified energy control by aggregators. The pseudocode of the algorithm is shown in Algorithm 1. Specifically, the action network is responsible for determining the state s. M,t Determine power scheduling action a M,t Joint action a t = (a 1,t , a 2,t ,…, a N,t The current state and the action determine the next state of the environment and the reward function of the DRL controller. Information about the state, action, and reward is represented as s. M,t ,a M,t , r M,t , and the next state s M,t+1 It is recorded and stored in relay buffer D i In this context, it is used to train the DRL controller.

[0120] As shown below, the proposed algorithm includes two neural networks: an actor network. And soft commentator network q θ These are updated through interactions with the environment. Furthermore, the algorithm includes two identical action networks, denoted as... and They interact with the environment and determine power scheduling. M,t Meanwhile, the algorithm uses a soft critic network q θ The action-value function is approximated by using the soft Bellman equation to evaluate s. M,t Next action a M,t The quality. (Commentator network q) θ State-action pair (s) t , a t The SAC algorithm uses a target network and an experience replay buffer to train the action and critic networks. The target network is a copy of the action or critic network used to compute the target value during training. The experience replay buffer stores state-action-reward-next state information for training.

[0121]

[0122] After determining the power scheduling plan of each aggregator using Algorithm 1, a dynamic user behavior-based aggregator deaggregation method is proposed for the decomposition control of home appliances within a single residential aggregator. This method can decompose the aggregated power scheduling into scheduling plans for each home appliance. The pseudocode of the algorithm is shown in Algorithm 2.

[0123]

[0124] In the Soft Actor-Critic algorithm used for the unified energy control of the aggregator, the soft Q function is parameterized using a neural network, with parameter θ serving as... Furthermore, the commentator network parameters are trained by minimizing the squared residual error, expressed as the following soft Bellman equation:

[0125]

[0126]

[0127]

[0128] in, It is a soft state value function, and D is the empirical replay buffer. It uses the state estimate of the target network. and Let represent the action and state at time t, respectively, and γ be the discount rate.

[0129] By parameterizing the soft Q function using a neural network, the SAC algorithm can adapt to the control requirements of continuous states and action spaces, thus meeting the real-world scenario of continuously adjustable power for various home appliances in residential energy management.

[0130] The Soft Actor-Critic (SAC) framework completes the decision-making process by performing the following steps: 1) State observation: The controller observes the current state of the aggregator from the environment, i.e., the state of the s... M,t 2) Action generation: Generate state s M,t Input to online action network And by the online action network Determine the action of the aggregator, i.e., the power output rate a. M,t 3) Environmental interaction and reward calculation: Aggregators calculate rewards based on action a. M,t Output its power P M (t), the environment transitions to the next state s. M,t+1 Simultaneously calculate the instant reward r t (Including operating costs, noise costs, and tracking errors); 4) Experience storage: storing the quadruple (s M,t , a M,t ,r M,t , s M,t+1 Store in the experience replay buffer D i In the middle, it is used to train deep reinforcement learning agents; 5) Network update: from buffer D i Randomly sample small batches of data and update them sequentially: Soft Commentator Network q θ (Minimizing the residuals of the soft Bellman equation), online action network (Minimize KL divergence), entropy regularization parameter α, and target action network Repeat steps 1 to 5 above until the algorithm converges, thus completing the first stage of determining the optimal system-level power for the aggregator.

[0131] The aggregator's unified energy control is based on a soft actor-critic framework, and the control algorithm includes two neural networks, namely, action networks. And soft commentator network q θ They are updated through interaction with the environment, while using two identical action networks. and They interact with the environment and determine power scheduling. M,t In addition, the soft critic network q was used. θ The network evaluates s by approximating the action-value function using the soft Bellman equation. M,t Next action a M,t Quality, online critics q θ State-action pair (s) t , a t The input is the estimated Q value.

[0132] By interacting with the environment through two identical action networks and determining power scheduling, the optimal action can be selected more accurately based on the current state, thereby achieving more efficient energy allocation and utilization and optimizing power scheduling. The Soft Commentator Network uses the soft Bellman equation to approximate the action value function, evaluates the quality of the action in the current state, and outputs the estimated Q value, which helps to more accurately evaluate the merits of different actions, thereby optimizing the control strategy.

[0133] The three-step decomposition algorithm first allocates the initial control signal for storable load power proportionally according to the device power range:

[0134]

[0135] in, It is the power control signal for home appliance i. and These are the upper and lower limits of the power of appliance i, respectively.

[0136] The three-step decomposition algorithm considers the individual user behavior of electric vehicles as their charging needs, and the power command allocated to the electric vehicle can be expressed as:

[0137]

[0138]

[0139] in, It is the power control signal for electric vehicle i. For the charging power of electric vehicles, α and β are shape parameters. A larger α leads to a higher SoC at departure time, while a larger β indicates a higher SoC during charging. This precisely characterizes the driver's range anxiety (RA) and time anxiety (TA). a1 and a2 are variables related to the vehicle's arrival and departure times.

[0140] By introducing "range anxiety (RA)" and "time anxiety (TA)" of electric vehicles, a quantitative model reflecting user anxiety was constructed. This model can characterize the psychological sensitivity of users' charging needs, dynamically adjust charging priority and power allocation strategies, and ensure that the residential energy optimization management process can take into account the dynamic needs of users, significantly improving user satisfaction and charging demand fulfillment rate.

[0141] To meet the charging needs of electric vehicles, aggregators require additional power. This additional power should be supplemented by other storage-capable appliances within the same aggregator in proportion to their power range, as follows:

[0142]

[0143]

[0144] in, This indicates the additional power required by aggregator M to meet the dynamic user behavior of electric vehicles. For the control signal corresponding to the additional power of the storable device i, agg M Let M be the aggregator.

[0145] The described three-step decomposition algorithm employs an iterative process to allocate additional power while ensuring feasibility and meeting SoC / temperature boundaries. The iterative rule for this allocation method is as follows: Initial state and initial additional power As defined in claim 15, the initial power range is , The initial power allocated to each individual load is calculated as follows:

[0146]

[0147] In each iteration step (the iteration number is named iter), for each storable load, if p i,max >0, then:

[0148]

[0149] Among them, for or The load prediction reaches the power and SoC boundary, known as saturated load. Here, the power range of saturated load is set to 0 to avoid allocating power to them:

[0150]

[0151]

[0152] Minimum aggregator M Calculate the additional power:

[0153]

[0154]

[0155] If all aggregators satisfy If the extra power has been fully allocated, the algorithm's output is sent to each cell in step 3. Otherwise, iter = iter + 1, and the iteration continues.

[0156] By using an iterative design process to allocate aggregated power to appliance units, not only is feasibility ensured—that is, the power allocated to each device is within its upper and lower power limits—but the SoC / temperature boundaries of storable devices are also taken into account to meet the dynamic needs and comfort of users using the devices. It can be seen from the iterative process that as long as there is residual energy released from the battery and HVAC, the charging needs of electric vehicles and the power plan defined in the first decision phase are met.

[0157] Figure 1 The effectiveness of the proposed method is verified using a schematic diagram of a photovoltaic integrated residential building as an example. The simulation considered eight aggregators, each comprising approximately 1000 households. The simulation environment is based on real-world data, including various parameters and operating cost coefficients, as shown in Table 1.

[0158] Each household is equipped with an electric vehicle, HVAC system, and energy storage battery. The photovoltaic power station's output power comes from a factory in eastern China, with the first month (30 days) of each quarter serving as the training set, totaling 120 days, and the remaining days used for evaluation.

[0159] Table 1. Device parameters for storable loads

[0160] The proposed algorithm's critic network and action network both consist of four consecutive fully connected layers, each containing 128 hidden units. The aggregator operator is implemented in PyTorch using Python. Table 2 lists the hyperparameters of the learning algorithm in the proposed method.

[0161] Table 2 Hyperparameters of the learning algorithm

[0162] The proposed method was compared with two benchmarks:

[0163] 1) Perfect Information Optimization (PIO): PIO is an offline optimization method that can accurately predict all uncertain information, including future outdoor temperature, driver behavior, and aggregator parameters, and then use optimization solvers such as Cplex to solve the optimal running problem.

[0164] 2) Model Predictive Control (MPC): MPC relies on predictions of short-term future environmental information and assumes that the controller knows the distribution of each device parameter and predicts based on the average value. Therefore, the operational problem is transformed into a deterministic mathematical model, and only the first step of scheduling is performed.

[0165] like Figure 3 As shown, the shaded curves represent the average and actual daily episode rewards, respectively. The total reward is divided into three parts: tracking error, operational cost, and acoustic discomfort cost, which are represented as follows:

[0166]

[0167] from Figure 3 It can be seen that the proposed algorithm can learn a stable operating strategy through interaction with the environment within the first 6000 rounds. This result demonstrates that the SAC method is effective in finding the optimal strategy for energy storage aggregators.

[0168] This algorithm can also improve residential power aggregators' ability to track electricity demand curves. The aggregator's actual power output versus power demand curve is shown below. Figure 4 As shown, the orange curve represents the electricity demand curve, and the actual power of the aggregator is represented by bars of different colors. The results demonstrate that the DRL-based algorithm exhibits good performance in terms of tracking accuracy and stability, proving its effectiveness in solving the HEM problem. Figure 5 As shown, the tracking error varies with the weighting coefficient. The noise level decreases as the energy consumption increases. This algorithm not only reduces operating costs but also improves user acoustic comfort by balancing power demand and supply, while taking into account the impact of residential user behavior on energy consumption.

[0169] The deaggregation performance of aggregator 1 (i.e., the unified scheduling of power distribution to various internal appliances) is as follows: Figure 6 As shown in the figure, simulation results validate the effectiveness of the proposed two-stage HEM algorithm in controlling indoor temperature and the system-on-chip (SoC) of the battery and electric vehicle (EV). The proposed two-stage HEM algorithm allows HVAC and the battery to release more energy to support EV charging when the driver is more concerned about uncertainty, thereby reducing the anxiety penalty. Therefore, before the anxiety period, the EV's SoC level can exceed 80% of the expected level, indicating that HVAC, battery, and EV can be effectively coordinated to meet power demand and maintain a comfortable indoor temperature (20–24°C). Figure 7 Simulation results show that, using the MPC method, due to prediction errors, the residential indoor temperatures of aggregators 1, 4, 5, and 6 remain near the lower bound, and after t = 48 h, the battery SoC of aggregators 4, 5, and 6 drops to 0. These results indicate that the active power allocated to these aggregators by the predictive controller is insufficient to maintain normal operation for residential users. Furthermore, on day 5, after t = 96 h, the electric vehicles of aggregators 4, 5, and 6 still have not reached the expected 80% SoC level before the anxiety period.

[0170] The performance comparison between the proposed algorithm and the benchmark can also be achieved through... Figure 8The cumulative reward is used to display the results. Besides PIO and MPC, we also considered TD3 and PPO as benchmarks. The test set contains 300 days of data from the database. The PIO algorithm performed best and was considered a relatively ideal result. The MPC method is highly dependent on prediction accuracy. Figure 7 (b) In highly uncertain environments, the MPC method cannot guarantee the long-term operation of residential aggregators. Therefore, the cumulative reward of the MPC method is the lowest. SAC's control performance outperforms other non-policy and policy DRL algorithms, including TD3 and PPO. SAC performs well, approaching the ideal results of PIO, and its advantage stems from non-policy updates and the maximum entropy framework.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for optimal energy management of residential aggregators based on deep reinforcement learning, characterized in that, Includes the following steps: (1) Construct an aggregator model. First, construct a unified model of generalized energy storage forms for three types of residential appliances: batteries, HVAC, and electric vehicles. Then, aggregate them into residential aggregators for centralized management based on the generalized energy storage model of residential appliances. (2) Generate scheduling instructions for aggregators. In the first decision-making stage, the optimization control process is formalized into a Markov decision process. The deep reinforcement learning controller of the residential aggregator interacts with the environment based on the Soft Actor-Critic framework, learns the optimal scheduling strategy, and tracks the upper-level power grid scheduling under the premise of ensuring operational constraints. (3) Decompose the control instructions of the aggregator. In the second decision stage, the residential aggregator generates the scheduling plan of each home appliance through a three-step decomposition algorithm to follow the aggregated power scheduling defined in the first decision stage. First, the decomposition algorithm collects the status information of the home appliances, including SoC, individual user behavior, and power range. Second, the algorithm generates individual power instructions for each home appliance. Finally, each control power value is sent to the corresponding home appliance unit.

2. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 1, characterized in that, The unified model for generalized energy storage in step (1) is as follows: ， ， ， ， Where, p i Indicates the power consumption of appliance i; e i p represents the state of charge of electrical appliance i; i and e i θ represents the state parameters of the electrical appliance. i η represents the dissipation rate of appliance i; i σ represents the conversion factor of appliance i; i θ represents the influence factor of ambient temperature on appliance i; i η i and σ i For electrical equipment parameters; p i,min and p i,max δ represents the lower and upper bounds of the power consumption of device i, respectively; i,max e represents the ramp-up limit of appliance i; i,min and e i,max These represent the lower and upper bounds of the indoor temperature of appliance i, respectively.

3. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 2, characterized in that, In step (1), the equivalent parameters of the aggregator model used for upper-level scheduling are obtained by calculating the weighted average of the parameters of each household appliance, specifically expressed as follows: ， ， ， ， in, and These are the power consumption and remaining energy of aggregator M, respectively; and These are the lower and upper limits of the power consumption of aggregator M; and These are the lower and upper limits of the remaining energy of the aggregator M; , and Let M represent the dissipation rate, conversion coefficient, and ambient temperature influence factor, respectively. The weighted average of the aggregator model is calculated by directly summing the power and energy boundary parameters, and weighting the dissipation rate, conversion coefficient, and temperature influence factor according to the rated power of each device.

4. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 1, characterized in that, In step (1), the aggregator model is constructed considering the operating costs of residential appliances, noise costs, and power point tracking errors, which are expressed as follows: ， ， ， in, , and These represent the total operating cost, total noise cost, and power point tracking error of aggregator M, respectively. and This is an approximation of the operating cost coefficient of aggregator M. It is an approximation of the noise figure of the aggregator M. For electricity demand; The local power load, including rooftop photovoltaic units and other non-storable loads, operating costs, noise costs, and power point tracking errors, are used to construct the reward function for the deep reinforcement learning controller in the first decision-making stage.

5. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 1, characterized in that, The decision-making method in step (2) is based on the Markov decision process, which is defined by variables {S, A, P, R, γ}, where S represents the set of environmental states observed by the deep reinforcement learning controller, A represents the set of actions, P represents the transition probability from any state s∈S to any state s'∈S, R is the set of immediate rewards for any action a∈A, and γ∈[0,1] represents the discount rate used to penalize future rewards. The Markov decision process is solved using the Soft Actor-Critic algorithm, which uses the maximum entropy reinforcement learning framework. The Soft Actor-Critic framework completes the decision-making process by executing the following steps: (2-1) State observation: The controller observes the current state s of the aggregator from the environment. M,t (2-2) Action generation: Generate state s M,t Input to online action network And by the online action network Determine the action of the aggregator, i.e., the power output rate a. M,t (2-3) Environmental interaction and reward calculation: Aggregator according to action a M,t Output its power P M (t), the environment transitions to the next state s. M,t+1 Simultaneously calculate the instant reward r t (2-4) Experience storage: storing the quadruple (s M,t , a M,t , r M,t , s M,t+1 Store in the experience replay buffer D i In the middle, it is used to train the deep reinforcement learning agent; (2-5) Network update: from buffer D i Randomly sample small batches of data and update them sequentially: Soft Commentator Network q θ Online Action Network Entropy regularization parameter α, and target action network Repeat steps (2-1)-(2-5) above until the algorithm converges, thus completing the first stage of determining the optimal power at the system level for the aggregator.

6. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 5, characterized in that, Deep reinforcement learning controllers based on local observers M,t Solve operational problems. ， in, and These are the power consumption and remaining energy of aggregator M, respectively; and These are the lower and upper limits of the power consumption of aggregator M; It is a power requirement; It is the discrepancy between power demand and actual power consumption; Action a M,t ∈[0,1] is defined as power output rate, power consumption P M (t) is naturally restricted to [P] M,min , P M,max Within the range of , the joint action at time step t is represented as a t = (a 1,t , a 2,t ,…, a N,t ), P M The formula for calculating (t) is: ； Among them, P M (t) and E M (t) represents the power consumption and remaining energy of aggregator M, respectively; P M,min (t) and P M,max (t) represents the lower and upper limits of the power consumption of aggregator M; P D (t) represents the power demand; It is the discrepancy between power demand and actual power consumption. System status can be obtained from s t Transfer to s t+1 The probability is P(s) t , s t+1 ) = Pr(s t+1 | s t , a t When the system state changes from s t Transfer to s t+1 At that time, the aggregator's controller will receive a reward r t : ， The reward function consists of three parts: tracking error cost, operating cost, and acoustic discomfort cost. , and These are the weighting coefficients for tracking error cost, operating cost, and acoustic discomfort cost, respectively. The goal of the aggregator deep reinforcement learning controller is to maximize the expected reward value at T time steps, and its scaling function J is expressed as: ， Where π is the generation action. Control strategy, based on state The discount rate γ determines the impact of future rewards on current rewards. It is a state The entropy of the next strategy π, where α is a temperature parameter, determines the importance of entropy relative to the reward.

7. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 1, characterized in that, In step (3), the three-step decomposition algorithm first allocates the initial control signal for storable load power proportionally according to the device power range: ， in, It is the power control signal for home appliance i. and These are the upper and lower power limits for appliance i, respectively. The three-step decomposition algorithm considers the individual user behavior of electric vehicles as their charging needs, and the power command allocated to the electric vehicle is expressed as: ， ， in, It is the power control signal for electric vehicle i. The charging power for electric vehicles is given by α and β, which are shape parameters, and a1 and a2, which are variables related to the arrival and departure times of the vehicle.

8. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 7, characterized in that, To meet the charging needs of electric vehicles, aggregators require additional power, which is supplemented by other storage-capable appliances within the same aggregator in proportion to their power range, as shown below: ， ， in, This indicates the additional power required by aggregator M to meet the dynamic user behavior of electric vehicles. For the control signal corresponding to the additional power of the storable device i, agg M Let M be the aggregator.

9. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 8, characterized in that, In step (3), additional power is allocated through an iterative process, while ensuring feasibility and meeting SoC / temperature boundaries. The iteration rule is as follows: Initial state and initial additional power The initial power range is , The initial power allocated to each individual load is calculated as follows: 。 10. The optimal energy management method for residential aggregators based on deep reinforcement learning according to claim 9, characterized in that, The iteration number is named iter. In each iteration step, for each storable load, if p i,max >0, then: ， Among them, for or When the predicted load reaches the power and SoC boundary, known as saturated load, the power range of saturated load is set to 0 to avoid allocating power to them: ， ， Minimum aggregator M Calculate the additional power: ， ， If all aggregators satisfy If the extra power has been fully allocated, the algorithm's output is sent to each unit in step (3); otherwise, iter = iter+1, and the iteration continues.

Images