Household energy management optimization method and device, electronic equipment and medium

Abstract

The invention discloses a household energy management optimization method and device, electronic equipment and a medium, and belongs to the field of household energy management. The method comprises the following steps: modeling loads with different electrical characteristics in the household energy management system; according to the established model, converting a household energy management problem into a Markov decision process; according to a Markov decision process, a hybrid motion space algorithm is used to carry out energy optimization scheduling on the household energy management system so as to reduce the operation cost and the power consumption of the household energy system. Wherein the hybrid action space algorithm integrates a DQN algorithm and a DDPG algorithm. According to the method, the MDRL algorithm combining the DQN and the DDPG is adopted, the hybrid action space problem in the HEMS can be solved, the hybrid action space problem comprises combination of discrete actions and continuous actions, and higher flexibility and adaptability are achieved. In addition, an accurate discretization action processing method is adopted, and the problem that the complexity of an action space is increased in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the field of home energy management, and in particular to a home energy management optimization method, device, electronic equipment and medium. Background Art

[0002] With the increase of smart devices in the home, more and more residents are able to participate in demand response management through the home energy management system (HEMS). In order to formulate the most effective energy management strategy, a deep reinforcement learning (DRL)-based algorithm was developed to optimize the operation of a home energy system consisting of a photovoltaic (PV) power generation system, a battery energy storage system (ESS), an electric vehicle (EV), and household appliances.

[0003] Traditional model-based methods require the establishment of numerical models to characterize home energy systems and formulate optimization problems considering objectives and system constraints. By solving the optimization problem, the operation of the home energy system is optimized. The main challenges it faces are the modeling accuracy of the energy system and the prediction accuracy of unknown variables. Reinforcement learning (RL)-based methods do not require prior knowledge of the energy system, and the intelligent agent learns the optimal decision-making strategy through interaction with the energy system. This feature can solve the problems of the difficulty in modeling the dynamic process of complex energy systems composed of unknown variables, as well as the uncertainty of renewable energy generation and electricity prices. However, current DRL-based methods, such as DQN and DDPG, mainly consider discrete action spaces or continuous action spaces, while the discrete-continuous mixed action space problem of home energy systems poses a challenge to existing DRL algorithms. Discrete action spaces usually include the "on / off" operation mode of household appliances such as washing machines and dishwashers, while continuous action spaces are usually used to control the power of HVAC systems and energy storage systems. One way to deal with discrete-continuous mixed action spaces is to discretize continuous actions so that RL algorithms based on discrete action spaces can be applied. Some studies have used discrete action spaces to model the continuous operation modes of air conditioners and electric vehicles and used the DQN algorithm for optimal scheduling. However, the study found that the granularity of the discretization of the continuous action space has a significant impact on the performance of DQN. The prior art proposed a residential multi-energy system management strategy based on DDPG, which considers the discrete-continuous hybrid action space. First, discrete actions are treated as continuous actions, and then the continuous actions output by the actor network are discretized to obtain discrete actions. However, treating discrete actions as continuous actions will significantly increase the complexity of the action space. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a method, device, electronic device and medium for optimizing home energy management.

[0005] The first technical solution adopted by the present invention is:

[0006] A method for optimizing household energy management comprises the following steps:

[0007] Modeling loads with different electrical characteristics in home energy management systems;

[0008] According to the established model, the household energy management problem is transformed into a Markov decision process (MDP) that does not require the prediction of unknown variables or their dynamic models;

[0009] According to the Markov decision process, the hybrid action space algorithm (MDRL) is used to optimize the energy scheduling of the home energy management system to reduce the operating cost and power consumption of the home energy system;

[0010] The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

[0011] Furthermore, the loads with different electrical characteristics include: uncontrollable loads, time-controllable loads, power-controllable loads, battery energy storage systems, electric vehicles, and photovoltaic power generation systems.

[0012] Furthermore, various loads are modeled as follows:

[0013] 1) Uncontrollable load NSL

[0014] The power demand of the uncontrollable load NSL is fixed every hour during operation, and the expression of its energy consumption is as follows:

[0015]

[0016] In the formula, e i represents the energy consumption of the ith NSL at time step t, T i work represents the working time of the i-th NSL;

[0017] 2) Time Controllable Load TSL

[0018] The time-controlled load TSL can adjust its operating time within a certain time range while maintaining a fixed power demand. The expression of its energy consumption is as follows:

[0019]

[0020] In the formula, e j represents the energy consumption of the jth TSL at time step t, represents the working time of the jth TSL. If t is not within the working time, the energy consumption is 0;

[0021] 3) Power controllable load PSL

[0022] The power controllable load PSL has a fixed usage time, but its power can be adjusted according to the electricity price and demand. The expression of its energy consumption is as follows:

[0023]

[0024] In the formula, e k represents the energy consumption of the kth PSL at time step t, Indicates its power ratio;

[0025] 4) Battery Energy Storage System ESS

[0026] The battery energy storage system ESS is used as a flexible load to maintain the balance between supply and demand. The expression of its state of charge SOC is as follows:

[0027]

[0028] st ch β dch =0, SOC t ∈[0.2,1]

[0029] Where η ch With η dch They represent the charge and discharge efficiency of ESS, C EES represents the capacity of ESS, e ch and e dch Respectively represent the maximum value of charge and discharge per hour; β ch and β dch is the charge and discharge coefficient;

[0030] 5) Electric Vehicles (EV)

[0031] The relationship between the state of charge of the electric vehicle EV at time step t and its power and the previous time step is expressed as follows:

[0032]

[0033] In the formula, and They represent the charging and discharging efficiency of EV, C EV represents the capacity of EV, e EV,ch and e EV,dch Respectively represent the maximum charge and discharge capacity per hour; and They represent the charge and discharge coefficients of electric vehicles respectively; represents the battery status of the electric vehicle at time t, Indicates the minimum battery state of an electric vehicle, Indicates the maximum charging state of an electric vehicle;

[0034] 6) Photovoltaic power generation system

[0035] Using deep learning algorithms to predict photovoltaic power generation, photovoltaic power generation From 0 to the predicted value Changes between, as shown below:

[0036]

[0037] Furthermore, the home energy management problem is converted into a Markov decision process, including:

[0038] The Markov decision process MDP consists of a 5-tuple (S, A, P, R, γ), where S represents the finite state space consisting of the environment state; A is the action space, which represents the actions that can be taken in the current state; P is the state transition probability function, which represents the state in state s. t Next action is taken from state s t Transition to next state s t+1 The probability of ; R is the reward function that provides immediate rewards during state transitions; γ is the discount factor used to balance the importance of immediate rewards with future rewards;

[0039] The MDP of the home energy management system is as follows:

[0040] 1) Status s t The expression is as follows:

[0041]

[0042] In the formula, hour represents hourly information, day represents daily information, represents the solar power generation, λ t Indicates the electricity price, Indicates the SOC of EES, Indicates the SOC of the EV;

[0043] 2) Action a t The expression is as follows:

[0044]

[0045] In the formula, Indicates the charge-discharge ratio of the maximum power of the battery energy storage system and electric vehicle. is a binary variable 0 / 1, corresponding to the on / off state of the appliance, It represents the power ratio of two power controllable loads PSL;

[0046] 3) Rewards:

[0047]

[0048] In the formula, C pen Indicates the penalty for violating the constraint; represents the energy consumption of uncontrollable loads, represents the energy consumption of the time-controllable load, represents the energy consumption of the power controllable load, represents the energy consumption of electric vehicles, Represents the energy consumption of the battery energy storage system.

[0049] Furthermore, the hybrid action space algorithm includes an actor network and a critic network; the critic network Q(s,a d ,a c ; θ) is used to approximate the state-action value function, and the actor network To produce continuous action Among them, θ and are the corresponding network parameters including weights and biases; a d represents a discrete action, a c Indicates continuous action;

[0050] where the squared loss L in is minimized by using the gradient descent method θ To optimize the critic network parameters θ:

[0051]

[0052] In the formula, E represents, Q represents Bellman equation; s t Indicates the state, a d,t represents a discrete action at time t, a c,t represents a continuous action at time t, y t Expressed as target Q value;

[0053] By gradient Adjustment in direction To improve the Q value and optimize the actor network parameters;

[0054] The target network is introduced into the actor network and the critic network to evaluate the target Q value, which are and Q'(s,a d ,a c; θ'); The parameters of the target network use a soft update strategy as follows:

[0055]

[0056] In the formula, τ is the adjustment parameter.

[0057] Furthermore, the hybrid action space algorithm is trained in the following manner:

[0058] 1) Use random weights θ and Initialize the actor network and critic network;

[0059] 2) By copying θ'←θ and Initialize the target network;

[0060] 3) Initialize buffer M;

[0061] 4) For e = 1: E;

[0062] 5) Obtain the initial state s0 from a random day through a random SoC;

[0063] 6) For t = 0:23 do;

[0064] 7) Use the exploration strategy to select action a t ={a d,t ,a c,t};

[0065] 8) Execute action a t , observe the reward r t+1 , and transfer to the next state s t+1 ;

[0066] 9)Storing tuples(s) t ,a t ,s t+1 ,r t+1 ) to M;

[0067] 10) Sample from M (s) b ,a b ,s b+1 ,r b+1 )for b=1,2,...,B;

[0068] 11) Get the target Q value:

[0069] 12) By minimizing losses To update θ;

[0070] 13) Update by sampling strategy gradient

[0071]

[0072] 14) Target network soft update;

[0073] 15) End for;

[0074] 16) End for.

[0075] Furthermore, the expression of the exploration strategy is as follows:

[0076]

[0077] In the formula, A d represents the discrete action space, A c represents the continuous action space, Ν(0,δ 2 I) is Gaussian noise, and ε is the exploration probability.

[0078] The second technical solution adopted by the present invention is:

[0079] A household energy management optimization device, comprising:

[0080] HEMS modeling module, used to model loads with different electrical characteristics in home energy management systems;

[0081] A problem conversion module is used to convert the home energy management problem into a Markov decision process (MDP) that does not require the prediction of unknown variables or its dynamic model according to the established model;

[0082] An algorithm operation module is used to optimize the energy scheduling of the home energy management system based on the Markov decision process using a hybrid action space algorithm (MDRL) to reduce the operating cost and power consumption of the home energy system;

[0083] The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

[0084] The third technical solution adopted by the present invention is:

[0085] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a home energy management optimization method as described above.

[0086] The fourth technical solution adopted by the present invention is:

[0087] A computer-readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a home energy management optimization method as described above.

[0088] The fifth technical solution adopted by the present invention is:

[0089] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method.

[0090] The beneficial effects of the present invention are as follows: the present invention can handle the mixed action space problem in HEMS, including the combination of discrete and continuous actions, by adopting the MDRL algorithm combining DQN and DDPG, and has higher flexibility and adaptability. In addition, the present invention adopts an accurate discretization action processing method, which can solve the problem of increased complexity of action space in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0092] Figure 1 is a schematic diagram of the structure of a home energy management system according to an embodiment of the present invention;

[0093] Figure 2 is a schematic diagram of the network structure of the MDRL algorithm in an embodiment of the present invention;

[0094] Figure 3 It is a flowchart of the steps of a household energy management optimization method in an embodiment of the present invention. DETAILED DESCRIPTION

[0095] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0096] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0097] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0098] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0099] In order to solve the problem that treating discrete actions as continuous actions will lead to increased complexity of the action space, a reinforcement learning-based method with the ability to simultaneously process discrete-continuous mixed action spaces is needed. This method can provide a more flexible, precise, efficient and comprehensive decision-making framework, which is suitable for practical application fields such as complex energy systems and has important value for home energy management.

[0100] Based on this, the scheme of the present invention adopts an RL-based algorithm, and through interactive learning with HEMS, the optimal decision-making strategy can be adapted to the dynamically changing energy environment and unknown variables without prior knowledge, and can cope with the management problems of complex energy systems of various household appliances and energy equipment, so as to achieve efficient utilization of energy and cost reduction. Secondly, this scheme adopts the MDRL algorithm combining DQN and DDPG, which can handle the mixed action space problems in HEMS, including the combination of discrete and continuous actions, and has higher flexibility and adaptability. Finally, the scheme of the present invention adopts a precise discretized action processing method, which can solve the problem of increased complexity of the action space in the prior art. Through precise discretization processing, more accurate and controllable action selection can be provided, thereby achieving more optimized operation and management of home energy systems. In summary, the scheme of the present invention can handle mixed action spaces, adopt precise discretized action processing methods, and achieve optimal decision-making strategies through interactive learning, thereby optimizing the operation of complex energy systems.

[0101] Example 1

[0102] like Figure 3 As shown, this embodiment provides a method for optimizing home energy management, comprising the following steps:

[0103] S1. Modeling loads with different electrical characteristics in home energy management systems (HEMS);

[0104] S2. According to the established model, the household energy management problem is transformed into a Markov decision process MDP that does not require the prediction of unknown variables or its dynamic model;

[0105] S3. According to the Markov decision process, the hybrid action space algorithm (MDRL) is used to optimize the energy scheduling of the home energy management system to reduce the operating cost and power consumption of the home energy system.

[0106] The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

[0107] As an optional implementation, Figure 1As shown, HEMS includes electric vehicles EV, battery energy storage systems ESS and photovoltaic PV systems and various household appliances, such as televisions, air conditioners, washing machines, dishwashers, etc. Household appliances are divided into non-schedulable loads (NSL; i.e. non-controllable appliances, such as refrigerators, televisions, computers, etc.), time-schedulable loads (TSL; i.e. time-controllable appliances, such as washing machines, dishwashers, dryers, etc.), and power-schedulable loads (PSL; i.e. power-controllable appliances, such as air conditioners, etc.) according to their characteristics. In this embodiment, each load in HEMS is first modeled, and then the home energy management problem is converted into a Markov decision process (MDP) that does not require the prediction of unknown variables or their dynamic models. Finally, the hybrid action space algorithm MDRL is used to optimize the energy scheduling of HEMS to reduce the operating cost and power consumption of the home energy system.

[0108] The above method is explained in detail below with reference to the accompanying drawings and specific embodiments.

[0109] (1) Building a HEMS model

[0110] 1.1) Uncontrollable Load (NSL)

[0111] Uncontrollable loads NSL usually refer to loads such as lights (Light, L), televisions (Television, TV), smoke alarms, and electronic equipment chargers. Unlike other loads, the hourly power demand of NSL during operation is fixed, and its energy consumption is shown in formula (1):

[0112]

[0113] In the formula, e i represents the energy consumption of the ith NSL at time step t, T i work Represents the working time of the i-th NSL.

[0114] 1.2) Time Controllable Load TSL

[0115] Compared with NSL, time-controlled load TSL (mainly referring to water heater (Heater, H) and washing machine (Washing machine, WM)) can adjust its running time within a certain time range while maintaining a fixed power demand. Its energy consumption is shown in formula (2):

[0116]

[0117] In the formula, ej represents the energy consumption of the jth TSL at time step t, represents the working time of the jth TSL. If t is not within the working time, the energy consumption is 0.

[0118] 1.3) Power Controllable Load PSL

[0119] Power controllable load PSL has a fixed usage time, but its power can be adjusted according to electricity price and demand. Typical PSL devices refer to high-power devices, such as air conditioners (AC) and microwave ovens (MW), and their energy consumption is shown in formula (3):

[0120]

[0121] In the formula, e k represents the energy consumption of the kth PSL at time step t, Indicates its power ratio.

[0122] 1.4) Battery Energy Storage System ESS

[0123] The battery energy storage system ESS is used as a flexible load to maintain the balance between supply and demand. The expression of its state of charge SOC is as follows:

[0124]

[0125] st

[0126] β ch β dch =0 (5)

[0127] SOC t ∈[0.2,1](6)

[0128] Where η ch With η dch They represent the charging and discharging efficiency of ESS, C EES represents the capacity of ESS, e ch and e dch They represent the maximum value of charge and discharge per hour respectively.

[0129] 1.5) Electric Vehicles (EV)

[0130] Similar to ESS, the state of charge of the electric vehicle EV at time step t and its power in relation to the previous time step are expressed as follows:

[0131]

[0132] In the formula, and They represent the charging and discharging efficiency of EV, C EV represents the capacity of EV, e EV,ch and e EV,dch Respectively represent the maximum charge and discharge capacity per hour.

[0133] 1.6) Photovoltaic power generation system

[0134] Using deep learning algorithms to predict photovoltaic power generation, photovoltaic power generation From 0 to the predicted value Changes between, as shown below:

[0135]

[0136] 1.7) Family energy balance

[0137] In order to ensure the coordinated and stable operation of HEMS and the power grid, it is necessary to maintain the balance between supply and demand at each time step t. and Respectively represent the energy exchange with the main grid and solar energy:

[0138]

[0139] (2) Modeling the home energy management problem as an MDP

[0140] DRL problems are usually modeled as MDPs. MDPs consist of a 5-tuple (S, A, P, R, γ), where S represents a finite state space consisting of environmental states; A is the action space, which represents the actions that can be taken in the current state; P∶S×A×S→[0,1] is the state transition probability function, which represents the state of the action in state s. t Next action is taken from state s t Transition to next state s t+1 The probability of ; R is the reward function that provides immediate rewards during state transitions; the discount factor γ∈[0,1] is used to balance the importance of immediate rewards and future rewards. The MDP details of HEMS are as follows.

[0141] 1) State: Construct a 6-dimensional state vector containing daily information, solar power generation, electricity price, SOC of EES and EV. Use standard normalization to improve training efficiency. The state vector is as follows:

[0142]

[0143] 2) Action: Since NSL must always be satisfied during its operation, the action vector controls EES, EV, TSL and PSL. The action control switch state of TSL is a discrete variable, and the other action vectors are continuous variables that control the power ratio, as shown below:

[0144]

[0145] Where: Indicates the charge-discharge ratio of its maximum power, It is a binary variable 0 / 1, corresponding to the "on / off" state of the appliance. Represents the power ratio of two PSLs.

[0146] 3) Rewards:

[0147]

[0148] Where: C pen It represents the penalty for violating the constraint, such as SOC constraint, and the satisfaction of TSL and PSL load usage, as shown below:

[0149] C pen =C ESS +C EV +C TSL (15)

[0150] Each term on the right side of formula (15) is a scaled value between [0, 1], indicating that the SOC constraint is violated or the energy demand of the TSL device is not met on that day.

[0151] (3) MDRL algorithm

[0152] For existing DRL algorithms, most of them require the interaction space to be discrete or continuous. For example, DQN and its variants are suitable for discrete action spaces, while DDPG is widely used in continuous action spaces. In order to handle discrete-continuous mixed action spaces, an MDRL algorithm that integrates DQN and DDPG is considered.

[0153] Use a d ∈A d and a c ∈A c represents discrete actions and continuous actions, where A d and A c denote the discrete action space and the continuous action space respectively, so the discrete-continuous mixed action space can be expressed as a={a d ,a c}. Then the Bellman equation is:

[0154]

[0155] Among them, a d,t and a c,t are discrete and continuous actions at time t respectively; a' d and a' care the discrete and continuous actions to be performed in the next moment, respectively. If it holds, then formula (16) can be written as:

[0156]

[0157] The right side of formula (17) processes continuous actions through the actor-critic network. Similar to DDPG, the deep critic network Q(s,a d ,a c ; θ) is used to approximate the state-action value function, while the deterministic deep policy network To produce continuous action where θ and are the corresponding network parameters including weights and biases.

[0158] The network structure diagram of the MDRL algorithm proposed in the embodiment of the present invention is as follows: Figure 2 By searching the discrete action space, the optimal discrete action is obtained, that is, The selection of the discrete action corresponding to the highest Q value is the same as that of DQN. Therefore, the MDRL algorithm proposed in the embodiment of the present invention inherits the advantages of both DDPG and DQN.

[0159] Similar to DDPG, we minimize the squared loss L in using gradient descent θ To optimize the critic network parameters θ:

[0160]

[0161] in, is the target Q value. To optimize the actor network parameters, the basic idea is to Adjustment in direction To improve the Q value. Applying the chain rule, the gradient can be decomposed into the gradient of the state-action value function relative to the continuous action and the gradient of the strategy relative to the strategy parameters, and the state distribution ρ is considered. μ Policy gradient for updating the policy parameters of (s)

[0162]

[0163] In DRL, the balance between exploration and exploitation is the key to training efficient decision-making agents. In order to facilitate the training of deep networks considering discrete-continuous mixed action spaces, the embodiment of the present invention proposes a special exploration strategy, which combines the ε-greedy strategy in DQN with the DDPG strategy to add Gaussian noise N(0,δ 2 I) strategies combined.

[0164]

[0165] In order to solve the challenges of sample time correlation in DRL to network optimization, experience replay is considered. t ,a t ,s t+1 ,r t+1 ) is stored in a playback buffer M of size m. When the buffer is full, the tuple with the longest storage time is deleted. At each time step, a small batch of tuples is uniformly sampled for network update. In order to stabilize the learning process, a target network is introduced in the actor network and the critic network to evaluate the target Q value, which are and Q'(s,a d ,a c ; θ'). The parameters of the target network use a soft update strategy as follows:

[0166]

[0167] Here, τ<<1 ensures that the target network parameters change slowly, thereby improving the stability of the learning process.

[0168] See Table 1 below. Algorithm 1 describes the training process of the network, including the initialization of the network and the main loop of the training process. In the main loop, one day is an event. At each time step of the day, the agent receives the state s t And select action a according to the exploration strategy in formula (20) t After executing the action, the state is transferred to s t+1 And get reward t+1 Then the tuple (s t ,a t ,s t+1 ,r t+1 ) is stored in the replay buffer, and finally, based on the sampled mean square loss and policy gradient, the B tuples uniformly sampled from the replay buffer are used to update θ and

[0169] Table 1

[0170]

[0171]

[0172] In summary, the existing RL-based methods mainly consider discrete action space or continuous action space, while the home energy system usually contains a variety of household appliances and energy equipment, and its action space is a mixture of discrete and continuous actions. The method for dealing with the discrete-continuous mixed action space problem in home energy is to discretize the continuous action or to make the discrete action continuous. Although this processing method can optimize the home energy management problem to a certain extent, the study found that the granularity of the discretization of the continuous action space has a significant impact on the performance of DQN, and if the discrete action is treated as a continuous action, the complexity of the action space will increase significantly. The present invention uses the hybrid action space algorithm MDRL to interact with the system to learn the optimal decision strategy for HEMS composed of PV, ESS, EV and a variety of household appliances, without prior knowledge, and better adapt to the dynamically changing energy environment and unknown variables, thereby achieving more optimized and sustainable home energy management. The present invention can solve the problem that the dynamic process of complex energy systems is difficult to model and the uncertainty of new energy generation, electricity prices, etc. At the same time, the method has outstanding advantages in processing discrete-continuous mixed action space, providing accurate discretized action processing and achieving comprehensive performance optimization, and is of great value to home energy management.

[0173] Example 2

[0174] This embodiment provides a home energy management optimization device, including:

[0175] HEMS modeling module, used to model loads with different electrical characteristics in home energy management systems;

[0176] A problem conversion module is used to convert the home energy management problem into a Markov decision process (MDP) that does not require the prediction of unknown variables or its dynamic model according to the established model;

[0177] An algorithm operation module is used to optimize the energy scheduling of the home energy management system based on the Markov decision process using a hybrid action space algorithm (MDRL) to reduce the operating cost and power consumption of the home energy system;

[0178] The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

[0179] Since the device is a home energy management optimization device of an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0180] Example 3

[0181] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 3 A method for optimizing home energy management is shown.

[0182] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0183] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0184] Since the electronic device is an electronic device corresponding to a home energy management optimization method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0185] Example 4

[0186] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 3 A method for optimizing home energy management is shown.

[0187] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0188] Since the storage medium is a storage medium corresponding to a home energy management optimization method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0189] Example 5

[0190] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a home energy management optimization method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0191] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0192] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0193] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for optimizing household energy management, characterized in that: The following steps are involved: Modeling loads with different electrical characteristics in home energy management systems; According to the established model, the household energy management problem is transformed into a Markov decision process; According to the Markov decision process, a hybrid action space algorithm is used to optimize the energy scheduling of the home energy management system to reduce the operating cost and power consumption of the home energy system; The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

2. A household energy management optimization method according to claim 1, characterized in that: The loads with different electrical characteristics include: uncontrollable loads, time-controllable loads, power-controllable loads, battery energy storage systems, electric vehicles, and photovoltaic power generation systems.

3. A household energy management optimization method according to claim 2, characterized in that: The various loads are modeled as follows: 1) Uncontrollable load NSL The power demand of the uncontrollable load NSL is fixed every hour during operation, and the expression of its energy consumption is as follows: In the formula, e i represents the energy consumption of the ith NSL at time step t, T i work represents the working time of the i-th NSL; 2) Time Controllable Load TSL The time-controlled load TSL can adjust its operating time within a certain time range while maintaining a fixed power demand. The expression of its energy consumption is as follows: In the formula, e j represents the energy consumption of the jth TSL at time step t, represents the working time of the jth TSL. If t is not within the working time, the energy consumption is 0; 3) Power controllable load PSL The power controllable load PSL has a fixed usage time, but its power can be adjusted according to the electricity price and demand. The expression of its energy consumption is as follows: In the formula, e k represents the energy consumption of the kth PSL at time step t, Indicates its power ratio; 4) Battery Energy Storage System ESS The battery energy storage system ESS is used to maintain the balance between supply and demand. The expression of its state of charge SOC is as follows: stβ ch β dch =0, SOC t ∈[0.2,1] Where η ch With η dch They represent the charge and discharge efficiency of ESS, C EES represents the capacity of ESS, e ch and e dch Respectively represent the maximum value of charge and discharge per hour; β ch and β dch is the charge and discharge coefficient; 5) Electric Vehicles (EV) The relationship between the state of charge of the electric vehicle EV at time step t and its power and the previous time step is expressed as follows: In the formula, and They represent the charging and discharging efficiency of EV, C EV represents the capacity of EV, e EV,ch and e EV,dch Respectively represent the maximum charge and discharge capacity per hour; and They represent the charge and discharge coefficients of electric vehicles respectively; represents the battery status of the electric vehicle at time t, Indicates the minimum battery state of an electric vehicle, Indicates the maximum charging state of an electric vehicle; 6) Photovoltaic power generation system Using deep learning algorithms to predict photovoltaic power generation, photovoltaic power generation From 0 to the predicted value Changes between, as shown below:

4. A method for optimizing household energy management according to claim 1, characterized in that: The home energy management problem is transformed into a Markov decision process, including: The Markov decision process MDP consists of a 5-tuple (S, A, P, R, γ), where S represents the finite state space consisting of the environment state; A is the action space, which represents the actions that can be taken in the current state; P is the state transition probability function, which represents the state in state s. t Next action is taken from state s t Transition to next state s t+1 The probability of ; R is the reward function that provides immediate rewards during state transitions; γ is the discount factor used to balance the importance of immediate rewards and future rewards; The MDP of the home energy management system is as follows: 1) Status s t The expression is as follows: In the formula, hour represents hourly information, day represents daily information, represents the solar power generation, λ t Indicates the electricity price, Indicates the SOC of EES, Indicates the SOC of the EV; 2) Action a t The expression is as follows: In the formula, Indicates the charge-discharge ratio of the maximum power of the battery energy storage system and electric vehicle. is a binary variable 0 / 1, corresponding to the on / off state of the appliance, It represents the power ratio of two power controllable loads PSL; 3) Rewards: In the formula, C pen Indicates the penalty for violating the constraint; represents the energy consumption of uncontrollable loads, represents the energy consumption of the time-controllable load, represents the energy consumption of the power controllable load, represents the energy consumption of electric vehicles, Represents the energy consumption of the battery energy storage system.

5. A household energy management optimization method according to claim 1, characterized in that: The hybrid action space algorithm includes an actor network and a critic network; the critic network Q(s,a d ,a c ; θ) is used to approximate the state-action value function, and the actor network To produce continuous action Among them, θ and are the corresponding network parameters including weights and biases; a d represents a discrete action, a c Indicates continuous action; where the squared loss L in is minimized by using the gradient descent method θ To optimize the critic network parameters θ: In the formula, E represents, Q represents Bellman equation; s t Indicates the state, a d,t represents a discrete action at time t, a c,t represents a continuous action at time t, y t Expressed as target Q value; By gradient Adjustment in direction To improve the Q value and optimize the actor network parameters; The target network is introduced into the actor network and the critic network to evaluate the target Q value, which are and Q'(s,a d ,a c ; θ'); The parameters of the target network use a soft update strategy as follows: In the formula, τ is the adjustment parameter.

6. A method for optimizing household energy management according to claim 5, characterized in that: The hybrid action space algorithm is trained in the following way: 1) Use random weights θ and Initialize the actor network and critic network; 2) By copying θ'←θ and Initialize the target network; 3) Initialize buffer M; 4) For e = 1: E; 5) Obtain the initial state s0 from a random day through a random SoC; 6) For t = 0:23 do; 7) Use exploration strategy to select action a t ={a d,t ,a c,t }; 8) Execute action a t , observe the reward r t+1 , and transfer to the next state s t+1 ; 9)Storing tuples(s) t ,a t ,s t+1 ,r t+1 ) to M; 10) Sample from M (s) b ,a b ,s b+1 ,r b+1 )for b=1,2,...,B; 11) Get the target Q value: 12) By minimizing losses To update θ; 13) Update by sampling strategy gradient 14) Target network soft update; 15) End for; 16) End for.

7. A method for optimizing household energy management according to claim 6, characterized in that: The expression of the exploration strategy is as follows: In the formula, A d represents the discrete action space, A c represents the continuous action space, Ν(0,δ 2 I) is Gaussian noise, and ε is the exploration probability.

8. A household energy management optimization device, characterized in that: include: HEMS modeling module, used to model loads with different electrical characteristics in home energy management systems; The problem conversion module is used to convert the home energy management problem into a Markov decision process according to the established model; the algorithm operation module is used to optimize the energy scheduling of the home energy management system by using the hybrid action space algorithm according to the Markov decision process, so as to reduce the operating cost and power consumption of the home energy system; The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images