A household energy management optimization method and device, electronic equipment and medium

Abstract

The application discloses a kind of family energy management optimization method, device, electronic equipment and medium, belong to family energy management field.Therein method includes: the load with different electrical characteristics in family energy management system is modeled;According to the model established, family energy management problem is converted into Markov decision process;According to Markov decision process, the energy optimization scheduling of family energy management system is carried out using mixed action space algorithm, to reduce the operating cost and power consumption of family energy system;Wherein mixed action space algorithm integrates DQN algorithm and DDPG algorithm.The application can process the mixed action space problem in HEMS by using the MDRL algorithm combining DQN and DDPG, including the combination of discrete and continuous actions, with higher flexibility and adaptability.In addition, the application uses the precise discretization action processing method, which can solve the problem of increasing complexity of action space in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of home energy management, and in particular to a home energy management optimization method and device, electronic equipment and medium. BACKGROUND

[0002] With the increasing number of smart home devices, more and more residents can participate in demand response management through a home energy management system (HEMS). To develop the most effective energy management strategy, an algorithm based on deep reinforcement learning (DRL) is developed to optimize the operation of a home energy system composed of a photovoltaic (PV) power generation system, an energy storage system (ESS), an electric vehicle (EV), and household appliances.

[0003] Traditional model-based methods need to establish a numerical model to represent the home energy system, and formulate an optimization problem considering the target and system constraints. By solving the optimization problem, the operation of the home energy system is optimized. The main challenge it faces is the modeling accuracy of the energy system and the prediction accuracy of unknown variables. Reinforcement learning (RL) based methods do not require prior knowledge of the energy system, and the agent learns the optimal decision strategy through interaction with the energy system. This feature can solve the problem of complex energy system dynamic process composed of unknown variables and the uncertainty of new energy generation and electricity price. However, current DRL-based methods such as DQN and DDPG mainly consider discrete action space or continuous action space, while the discrete-continuous hybrid action space problem of the home energy system poses a challenge to existing DRL algorithms. Discrete action space usually includes "on / off" operation modes of household appliances such as washing machines and dishwashers, while continuous action space is usually used to control the power of HVAC systems and energy storage systems. One way to handle discrete-continuous hybrid action space is to discretize continuous actions so that RL algorithms based on discrete action space can be applied. Some research uses discrete action space to model the continuous operation mode of air conditioners and electric vehicles and uses DQN algorithm for optimal scheduling. However, research has found that the granularity of continuous action space discretization has a significant impact on the performance of DQN. Existing technology proposes a DDPG-based residential multi-energy system management strategy that considers discrete-continuous hybrid action space. First, the discrete action is treated as a continuous action, and second, the continuous action output by the actor network is discretized to obtain a discrete action. However, treating discrete actions as continuous actions will significantly increase the complexity of the action space. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a household energy management optimization method, device, electronic equipment and medium.

[0005] The first technical solution adopted by the present application is:

[0006] A household energy management optimization method, comprising the following steps:

[0007] Modeling loads with different electrical characteristics in a household energy management system;

[0008] According to the established model, the household energy management problem is converted into a Markov decision process MDP without predicting unknown variables or dynamic models thereof;

[0009] According to the Markov decision process, a mixed action space algorithm (MDRL) is used to optimize the scheduling of household energy management system to reduce the operation cost and power consumption of household energy system;

[0010] The mixed action space algorithm integrates DQN algorithm and DDPG algorithm.

[0011] Further, the loads with different electrical characteristics include uncontrollable loads, time-controllable loads, power-controllable loads, battery energy storage systems, electric vehicles and photovoltaic power generation systems.

[0012] Further, the modeling method of various loads is as follows:

[0013] 1) uncontrollable load NSL

[0014] The hourly power demand of uncontrollable load NSL during operation is fixed, and the energy consumption expression is as follows:

[0015] ,

[0016] In the formula, represents the energy consumption of the th NSL at time step t , represents the working time of the th NSL;

[0017] 2) time-controllable load TSL

[0018] The time-controllable load TSL can adjust its operating time within a certain time range while maintaining a fixed power demand, and the energy consumption expression is as follows:

[0019] ,

[0020] where, is the energy consumption of the th TSL at time step t , is the working time of the th TSL, and the energy consumption is 0 if t is not within the working time;

[0021] 3) Power-controllable load PSL

[0022] The power-controllable load PSL has a fixed working time, but its power can be adjusted according to the electricity price and demand. Its energy consumption expression is as follows:

[0023] ,

[0024] where, is the energy consumption of the th PSL at time step t , is its power ratio;

[0025] 4) Battery energy storage system ESS

[0026] The battery energy storage system ESS is used as a flexible load to maintain supply-demand balance. Its state of charge SOC expression is as follows:

[0027]

[0028] s.t. ,

[0029] where, and are the charging and discharging efficiencies of the ESS, respectively, is the capacity of the ESS, and are the maximum values of the charging and discharging amount per hour, respectively; and are the charging and discharging coefficients;

[0030] 5) Electric vehicle EV

[0031] The state of charge of the electric vehicle EV at time step t and its power relationship with the previous time step are represented as follows:

[0032]

[0033]

[0034]

[0035] wherein, and respectively represent the charge-discharge efficiency of EV, represents the capacity of EV, and respectively represent the maximum charge-discharge amount per hour; and respectively represent the charge-discharge coefficient of electric vehicle; represents the battery state of electric vehicle at time t, represents the minimum battery state of electric vehicle, represents the maximum charge state of electric vehicle;

[0036] 6) Photovoltaic power generation system

[0037] The photovoltaic power generation amount is predicted by using deep learning algorithm, and the photovoltaic power generation amount varies between 0 and the predicted value , which is represented as follows:

[0038]

[0039] Further, the conversion of the home energy management problem into a Markov decision process includes:

[0040] The Markov decision process MDP is composed of a 5-tuple , wherein S represents a finite state space composed of environmental states; A is an action space, representing actions that can be taken in the current state; P is a state transition probability function, representing the probability of transitioning from state to the next state by taking action in state R is a reward function that provides an immediate reward during state transition; gamma is a discount factor used to balance the importance of immediate rewards and future rewards;

[0041] The MDP of the home energy management system is as follows:

[0042] 1) State is expressed as follows:

[0043]

[0044] wherein, hour represents per-hour information, dayIndicates daily information. Indicates the amount of solar power generated. Indicates electricity price, Indicates the SOC of EES. Indicates the SOC of the EV;

[0045] 2) Actions The expression is as follows:

[0046]

[0047] In the formula, , This indicates the charge-discharge ratio of the battery energy storage system or electric vehicle, representing the maximum power output. , It is a binary variable 0 / 1, corresponding to the on / off state of the appliance. , This represents the power ratio of two power-controllable loads PSL;

[0048] 3) Rewards:

[0049]

[0050] In the formula, Indicates punishment for violating the rules; This represents the energy consumption of uncontrollable loads. This indicates the energy consumption of a time-controlled load. This indicates the energy consumption of a power-controlled load. Indicates the energy consumption of electric vehicles. This indicates the energy consumption of the battery energy storage system.

[0051] Furthermore, the hybrid action space algorithm includes an actor network and a critic network; the critic network... Used for approximating state-action value functions, while actor networks Used to generate continuous motion ,in, and These are the corresponding network parameters, including weights and biases; Indicates discrete actions. Indicates continuous actions;

[0052] Specifically, the squared loss is minimized using the gradient descent method. To optimize the critic network parameters :

[0053]

[0054] In the formula, E express,Q Represent the Bellman equation; Indicates state, Indicates time discrete actions, Indicates time Continuous actions, Represented as target Q value;

[0055] By gradient Direction adjustment In order to improve Q This value allows for the optimization of actor network parameters;

[0056] Introduce a target network into the actor and critic networks to evaluate the target. Q The values ​​are respectively and The target network parameters use a soft update strategy, as shown below:

[0057]

[0058] In the formula, To adjust the parameters.

[0059] Furthermore, the hybrid action space algorithm is trained in the following manner:

[0060] 1) Use random weights respectively and Initialize the actor network and the critic network;

[0061] 2) By copying and Initialize the target network;

[0062] 3) Initialize the buffer M ;

[0063] 4) Fore = 1:E';

[0064] 5) Through random state of charge Obtain the initial state from a random day. ;

[0065] 6) ;

[0066] 7) Use an exploratory strategy to select actions. ;

[0067] 8) Perform the action The reward observed and move to the next state. ;

[0068] 9) Storing tuples arrive M middle;

[0069] 10) From M Sampling for ;

[0070] 11) Achieve the goal value: ;

[0071] 12) By minimizing the loss Update ;

[0072] 13) Update via gradient using sampling strategy :

[0073]

[0074] 14) Target network soft update;

[0075] 15) End for;

[0076] 16) End for.

[0077] Furthermore, the expression for the exploration strategy is as follows:

[0078]

[0079] In the formula, Represents the discrete action space. Represents the space of continuous actions. It is Gaussian noise. To explore probability.

[0080] The second technical solution adopted in this invention is:

[0081] A home energy management optimization device, comprising:

[0082] The HEMS modeling module is used to model loads with different electrical characteristics in a home energy management system.

[0083] The problem transformation module is used to transform household energy management problems into Markov decision processes (MDPs) that do not require prediction of unknown variables or their dynamic models, based on the established model.

[0084] The algorithm computation module is used to optimize the energy scheduling of the home energy management system based on the Markov decision process and the Hybrid Action Space Algorithm (MDRL) to reduce the operating cost and electricity consumption of the home energy system.

[0085] The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

[0086] The third technical solution adopted in this invention is:

[0087] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a home energy management optimization method as described above.

[0088] The fourth technical solution adopted in this invention is:

[0089] A computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a home energy management optimization method as described above.

[0090] The fifth technical solution adopted in this invention is:

[0091] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the method described above.

[0092] The beneficial effects of this invention are as follows: By employing an MDRL algorithm combining DQN and DDPG, this invention can handle mixed action space problems in HEMS, including combinations of discrete and continuous actions, exhibiting greater flexibility and adaptability. Furthermore, this invention employs a precise discretization method for action processing, which can solve the problem of increased action space complexity in existing technologies. Attached Figure Description

[0093] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0094] Figure 1 This is a schematic diagram of the structure of the home energy management system in an embodiment of the present invention;

[0095] Figure 2 This is a schematic diagram of the network structure of the MDRL algorithm in an embodiment of the present invention;

[0096] Figure 3 This is a flowchart illustrating the steps of a home energy management optimization method according to an embodiment of the present invention. Detailed Implementation

[0097] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0098] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0099] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0100] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0101] To address the issue that treating discrete actions as continuous actions leads to increased complexity in the action space, a reinforcement learning-based method is needed that can simultaneously handle mixed discrete and continuous action spaces. This method can provide a more flexible, accurate, efficient, and comprehensive decision-making framework, suitable for practical applications such as complex energy systems, and has significant value for home energy management.

[0102] Based on this, the present invention employs a RL-based algorithm that learns the optimal decision-making strategy through interaction with the HEMS (Heated Home Energy System). This requires no prior knowledge, adapts to dynamically changing energy environments and unknown variables, and addresses the management problems of complex energy systems involving various household appliances and energy devices, achieving efficient energy utilization and cost reduction. Secondly, the present invention uses an MDRL algorithm combining DQN and DDPG, capable of handling mixed action space problems in HEMS, including combinations of discrete and continuous actions, offering greater flexibility and adaptability. Finally, the present invention employs a precise discretization action processing method, which solves the problem of increased action space complexity in existing technologies. Precise discretization provides more accurate and controllable action selection, thereby achieving more optimized operation and management of the home energy system. In summary, the present invention can handle mixed action spaces, employs a precise discretization action processing method, and achieves the optimal decision-making strategy through interactive learning, thereby optimizing the operation of complex energy systems.

[0103] Example 1

[0104] like Figure 3 As shown, this embodiment provides a method for optimizing home energy management, including the following steps:

[0105] S1. Model loads with different electrical characteristics in a home energy management system (HEMS);

[0106] S2. Based on the established model, the household energy management problem is transformed into a Markov decision process (MDP) that does not require prediction of unknown variables or their dynamic models.

[0107] S3. Based on the Markov decision process, the Hybrid Action Space Algorithm (MDRL) is used to optimize the energy scheduling of the home energy management system in order to reduce the operating cost and electricity consumption of the home energy system.

[0108] The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

[0109] As an optional implementation method, such as Figure 1As shown, the HEMS includes electric vehicles (EVs), battery energy storage systems (ESSs), photovoltaic (PV) systems, and various household appliances such as televisions, air conditioners, washing machines, and dishwashers. Household appliances are categorized based on their characteristics into non-schedulable loads (NSLs, such as refrigerators, televisions, and computers), time-schedulable loads (TSLs, such as washing machines, dishwashers, and dryers), and power-schedulable loads (PSLs, such as air conditioners). In this embodiment, each load in the HEMS is first modeled. Then, the home energy management problem is transformed into a Markov decision process (MDP) that does not require predicting unknown variables or their dynamic models. Finally, the Hybrid Action Space Algorithm (MDRL) is used to optimize the energy scheduling of the HEMS, thereby reducing the operating costs and power consumption of the home energy system.

[0110] The above method will be explained in detail below with reference to the accompanying drawings and specific embodiments.

[0111] (1) Constructing the HEMS model

[0112] 1.1) Uncontrollable load NSL

[0113] Uncontrollable loads (NSLs) typically refer to loads such as lights (L), televisions (TV), smoke detectors, and electronic device chargers. Unlike other loads, NSLs have a fixed hourly power demand during operation, and their energy consumption is shown in equation (1):

[0114] , (1)

[0115] In the formula, Indicates the first NSL at time step t energy consumption Indicates the first NSL working hours.

[0116] 1.2) Time-Controllable Load (TSL)

[0117] Compared to NSL, time-controllable loads (TSL) (mainly referring to water heaters (H) and washing machines (WM)) can adjust their operating time within a certain time range while maintaining a fixed power demand. Their energy consumption is shown in equation (2):

[0118] , (2)

[0119] In the formula, Indicates the first Each TSL at time step t energy consumption Indicates the first The working hours of each TSL, if t Energy consumption is zero outside of working hours.

[0120] 1.3) Power-Controllable Load PSL

[0121] Power controllable loads (PSLs) have a fixed usage time, but their power can be adjusted according to electricity prices and demand. Typical PSL equipment refers to high-power equipment, such as air conditioners (ACs) and microwave machines (MWs), whose energy consumption is shown in equation (3):

[0122] , (3)

[0123] In the formula, Indicates the first PSL at time step t energy consumption This indicates its power ratio.

[0124] 1.4) Battery Energy Storage System (ESS)

[0125] Battery energy storage systems (ESS) serve as flexible loads to maintain supply and demand balance. Their state of charge (SOC) is expressed as follows:

[0126] (4)

[0127] st

[0128] (5)

[0129] (6)

[0130] In the formula, and These represent the charge and discharge efficiencies of the ESS. Indicates the capacity of the ESS. and These represent the maximum charge / discharge rate per hour.

[0131] 1.5) Electric Vehicles (EVs)

[0132] Similar to ESS, electric vehicles (EVs) are in time stept The relationship between the state of charge and its power and the previous time step is expressed as follows:

[0133] (7)

[0134] (8)

[0135] (9)

[0136] In the formula, and These represent the charge and discharge efficiencies of the EV, respectively. Indicates the capacity of EV, and These represent the maximum charge / discharge rate per hour.

[0137] 1.6) Photovoltaic power generation system

[0138] Photovoltaic power generation is predicted using deep learning algorithms. From 0 to the predicted value The changes between them are represented as follows:

[0139] (10)

[0140] 1.7) Family Energy Balance

[0141] To ensure the coordinated and stable operation of HEMS with the power grid, a supply-demand balance needs to be maintained at each time step t. and These represent energy exchanges with the main grid and solar energy, respectively:

[0142] (11)

[0143] (2) Model the household energy management problem as an MDP

[0144] DRL problems are typically modeled as MDPs. An MDP consists of 5 tuples. Composition, in which This represents a finite state space composed of environmental states; It is the action space, representing the actions that can be taken in the current state; : Let be the state transition probability function, representing the state transition probability in state . Take action from state Transition to the next state The probability of; It is a reward function that provides immediate rewards during state transitions; discount factor. gamma∈[0,1] is used to balance the importance of immediate rewards and future rewards. The MDP details for HEMS are as follows.

[0145] 1) State: Construct a 6-dimensional state vector containing daily information, solar power generation, electricity price, and the State of Charge (SOC) of EES and EV. Standard normalization is used to improve training efficiency. The state vector is shown below:

[0146] (12)

[0147] 2) Actions: Since NSL must be satisfied at all times during its operation, action vectors control EES, EV, TSL, and PSL. The action vector for TSL controls the switching state as a discrete variable, while the other action vectors are continuous variables controlling the ratio of power, as shown below:

[0148] (13)

[0149] In the formula: This indicates the charge-discharge ratio of its maximum power. It is a binary variable 0 / 1, corresponding to the "on / off" state of the appliance. This indicates the power ratio of the two PSLs.

[0150] 3) Rewards:

[0151] (14)

[0152] in: The following table shows the penalties for violating constraints, such as SOC constraints, and the satisfaction level of TSL and PSL load usage:

[0153] (15)

[0154] Each term on the right side of formula (15) is a scaling value between [0,1], representing either a violation of SOC constraints or an unmet energy demand of the TSL device on that day.

[0155] (3) MDRL algorithm

[0156] Most existing DRL algorithms require the interaction space to be either discrete or continuous. For example, DQN and its variants are suitable for discrete action spaces, while DDPG is widely used for continuous action spaces. To handle mixed discrete-continuous action spaces, consider an MDRL algorithm that integrates DQN and DDPG.

[0157] Use respectively and Representing discrete actions and continuous actions, where, and Let the discrete action space and the continuous action space be represented respectively. Therefore, the discrete-continuous hybrid action space can be represented as follows: Then the Bellman equation is:

[0158] (16)

[0159] in, and These represent discrete and continuous actions at time t, respectively. and These are the discrete and continuous actions to be performed in the next time step, respectively. If If true, then equation (16) can be written as:

[0160] (17)

[0161] The right side of equation (17) uses an actor-critic network to handle continuous actions. Similar to DDPG, a deep critic network... For approximating state-action value functions, while deterministic deep policy networks Used to generate continuous motion ,in and These are the corresponding network parameters, including weights and biases.

[0162] The network structure diagram of the MDRL algorithm proposed in this embodiment of the invention is shown below. Figure 2 As shown. By searching the discrete action space, the optimal discrete action is obtained, i.e. Corresponding to the highest The choice of discrete action for the value is the same as that for DQN. Therefore, the MDRL algorithm proposed in this embodiment of the invention inherits the advantages of both DDPG and DQN.

[0163] Similar to DDPG, the squared loss in the algorithm is minimized using gradient descent. To optimize the critic network parameters :

[0164] (18)

[0165] in, For the goal Value. To optimize the actor network parameters, the basic idea is to optimize the gradient. Direction adjustment To improve Value. Applying the chain rule, the gradient can be decomposed into the gradient of the state-action value function with respect to consecutive actions and the gradient of the policy with respect to the policy parameters, resulting in a value considering the state distribution. Policy gradient updated by policy parameters .

[0166] (19)

[0167] In DRL, balancing exploration and exploitation is crucial for training efficient decision-making agents. To facilitate the training of deep networks considering discrete-continuous hybrid action spaces, this invention proposes a special exploration strategy that incorporates elements from DQN. - Greedy strategy and adding Gaussian noise to the action in DDPG Combined with strategies.

[0168] (20)

[0169] To address the challenges posed by temporal correlation of samples in DRL for network optimization, empirical replay was considered. (Tuple) Stored in a playback buffer of size m M In the learning process, when the buffer is full, the oldest tuple is deleted. At each time step, a mini-batch of tuples is sampled evenly for network updates. To stabilize the learning process, a target network is introduced into the actor and critic networks to evaluate the target. The values ​​are respectively and The target network parameters use a soft update strategy, as shown below:

[0170] (twenty one)

[0171] in This ensures that the parameters of the target network change slowly, thereby improving the stability of the learning process.

[0172] Referring to Table 1 below, Algorithm 1 describes the network training process, including network initialization and the main loop of the training process. In the main loop, each day is considered an event. At each time step of the day, the agent receives the state. And select actions according to the exploration strategy in equation (20). After the action is performed, the state transitions. And receive a reward Then the tuple The data is stored in the replay buffer, and finally, based on the sampled mean squared loss and policy gradient, the data obtained from uniform sampling from the replay buffer is used. B Tuples for updating and .

[0173] Table 1

[0174]

[0175] In summary, existing RL-based methods primarily consider discrete or continuous action spaces. However, home energy systems typically include various household appliances and energy devices, resulting in a mixed discrete and continuous action space. The current approach to handling the mixed discrete-continuous action space problem in home energy systems involves discretizing continuous actions or treating discrete actions as continuous. While this approach can optimize home energy management to some extent, research has shown that the granularity of continuous action space discretization significantly impacts the performance of Discrete-Quantitative Energy Management (DQN), and treating discrete actions as continuous leads to a significant increase in the complexity of the action space. This invention addresses the Home Energy Management System (HEMS) composed of PV, ESS, EV, and various household appliances. It utilizes the Hybrid Action Space Algorithm (MDRL) to interactively learn the optimal decision-making strategy with the system. This approach requires no prior knowledge, better adapts to dynamically changing energy environments and unknown variables, and thus achieves more optimized and sustainable home energy management. This invention can solve the problems of difficulty in modeling the dynamic processes of complex energy systems and the uncertainties surrounding renewable energy generation and electricity prices. Furthermore, this method has significant advantages in handling the mixed discrete-continuous action space, providing accurate discretized action processing, and achieving comprehensive performance optimization, making it of significant value for home energy management.

[0176] Example 2

[0177] This embodiment provides a home energy management optimization device, including:

[0178] The HEMS modeling module is used to model loads with different electrical characteristics in a home energy management system.

[0179] The problem transformation module is used to transform household energy management problems into Markov decision processes (MDPs) that do not require prediction of unknown variables or their dynamic models, based on the established model.

[0180] The algorithm computation module is used to optimize the energy scheduling of the home energy management system based on the Markov decision process and the Hybrid Action Space Algorithm (MDRL) to reduce the operating cost and electricity consumption of the home energy system.

[0181] The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

[0182] Since this device is a home energy management optimization device according to an embodiment of the present invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of this device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0183] Example 3

[0184] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 3 This illustrates a method for optimizing home energy management.

[0185] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0186] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, performing various server functions and processing data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by accessing data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or more of a Central Processing Unit (CPU) and a modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0187] Since this electronic device is the electronic device corresponding to a home energy management optimization method in this embodiment of the invention, and the principle of the electronic device in solving the problem is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0188] Example 4

[0189] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 3 This illustrates a method for optimizing home energy management.

[0190] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0191] Since this storage medium is the storage medium corresponding to a home energy management optimization method in an embodiment of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and repeated parts will not be described again.

[0192] Example 5

[0193] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a home energy management optimization method according to various exemplary embodiments of this application as described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0194] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0195] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0196] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing home energy management, characterized in that, Includes the following steps: Model loads with different electrical characteristics in a home energy management system; Based on the established model, the household energy management problem is transformed into a Markov decision process; Based on the Markov decision process, a hybrid action space algorithm is used to optimize the energy scheduling of a home energy management system, thereby reducing the operating costs and electricity consumption of the home energy system. The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm. The hybrid action space algorithm includes an actor network and a critic network; the critic network... Used for approximating state-action value functions, while actor networks Used to generate continuous motion ,in, and These are the corresponding network parameters, including weights and biases; Indicates discrete actions. Indicates continuous actions; Specifically, the squared loss is minimized using the gradient descent method. To optimize the critic network parameters : In the formula, Q Represent the Bellman equation; Indicates state, Indicates time discrete actions, Indicates time Continuous actions, Represented as target Q value; By gradient Direction adjustment In order to improve Q This value allows for the optimization of actor network parameters; Introduce a target network into the actor and critic networks to evaluate the target. Q The values ​​are respectively and The target network parameters use a soft update strategy, as shown below: In the formula, To adjust the parameters; The hybrid action space algorithm is trained in the following way: 1) Use random weights respectively and Initialize the actor network and the critic network; 2) By copying and Initialize the target network; 3) Initialize the buffer M ; 4) For e=1:E'; 5) Through random SoC Obtain the initial state from a random day. ; 6) For t=0:23 do; 7) Use an exploratory strategy to select actions. ; 8) Perform the action The reward observed and move to the next state. ; 9) Storing tuples arrive M middle; 10) From M Sampling for ; 11) Achieve the goal value: ; 12) By minimizing the loss Update ; 13) Update via gradient using sampling strategy : 14) Target network soft update; 15) End for; 16) End for; The expression for the exploration strategy is as follows: In the formula, Represents the discrete action space. Represents the space of continuous actions. It is Gaussian noise. To explore probability.

2. The method for optimizing home energy management according to claim 1, characterized in that, The loads with different electrical characteristics include: uncontrollable loads, time-controllable loads, power-controllable loads, battery energy storage systems, electric vehicles, and photovoltaic power generation systems.

3. The method for optimizing home energy management according to claim 2, characterized in that, The modeling methods for various loads are as follows: 1) Uncontrollable load NSL The uncontrollable load NSL has a fixed hourly power demand during operation, and its energy consumption is expressed as follows: ， In the formula, Indicates the first NSL at time step t energy consumption Indicates the first The working hours of each NSL; 2) Time-Controllable Load (TSL) A time-controlled load (TSL) can adjust its operating time within a certain time range while maintaining a fixed power demand. Its energy consumption expression is as follows: ， In the formula, Indicates the first Each TSL at time step t energy consumption Indicates the first The working hours of each TSL, if t Outside of working hours, energy consumption is 0. 3) Power-controllable load PSL Power-controlled loads (PSLs) have a fixed usage time, but their power can be adjusted according to electricity prices and demand. Their energy consumption is expressed as follows: ， In the formula, Indicates the first PSL at time step t energy consumption Indicates its power ratio; 4) Battery Energy Storage System (ESS) Battery energy storage systems (ESS) are used to maintain supply and demand balance, and their state of charge (SOC) is expressed as follows: s.t. , In the formula, and These represent the charge and discharge efficiencies of the ESS. Indicates the capacity of the ESS. and These represent the maximum charge / discharge capacity per hour, respectively. and The charge / discharge coefficient; 5) Electric Vehicles (EVs) Electric vehicles (EVs) in time step t The relationship between the state of charge and its power and the previous time step is expressed as follows: In the formula, and These represent the charge and discharge efficiencies of the EV, respectively. Indicates the capacity of EV, and These represent the maximum charge / discharge capacity per hour, respectively. and These represent the charging and discharging coefficients of the electric vehicle, respectively. express t The battery status of electric vehicles at all times. This indicates the lowest battery state of an electric vehicle. Indicates the maximum charging state of an electric vehicle; 6) Photovoltaic power generation system Photovoltaic power generation is predicted using deep learning algorithms. From 0 to the predicted value The changes between them are represented as follows: 。 4. The method for optimizing home energy management according to claim 1, characterized in that, The process of transforming the household energy management problem into a Markov decision process includes: Markov Decision Process (MDP) consists of 5 tuples Composition, in which S This represents a finite state space composed of environmental states; A It is the action space, representing the actions that can be taken in the current state; P Let be the state transition probability function, representing the state transition probability in state . Take action from state Transition to the next state The probability of; R It is a reward function that provides immediate rewards during state transitions; γ It is a discount factor used to balance the importance of immediate rewards and future rewards; The specific details of a home energy management system's MDP are as follows: 1) Status The expression is as follows: In the formula, hour Indicates information for each hour. day Indicates daily information. Indicates the amount of solar power generated. Indicates electricity price, Indicates the SOC of EES. Indicates the SOC of the EV; 2) Actions The expression is as follows: In the formula, , This indicates the charge-discharge ratio of the battery energy storage system or electric vehicle, representing the maximum power output. , It is a binary variable 0 / 1, corresponding to the on / off state of the appliance. , This represents the power ratio of two power-controllable loads PSL; 3) Rewards: In the formula, Indicates punishment for violating the rules; This represents the energy consumption of uncontrollable loads. This indicates the energy consumption of a time-controlled load. This indicates the energy consumption of a power-controlled load. Indicates the energy consumption of electric vehicles. This indicates the energy consumption of the battery energy storage system.

5. A home energy management optimization device, used to implement the method according to any one of claims 1-4, characterized in that, include: The HEMS modeling module is used to model loads with different electrical characteristics in a home energy management system. The problem transformation module is used to transform household energy management problems into Markov decision processes based on the established model. The algorithm computation module is used to optimize the energy scheduling of the home energy management system based on the Markov decision process and the hybrid action space algorithm, so as to reduce the operating cost and electricity consumption of the home energy system. The hybrid action space algorithm integrates the DQN algorithm and the DDPG algorithm.

6. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 4.

Images