Home energy management method and system fusing action mask deep reinforcement learning

Abstract

The application relates to a household energy management method and system based on fusion of action mask deep reinforcement learning, and belongs to the field of household energy intelligent management. The method comprises the following steps: collecting environment parameters of a household energy system, and constructing a mathematical model of the household energy system; based on a household energy scheduling problem, a household energy management Markov decision process based on time sequence feature perception and a safe action mask is constructed; an improved DQN network fusing a double-channel time sequence prediction module is constructed and trained; based on real environment parameters of a to-be-tested day, the improved DQN network after training is adopted to output a final household energy system optimization scheduling result. Under the premise of strictly guaranteeing the system physical safety boundary and user comfort (including indoor temperature control and on-time completion of a load task), the application realizes global minimization of household electricity cost through intelligent scheduling, and effectively overcomes the limitations of high calculation complexity of a traditional model-driven method and poor flexibility of a rule control method.

Description

A Home Energy Management Method and System Integrating Action Masking and Deep Reinforcement Learning Technical Field

[0001] This invention belongs to the field of intelligent home energy management, specifically relating to a home energy management method and system that integrates action masking deep reinforcement learning. More specifically, it relates to a home energy management method and system that integrates Long Short-Term Memory Network (LSTM) and safe action masking mechanism, and uses an improved deep Q-network (DQN) to generate the optimal control strategy. Background Technology

[0002] With the rapid development of smart grids and distributed energy technologies, home energy management systems (HEMS) have become a crucial link connecting the user side and the grid side to achieve a balance between electricity supply and demand. However, the modern home energy environment is increasingly complex, integrating photovoltaic power generation, battery energy storage systems (ESS), electric vehicles (EVs), and various temperature-controlled / non-temperature-controlled loads, exhibiting a high degree of randomness and uncertainty. Existing management strategies, if unable to effectively address these characteristics, will lead to persistently high household electricity costs; simultaneously, if dispatching strategies lack flexibility, they can easily cause a decline in user thermal comfort or failure of critical electricity-consuming tasks (such as EV travel), making it difficult to achieve a balance between economy and comfort.

[0003] Currently, control strategies for home energy management are mainly divided into two categories: model-based and model-free. Although there has been extensive research, the following technical bottlenecks still exist:

[0004] ① Limitations of Model-Based Methods: Represented by Mixed Integer Linear Programming (MILP) and Model Predictive Control (MPC), these methods heavily rely on the accuracy of the system's mathematical model and the precise prediction of photovoltaic output and load demand. However, prediction errors are unavoidable in real-world scenarios, leading to poor robustness of the control performance. Furthermore, as the number of household devices increases, the computational complexity rises exponentially, making it difficult to meet the real-time scheduling requirements of home gateways.

[0005] ② Deficiencies of traditional model-free methods (rule-based): Although rule-based logic control is simple and easy to implement, the strategy is rigid and cannot adapt to real-time electricity price fluctuations and dynamic changes in user behavior, and usually cannot obtain the global optimal solution.

[0006] ③ Limitations of Conventional Deep Reinforcement Learning (DRL) Methods: Although algorithms such as Deep Q-Networks (DQN), as model-free methods, can solve the problem of model dependence by learning strategies through interaction with the environment, they still have two significant limitations when applied to complex HEMS: 1) Insufficient Feature Extraction Capability: Traditional DQN networks usually directly use the current state snapshot as input, ignoring the strong temporal correlations inherent in the intermittency of photovoltaic power generation and real-time electricity price fluctuations. This makes it difficult for the agent to capture the evolution of the environment, limiting decision-making performance. 2) Lack of Hard Safety Constraints: Most existing methods guide the agent to avoid violations (such as battery overcharging or exceeding temperature limits) by setting negative penalties in the reward function. This "soft constraint" mechanism cannot strictly shield infeasible actions at the physical level. Especially in the early training or exploration phases, the agent is prone to outputting actions that violate physical constraints, posing a risk of damaging equipment or causing safety accidents.

[0007] In summary, there is an urgent need for a home energy management method that can effectively extract environmental temporal characteristics and provide strict physical security guarantees. Summary of the Invention

[0008] The purpose of this invention is to address the technical problems of existing home energy management systems, such as insufficient feature extraction in complex dynamic environments, difficulty in effectively balancing economy and user comfort, and insufficient system security due to the "soft constraint" mechanism of traditional deep reinforcement learning algorithms relying solely on penalty terms. This invention provides a home energy management method and system that integrates action masking deep reinforcement learning. This method employs a dual-channel improved deep Q-network (DQN) that integrates Long Short-Term Memory (LSTM) networks, utilizing LSTM to capture the strong temporal correlation between photovoltaic power generation and time-of-use pricing. Simultaneously, it introduces an action masking mechanism to directly mask invalid actions that violate physical constraints at the decision output layer of the neural network. This invention aims to achieve global minimization of household electricity costs through intelligent scheduling, while strictly ensuring the system's physical safety boundaries and user comfort (covering indoor temperature control and timely completion of load tasks). This effectively overcomes the limitations of high computational complexity in traditional model-driven methods and poor flexibility in rule-based control methods.

[0009] To achieve the above objectives, the technical solution of the present invention is: a home energy management method integrating action mask deep reinforcement learning, comprising:

[0010] Collect environmental parameters of the home energy system and construct a mathematical model of the home energy system;

[0011] To address the home energy scheduling problem based on home energy systems, a Markov decision process for home energy management based on temporal feature perception and security action masking is constructed.

[0012] An improved DQN network with a fused dual-channel time prediction module was constructed and trained.

[0013] Based on the real environmental parameters of the test day, an improved DQN network that has been trained is used to output the final optimized scheduling result of the home energy system.

[0014] Further mathematical models of home energy systems include:

[0015] (1) Photovoltaic power generation system modeling:

[0016]

[0017] in, Let be the photovoltaic power generation at time t. , Let be the light intensity and outdoor temperature at time t, respectively;

[0018] (2) Household load modeling: Set the load for each household It has a fixed operating power. and duration Household load The allowed time window for execution is , Indicates the start time of the run. Indicates the end time of the run, household load. Once started within a window, it must run continuously. Each time step must be completed without interruption.

[0019] (3) Energy storage battery modeling:

[0020]

[0021] in, for The state of charge of the energy storage battery at time +1. for The state of charge of the energy storage battery at any given time. and They are respectively The charging power and discharging power at any given time. and These are charging efficiency and discharging efficiency, respectively. This refers to the total capacity of the energy storage battery.

[0022] (4) Electric vehicle modeling:

[0023]

[0024] in, This represents the state of charge of the electric vehicle at time t+1. Let represent the electric vehicle's state of charge at time t. For state variables, This indicates that the electric car is at home. Indicates that the electric car has left home. and These refer to the charging efficiency and discharging efficiency of electric vehicles, respectively. and These are the charging power and discharging power of the electric vehicle, respectively. This refers to the total capacity of the electric vehicle's battery.

[0025] Furthermore, based on temporal feature perception and safety action masking, the Markov decision process for home energy management is improved in terms of state space, action selection mechanism, and reward function, as follows:

[0026] (1) Define the enhanced state space :

[0027]

[0028] in, For the past Historical electricity price sequence at a given time point For the past The historical photovoltaic power generation sequence at each moment Let be the indoor temperature at time t. Remaining time for the load The electric vehicle is in or away from home.

[0029] (2) Construct a security action mask Introducing a security mask vector during the action selection phase. , Similar to the action space dimension, it is used to indicate a physically permissible and legal action at the current moment;

[0030]

[0031] in, It represents any candidate control action in the action space; the generation logic of the safety action mask is based on the physical hard constraints of energy storage battery power constraints, electric vehicle constraints, and household load time window constraints.

[0032] (3) Reward function :

[0033]

[0034] in, For electricity purchase costs, As a penalty for temperature discomfort, Penalties for electric vehicles failing to meet expected battery capacity targets. These are the weighting coefficients.

[0035] Furthermore, k=24.

[0036] Furthermore, the specific physical hard constraints are as follows:

[0037] Energy storage battery power constraint: if Charging is prohibited; if Then the discharge action is prohibited. , These represent the maximum and minimum states of charge of the energy storage battery, respectively.

[0038] Electric vehicle constraints: if This means that when an electric vehicle leaves home, all charging and discharging activities are prohibited; if the electric vehicle is at home and its battery is fully charged or depleted, charging and discharging activities are restricted.

[0039] Household load time window constraint: If the current time Not in household load The allowed time window for execution is Within the system, actions that enable home loads are prohibited.

[0040] Furthermore, the improved DQN network, which integrates a dual-channel timing prediction module, includes three parallel input branches, as detailed below:

[0041] Photovoltaic feature extraction branch: LSTM network is used for processing. Sequence, extracting the time-series fluctuation feature vector of photovoltaic power generation. ;

[0042] Electricity price feature extraction branch: Processed using an LSTM network with independent weights. Sequence, extracting market pattern feature vectors of electricity price changes ;

[0043] Device status branches: Processed using a fully connected layer, except... sequence, Extracting device state feature vectors from scalar states outside the sequence. ;

[0044] The three feature vectors obtained from the three parallel input branches are concatenated: The original Q-value is calculated by inputting the fully connected layer. .

[0045] Furthermore, the improved DQN network that integrates a dual-channel timing prediction module also includes a mask-based action selection strategy, as detailed below:

[0046] When selecting an action, combine the original Q value with the safe action mask. Combined, filtering out illegal actions:

[0047]

[0048] use The -greedy strategy is based on Select an action; among which, It represents any candidate control action in the action space (such as the charging / discharging power level of an energy storage system or the switching command of a load). The original action value function is generated by a deep neural network based on the current state. Direct output, targeting actions The expected return assessment value reflects the economic tendency of the action without considering physical constraints. Represents the security mask vector The overall physical meaning of the modified action value function lies in: introducing a security mask vector. This forces the value of all illegal actions that violate physical hard constraints (such as battery overcharging, running outside the time window, etc.) to be negative infinity. After this processing, the algorithm will automatically ignore all actions when executing the action selection strategy. A path with negative infinity ensures the final output scheduling instructions. It must be an executable action that satisfies both physical security boundaries and has the highest economic benefits.

[0049] Furthermore, the improved DQN network, which integrates a dual-channel time series prediction module, utilizes gradient descent to update the parameters of the LSTM network and the fully connected layers.

[0050] The present invention also provides a home energy management system that integrates action mask deep reinforcement learning, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the method described above.

[0051] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, which, when executed by the processor, can implement the method described above.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) This invention addresses the problem of balancing electricity costs and user comfort (including indoor temperature and completion of common household load tasks) in home energy management. It constructs an optimization problem with the goal of minimizing operating costs and with user comfort and equipment physical characteristics as constraints. Under the premise of strictly ensuring the user's quality of life and the safe operation of equipment, it minimizes the household's electricity expenditure.

[0054] (2) By constructing a dual-channel LSTM sensing network, this invention can independently extract the meteorological correlation features of photovoltaic power generation and the market fluctuation features of real-time electricity prices. Compared with the single fully connected input of traditional DQN, it can more accurately predict the trend of environmental change.

[0055] (3) The present invention is economically significant and highly adaptable. Compared with the traditional rule-based control (RBC) strategy, the present invention can automatically adapt to the fluctuation of real-time electricity prices, intelligently schedule the charging and discharging timing of electric vehicles (V2G) and energy storage systems, as well as the start and stop times of commonly used household loads, and make full use of the price difference for energy arbitrage and load transfer, thereby significantly reducing the electricity cost of households.

[0056] (4) This invention innovatively introduces an action masking mechanism, directly embedding physical hard constraints such as battery SOC limits and device operating windows into the action selection layer. Compared with the traditional 'soft constraints' that rely solely on negative rewards, this invention can fundamentally eliminate dangerous behaviors such as battery overcharging and over-discharging, while also reducing the exploration space and improving the convergence speed of the algorithm. Attached Figure Description

[0057] Figure 1 is a schematic diagram of the hardware topology of the home energy management system of the present invention;

[0058] Figure 2 is a software architecture diagram of the improved DQN home energy management system of the present invention;

[0059] Figure 3 is a flowchart of the improved DQN algorithm based on security action mask used in this invention;

[0060] Figure 4 shows the cumulative reward convergence curve during the training process of the improved DQN algorithm under the framework of this invention;

[0061] Figure 5 is a comparison chart of the operating costs of the home energy system under different control strategies according to the present invention;

[0062] Figure 6 shows the optimized scheduling diagram of the improved DQN system on a typical operating day according to the present invention. Detailed Implementation

[0063] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0064] This invention provides a home energy management method that integrates action mask deep reinforcement learning, including:

[0065] Collect environmental parameters of the home energy system and construct a mathematical model of the home energy system;

[0066] To address the home energy scheduling problem based on home energy systems, a Markov decision process for home energy management based on temporal feature perception and security action masking is constructed.

[0067] An improved DQN network with a fused dual-channel time prediction module was constructed and trained.

[0068] Based on the real environmental parameters of the test day, an improved DQN network that has been trained is used to output the final optimized scheduling result of the home energy system.

[0069] The following is a detailed implementation process of the present invention.

[0070] As shown in Figures 2 and 3, this invention provides a home energy management method that integrates action mask deep reinforcement learning, which mainly includes the following steps:

[0071] Step 1: First, determine the hardware configuration and environmental parameters of the home energy system to provide basic data for simulation modeling.

[0072] (1) Collect real-time electricity price data for the local area, as well as corresponding historical outdoor temperature and light intensity data.

[0073] (2) Set the user's comfort range for indoor temperature as follows: .

[0074] (3) Determine the attributes of uninterrupted household loads. Dishwasher (power 1.5kW, running time 2h, user-defined allowed window 21:00-23:00); Washing machine (power 2.0kW, running time 2h, user-defined allowed window 18:00-22:00).

[0075] (4) The energy storage battery capacity is set to 15 kWh and the maximum power is 3 kW; the electric vehicle battery capacity is set to 30 kWh and the maximum power is 4 kW. The charging and discharging efficiency of both is set to 0.9.

[0076] Step 2: Using the parameters collected in Step 1, construct mathematical models of each subsystem of the home energy system (see Figure 1) in the program to form a digital twin environment.

[0077] (1) Model the photovoltaic power generation system, input real-time illumination and temperature, and output the photovoltaic power generation at the current moment:

[0078]

[0079] in, Let be the photovoltaic power generation at time t. , Let be the light intensity and outdoor temperature at time t, respectively;

[0080] (2) Household load modeling: A logic control model is adopted to ensure that the household load starts within the window specified by the user, and once started, it must run continuously for a specified duration without being interrupted by the algorithm.

[0081]

[0082]

[0083]

[0084] in, Let be a binary state variable, representing the first... The household load at any time The running status, Indicates that the program is running. Indicates stop. Subscript Represents the household load number (such as washing machine, dishwasher, etc.), subscript Represents the current time step. Indicates the first The allowed time window for a household workload to run, i.e., the user-defined time window. The earliest start time and latest end time of each household load. Each household load It has a fixed operating power. and duration (number of time steps) . This represents the time offset for continuous operation constraints, used to describe the persistent state after load startup. The three formulas above correspond to:

[0085] ① Non-window constraint: Outside the allowed time window, the load state is forced to be 0;

[0086] ② Runtime constraint: Within the allowed window, the total number of time steps the load runs must be equal to its rated runtime. ;

[0087] ③ Continuity constraint: If the load is in Start at time (i.e., the previous time) =0, current time If it is 1), then in the following... The value must remain at 1 within each time step to ensure the task cannot be interrupted.

[0088] (3) Establish state equations for energy storage batteries and electric vehicle batteries. For electric vehicles, the model adds an additional state variable to determine whether they are at home (schedulable state);

[0089] Modeling of energy storage batteries (energy storage systems):

[0090]

[0091] in, for The state of charge of the energy storage battery at time +1. for The state of charge of the energy storage battery at any given time. and They are respectively The charging power and discharging power at any given time. and These are charging efficiency and discharging efficiency, respectively. This refers to the total capacity of the energy storage battery.

[0092] Electric vehicle modeling:

[0093]

[0094] in, This represents the state of charge of the electric vehicle at time t+1. Let represent the electric vehicle's state of charge at time t. For state variables, This indicates that the electric car is at home. Indicates that the electric car has left home. and These refer to the charging efficiency and discharging efficiency of electric vehicles, respectively. and These are the charging power and discharging power of the electric vehicle, respectively. This refers to the total capacity of the electric vehicle's battery.

[0095] Step 3: Formalize the household energy dispatching problem into an improved Markov decision process. Specific implementation details are as follows:

[0096] (1) State vector configuration: Set the length of the history sequence This means the state includes electricity price and photovoltaic data from the past 24 hours, used to capture daily periodic characteristics. The complete state vector has a dimension of [missing information]. dimension.

[0097] (2) Action mask generation module: Write the logical judgment function get_mask(state).

[0098] For example, for energy storage batteries, if currently... (The set upper limit protection threshold) then the mask bit for the corresponding "charging" action is set to 0; for washing machine load, if the current time And the user settings window is If the action mask bit corresponding to "turn on the washing machine" is set to 0, then this ensures that any high-scoring action output by the neural network is physically executable.

[0099] Step 4: Build and train a deep reinforcement learning model based on LSTM-DQN.

[0100] (1) Model architecture construction: Use PyTorch or TensorFlow to build the network.

[0101] Branch A (Electricity Price Prediction): Input dimension (Batch, 24, 1), contains a 64-unit LSTM layer, and outputs a 64-dimensional feature vector.

[0102] Branch B (Photovoltaic Prediction): Input dimension (Batch, 24, 1), contains a 64-unit LSTM layer, and outputs a 64-dimensional feature vector.

[0103] Branch C (State Features): Input dimension (Batch, 8), passes through two fully connected layers (128, 64), and outputs a 64-dimensional feature vector.

[0104] Fusion layer: The three 64-dimensional vectors above are concatenated into a 192-dimensional vector, which is then mapped to the output layer of the action space dimension through two fully connected layers (256, 128).

[0105] (2) Implementation of the training process:

[0106] Set the experience replay pool size to 10,000 and the batch size to 64.

[0107] In the training loop, after obtaining the Q-value vector through the forward propagation of the neural network each time, it is added to... This causes the Q value of the illegal action to become a very small negative number.

[0108] The argmax operation is used to select actions, thus ensuring that no illegal actions are selected.

[0109] This mechanism allows agents to learn advanced strategies such as "when to charge and discharge the battery most cost-effectively" without having to learn basic physical rules like "the battery cannot be charged when it is full" through trial and error, thus significantly improving learning efficiency.

[0110] Step 5: Continuously train the model and monitor its convergence. Figure 4 shows the cumulative reward convergence curve during the training process of the improved DQN algorithm under the framework of this invention. As shown in Figure 4, the horizontal axis represents the number of training rounds, and the vertical axis represents the average cumulative reward. The gray dashed line in the figure represents the traditional standard DQN algorithm, and the red solid line represents the improved DQN algorithm of this invention. It can be seen that the standard DQN algorithm has a slower convergence speed and greater fluctuations. This invention, however, benefits from the action masking mechanism, which effectively shields invalid explorations of illegal actions. Combined with the precise extraction of environmental features by the dual-channel LSTM, the algorithm exhibits a rapid upward trend in the early stages of training and reaches a high-score, stable convergence state around 300 rounds, significantly outperforming existing technologies. After the algorithm fully converges, the Q-network parameters are frozen, real environmental data from the test day is input, and the final optimized scheduling result for home devices is output. As shown in Figure 6, the generated strategy exhibits significant economic optimization characteristics: during the early morning and nighttime periods when electricity prices are low (e.g., 00:00-05:00 and after 21:00), the system controls the energy storage system to charge and store energy, and arranges electric vehicles to charge in an orderly manner while meeting the electricity constraints for the next day's travel; during the midday period when photovoltaic power generation is sufficient (10:00-15:00), priority is given to controlling the energy storage battery to absorb excess photovoltaic power to reduce curtailment; during the evening peak period when electricity prices are highest (17:00-20:00), the system instructs the energy storage battery and electric vehicles (V2G mode) to discharge simultaneously to support household load and take advantage of peak-valley price differences for arbitrage. Figure 5 shows a comparison of the total monthly operating costs under different control strategies. Among them, the gray bar chart represents the rule-based control (RBC) strategy, with a monthly cost as high as 65.4 yuan; the blue bar chart represents the standard DQN strategy, with a cost of 52.8 yuan; while the green bar chart represents the improved DQN strategy proposed in this invention, with a monthly cost of only 38.5 yuan. Data shows that, compared to the RBC strategy, this invention achieves a cost reduction of approximately 41% through precise "low storage, high release" timing scheduling and safety action constraints, which strongly demonstrates the significant beneficial effects of this system in reducing household energy expenditures and improving economic efficiency.

[0111] The present invention also provides a home energy management system that integrates action mask deep reinforcement learning, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the method described above.

[0112] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, which, when executed by the processor, can implement the method described above.

[0113] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A home energy management method integrating action masking and deep reinforcement learning, characterized in that, include: Collect environmental parameters of the home energy system and construct a mathematical model of the home energy system; Based on the problem of home energy scheduling in home energy systems, a Markov decision process for home energy management based on time-series feature perception and safety action masking is constructed. The state space, action selection mechanism and reward function are improved as follows: (1) Define the enhanced state space. ： in, For the past Historical electricity price sequence at a given time point Let be the electricity price at time t. For the past The historical photovoltaic power generation sequence at each moment Let be the photovoltaic power generation at time t. Let be the indoor temperature at time t. Let be the outdoor temperature at time t. Remaining time for the load Let represent the electric vehicle's state of charge at time t. For electric vehicles in and out of the home status. for (2) Constructing a safety action mask: Introducing a safety mask vector during the action selection phase. ， Similar to the action space dimension, it is used to indicate a physically permissible and legal action at the current moment; in, Represents any candidate control action in the action space; the generation logic of the safety action mask is based on the physical hard constraints of energy storage battery power constraints, electric vehicle constraints and household load time window constraints; (3) reward function ： in, For electricity purchase costs, As a penalty for temperature discomfort, Penalties for electric vehicles failing to meet expected battery capacity targets. The weights are used as coefficients; an improved DQN network with a fused dual-channel time series prediction module is constructed and trained, including three parallel input branches: photovoltaic feature extraction branch: processed using an LSTM network. Sequence, extracting the time-series fluctuation feature vector of photovoltaic power generation. Electricity price feature extraction branch: Processed using an LSTM network with independent weights. Sequence, extracting market pattern feature vectors of electricity price changes Device status branches: Processed using a fully connected layer, except... sequence, Extracting device state feature vectors from scalar states outside the sequence. The three feature vectors obtained from the three parallel input branches are concatenated: The original Q-value is calculated by inputting the fully connected layer. Based on the real environmental parameters of the test day, an improved DQN network that has been trained is used to output the final optimized scheduling result of the home energy system.

2. The home energy management method incorporating action mask deep reinforcement learning according to claim 1, characterized in that, Mathematical models of home energy systems include: (1) Photovoltaic power generation system modeling: in, (2) Household load modeling: Set the load of each household. It has a fixed operating power. and duration Household load The allowed time window for execution is ， Indicates the start time of the run. Indicates the end time of the run, household load. Once started within a window, it must run continuously. (3) Energy storage battery modeling: (4) Modeling the energy storage battery: (5) Modeling the energy storage battery: (6) Modeling the energy storage battery: (7) Modeling the energy storage battery: (8) Modeling the energy storage battery: (9) Modeling the energy storage battery: (10) Modeling the energy storage battery: (11) Modeling the energy storage battery: (12) Modeling the energy storage battery: (13) Modeling the energy storage battery: (14 in, for The state of charge of the energy storage battery at time +1. and They are respectively The charging power and discharging power at any given time. and These are charging efficiency and discharging efficiency, respectively. (3) Total capacity of energy storage batteries; (4) Electric vehicle modeling: in, This represents the state of charge of the electric vehicle at time t+1. For state variables, This indicates that the electric car is at home. Indicates that the electric car has left home. and These refer to the charging efficiency and discharging efficiency of electric vehicles, respectively. and These are the charging power and discharging power of the electric vehicle, respectively. This refers to the total capacity of the electric vehicle's battery.

3. The home energy management method incorporating action mask deep reinforcement learning according to claim 1, characterized in that, k=24。 4. The home energy management method incorporating action mask deep reinforcement learning according to claim 1, characterized in that, The specific physical hard constraints are as follows: Energy storage battery capacity constraint: If If so, charging is prohibited; like Then the discharge action is prohibited. 、 These represent the maximum and minimum states of charge of the energy storage battery, respectively; Electric vehicle constraints: If... This means that when an electric vehicle leaves home, all charging and discharging activities are prohibited; if the electric vehicle is at home and its battery is fully charged / fully discharged, charging / discharging activities are restricted; Household load time window constraint: if the current time... Not in household load The allowed time window for execution is Within the system, actions that enable home loads are prohibited.

5. The home energy management method incorporating action mask deep reinforcement learning according to claim 1, characterized in that, The improved DQN network, which integrates a dual-channel timing prediction module, also includes a mask-based action selection strategy, specifically as follows: when selecting an action, the original Q-value is compared with a safe action mask. Combined, filtering out illegal actions: use The -greedy strategy is based on Select an action; among which, The original action value function is based on the current state. Direct output, targeting actions The expected return assessment value; Represents the security mask vector The revised action value function.

6. The home energy management method incorporating action mask deep reinforcement learning according to claim 1, characterized in that, An improved DQN network that integrates a dual-channel time series prediction module utilizes gradient descent to update the parameters of the LSTM network and fully connected layers.

7. A home energy management system integrating action masking and deep reinforcement learning, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the home energy management method of fusion action mask deep reinforcement learning as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It stores computer program instructions that can be executed by a processor. When the processor executes the computer program instructions, it can implement the home energy management method of fusion action mask deep reinforcement learning as described in any one of claims 1-6.

Images