Smart home energy management method and system based on personalized federal reinforcement learning

Abstract

The invention discloses a smart home energy management method and system based on personalized federal reinforcement learning, and belongs to the field of smart home energy management, and the method comprises the following steps: modeling a plurality of heterogeneous smart home energy cost minimization problems, and designing an environment state, an action and a reward function corresponding to a Markov decision process; the edge end intelligent agents of the N heterogeneous smart home environments locally utilize a deep reinforcement learning algorithm optimization strategy and perform federated learning by means of the cloud center server to obtain a pre-training global model with stable training performance; performing post-training fine tuning on the edge end intelligent agent of each heterogeneous smart home environment to obtain N personalized energy management strategies suitable for the N heterogeneous smart home environments; and the personalized energy management strategy obtained through fine adjustment is deployed in an actual environment for operation, and the energy cost of the system can be effectively reduced on the premise that the thermal comfort of residents is fully guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of smart home energy management, and specifically relates to a smart home energy management method and system based on personalized federated reinforcement learning. Background Art

[0002] As an innovative form of modern power supply, the core feature of smart grid is the deep integration of advanced information and communication technologies (such as the Internet of Things and sensor networks) into all aspects of energy production, transmission, distribution and consumption. Under this technical framework, smart homes can effectively use the real-time electricity price mechanism to achieve energy economy management by dynamically optimizing the operation strategies of energy storage equipment and controllable loads. It is worth noting that the HVAC system, which accounts for nearly 40% of the total residential energy consumption, shows significant energy-saving and control value while maintaining the comfort of the indoor thermal environment. As a typical controllable load, the operation mode optimization of this type of temperature control equipment not only needs to meet the thermal comfort needs of the human body, but also needs to seek the optimal solution for operating costs in the dynamic energy price system, so as to achieve the coordinated optimization of living comfort and energy economy.

[0003] In recent years, researchers have conducted in-depth research on the issue of home energy management, which can be divided into model-based methods and learning-based methods according to whether they rely on system models. Traditional model-based energy management methods mainly rely on establishing an accurate model of the home energy system. In practice, due to the existence of various random factors, it is almost impossible to establish an accurate system model, so it is necessary to develop intelligent methods to manage home energy more effectively. Learning-based solutions have the advantage of relaxing the requirements for explicit system models, so they are another energy management strategy with more application prospects than traditional model-based methods, especially model-free deep reinforcement learning methods, which combine the advantages of reinforcement learning and deep learning, do not require environmental model information, and have achieved great success in smart grid optimization decision-making.

[0004] Although significant progress has been made in smart home energy management at home and abroad, there are still some shortcomings. Existing research has not fully considered the heterogeneity in different home energy systems. Heterogeneity is widespread in the real world. Due to the different parameters of different home energy systems (such as HVAC system rated power, energy storage system rated charging and discharging power, building structure, building materials, etc.), the dynamic model of each household energy system will also be different, which in turn determines that the optimal energy management strategy of each household will be different. Existing research lacks personalized energy management strategies designed for environmental heterogeneity. The algorithms proposed in many studies allow each household to train energy management strategies independently. Smart home energy management algorithms based on reinforcement learning / deep reinforcement learning have two shortcomings: (1) Most existing algorithms use the family's own data for training, which is prone to overfitting, resulting in weak performance of the trained energy management strategy; (2) The strategy trained in a specific home environment is difficult to migrate to a new heterogeneous home environment and has weak adaptability. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a smart home energy management method and system based on personalized federated reinforcement learning, which aims to make up for the shortcomings of the existing smart home energy management method that does not take into account the heterogeneity of the home energy system, and to solve the technical problem that the existing method cannot obtain a personalized energy management strategy with superior performance.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solution: The present invention provides a smart home energy management method based on personalized federated reinforcement learning, including: Step 1: Model the energy cost minimization problem of multiple heterogeneous smart homes and design the environment state, action and reward function corresponding to the Markov decision process; Step 2: The edge agents in a heterogeneous smart home environment use deep reinforcement learning algorithms to optimize strategies locally, and use the cloud central server for federated learning to obtain a pre-trained global model with stable training performance. Step 3: Post-train and fine-tune the edge agents in each heterogeneous smart home environment to obtain A heterogeneous smart home environment A personalized energy management strategy; Step 4: Deploy the fine-tuned personalized energy management strategy in the actual environment.

[0007] Further, The expression of the energy cost minimization problem in a heterogeneous smart home environment is as follows: , (1), (2), (3), (4), (5), (6), (7), (8), (9), in: represents the mathematical expectation, Indicates time slot, Indicated in The cost of purchasing electricity from the public grid during the time slot, Representatives in Depreciation cost of time-slot energy storage system; express The charging and discharging power of the time-slot energy storage system, Indicates charging. Indicates discharge; express Input power to the time-slot HVAC system; express The amount of electricity traded with the utility grid during the time slot, Indicates purchasing electricity from the main power grid. Indicates the sale of electricity to the main grid; express The price of purchasing electricity from the public grid during the time slot, express The price of electricity sold to the utility grid during the time slot; represents the depreciation cost coefficient of the energy storage system; express The energy level of the time-slot energy storage system, represents the minimum energy level of the energy storage system, represents the maximum energy level of the energy storage system, Indicates the maximum charging power of the energy storage system, Indicates the maximum discharge power of the energy storage system, express The energy level of the time-slot energy storage system, represents the charging efficiency coefficient of the energy storage system, Represents the discharge efficiency coefficient of the energy storage system; Indicates the maximum input power of the HVAC system; express The photovoltaic power generation power of the time slot, express The non-time-shiftable load power of the time slot; express The indoor temperature of the time slot, express The outdoor temperature of the time slot, express Random thermal perturbations of the time slots, express The indoor temperature of the time slot, Represents the unknown thermal dynamic model of the building; Indicates the lower limit of indoor comfort temperature; represents the upper limit of indoor comfort temperature; heterogeneous parameter set .

[0008] Furthermore, the environmental state and actions of the Markov decision process are as follows: (10), (11), (12), in: In heterogeneous smart homes The environmental status of the time slot, yes The relative time slot of the day, , Indicates that heterogeneous smart homes Actions in time slots; represent Rewards obtained from time slots, express Temperature deviation of the time slot, Indicates the conversion factor from temperature deviation to cost.

[0009] Further, in step 2, The edge agent network structure consists of an actor network and a network of critics , both are multi-layer neural networks, where the actor network is a mapping from environment state to action, using Parameterize the number of neurons in the input layer and the environment state Dimension alignment, number of neurons in the output layer and actions Dimension alignment, the activation function used in the hidden layer is the rectified linear unit function, the output layer uses the hyperbolic tangent function for range compression, and the critic network is used to calculate the action value using Parameterized, the input is the environment state and actions The output is in the state Next action The action value of the corresponding input layer neurons is the sum of the state dimension and the action dimension, the number of neurons in the output layer is 1, and the hidden layer also uses the rectified linear unit function. The edge agent also has two target networks: actor target network and critic target network , actor-target network and critic target network The structure of is the same as the corresponding actor and critic networks, and the parameters are regularly cloned from the original networks.

[0010] Further, in step 2, The local training process of the edge agent in a heterogeneous smart home environment is as follows: (1) Edge Agent Execution Strategy Interact with the environment to collect experience tuples , and stored in the experience buffer; (2) Sample small batches of experience from the experience buffer and update the parameters of the critic network and actor network of the edge agent. The loss function expressions of the critic network and actor network are as follows: (13), (14), in: represents the loss of the edge agent critic network, represents the loss of the edge agent actor network, is the edge agent critic network, represents the edge agent critic target network, is the discount factor of the Markov decision process, represents the edge agent actor network, represents the target network of edge agent actors; (3) Soft update the parameters of the edge agent actor target network and the critic target network, expressed as follows: (15), (16), in: , represents a merging factor; (4) Repeat steps (1) to (3) above. Second-rate.

[0011] Furthermore, in the process of federated learning in step 2, The parameter aggregation expression of an edge agent is as follows: (17), in: Represents the network layer index, represents the layer parameter index, Represents the aggregation Edge Agent Layer parameters, Represents the global model Layer The global model parameters are sent as follows: each edge agent model directly clones the network parameters of the global model as the initial weights of its new round of strategy training. The expression is as follows: (18).

[0012] Furthermore, during the post-training fine-tuning process, each node stops uploading edge agent parameters to the central server, stops obtaining global model parameters from the central server, limits the parameter distance between its own edge agent network model and the pre-trained global model, reduces the learning rate, and performs personalized fine-tuning in its own heterogeneous environment until the performance reaches stability. Each node adds a penalty term to its own edge agent network loss function to constrain the parameter distance between the model and the global model. The expression is as follows: (19), in: Represents the new loss function after adding the distance penalty term, represents the original loss function, represents a scaling factor, Represents any distance calculation method (such as , distance), Representative The network parameters of the edge agent, Represents the global model parameters.

[0013] Furthermore, the training process of the personalized energy management strategy can be divided into two parts: In the first part, the training process of the pre-trained global model is as follows: (1) Initialization Network parameters of edge agents in a heterogeneous smart home And experience buffer ; (2) In each heterogeneous smart home, the edge agent interacts with the environment to collect experience tuples and stores them in the experience cache. ; (3) From the experience buffer The edge agent is trained independently using the deep deterministic policy gradient algorithm by sampling experience tuples in the dataset. (4) Edge agent uploads network parameters To the cloud central server, after aggregation, the global model parameters are obtained ; (5) The cloud center server sends global model parameters To the edge agent of each edge node; (6) Repeat steps (2) to (5) above until a pre-trained global model with stable training performance is obtained; In the second part, the post-training fine-tuning process steps are as follows: (1) The edge agent loads the trained pre-trained global model parameters , and stop uploading parameters to the central server; (2) By adding a penalty term to softly constrain the parameter distance between the local model and the pre-trained global model, the loss function calculation method of the edge agent is changed from Updated to ; (3) The edge agent reduces the learning rate to the original , continue to interact with the environment to collect experience, and sample experience tuples to fine-tune the parameters of the edge agent using the deep deterministic policy gradient algorithm until the performance reaches stability; (4) Repeat the above steps (1) to (3) for each heterogeneous smart home environment to obtain A heterogeneous environment A personalized energy management strategy.

[0014] To solve the above problems, the present invention also includes a smart home energy management system based on personalized federated reinforcement learning, which includes the following modules: Edge information collection module: used to collect the current status information of the heterogeneous smart home environment for use by the edge learning module and edge reasoning module; Edge reasoning module: used to perform reasoning based on the current state information and the strategy neural network, and output the current actions of the energy storage system and the HVAC system; Edge-end action execution module: used to control the energy storage system and HVAC system and handle abnormal actions according to the current action; Edge learning module: used to calculate the reward information of the heterogeneous smart home environment based on the current state, current action and next time slot state, store experience tuples, and perform local energy management strategy training; Cloud center server: used to aggregate the model parameters of edge nodes and output global model parameters; Communication module: used for sending and receiving information between the edge and the cloud center server, regularly sending the local edge agent model parameters of all edge nodes to the cloud center server, and broadcasting the global model parameters of the cloud center server to all edge nodes.

[0015] Furthermore, the operation process steps are as follows: (1) The edge information collection module collects the current status information of the heterogeneous smart home environment and delivers it to the edge reasoning module and edge learning module; (2) The edge reasoning module infers the action based on the current state information and delivers it to the edge action execution module and the edge learning module; (3) The edge action execution module operates the energy storage system and HVAC system according to the action in the heterogeneous smart home environment. When the environment evolves to the next time slot, the edge information collection module collects the status information of the heterogeneous smart home environment again and delivers it to the learning module. (4) The edge learning module calculates the reward according to the designed reward function based on the current state, current action and next time slot state, stores the experience tuple in the experience buffer, and samples a batch of experience tuples in the experience buffer to update the local edge agent model parameters; (5) Repeat steps (1) to (4). Second-rate; (6) The communication module sends the local edge agent model parameters of all edge nodes to the cloud central server; (7) The cloud center server aggregates the edge agent model parameters from the edge nodes and outputs the global model parameters; (8) The communication module broadcasts the global model parameters to the edge inference modules and edge learning modules of all edge nodes; (9) Repeat steps (1) to (8).

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Compared with the existing method of independently learning energy management strategies, the method of the present invention enables each smart home to share knowledge, solves the overfitting problem caused by insufficient experience samples of a single family, and enhances the stability of the energy management strategy training process; (2) Compared with the existing method based on traditional federated learning, the method of the present invention takes into account the heterogeneity existing in the smart home energy system. Through the coordinated optimization of global model pre-training and personalized local fine-tuning, each family can learn an energy management strategy that adapts to its own heterogeneity and is naturally scalable, effectively reducing energy costs while fully ensuring the thermal comfort of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a personalized federated reinforcement learning architecture diagram proposed by the method of the present invention; Figure 3 It is a comparison chart of energy costs of the method of the present invention and other methods; Figure 4 It is a temperature deviation comparison diagram between the method of the present invention and other methods. DETAILED DESCRIPTION

[0018] In order to clearly explain the technical solution and innovative advantages of the present invention, the specific implementation scheme of the present invention is now described in conjunction with the accompanying drawings. It should be particularly pointed out that this implementation case is only used for explanatory description and does not constitute a limitation on the claims of the present invention. This solution proposes a smart home energy management method based on personalized federated reinforcement learning. The energy system consists of a distributed photovoltaic power generation device, a battery energy storage system, a load unit and a smart home energy management system: Specifically, the battery energy storage system optimizes household energy costs by dynamically storing surplus photovoltaic power and implementing energy feedback when there is a power shortage; the load unit includes two types of rigid loads and controllable loads, wherein rigid loads refer to necessary electrical appliances that maintain the basic living needs of users (such as televisions, microwave ovens and other household appliances), and controllable loads specifically refer to HVAC systems with power regulation capabilities; the smart home energy management system implements hourly dynamic optimization decisions on the charging and discharging power of the energy storage system and the HVAC input power based on real-time acquired operating parameters (including but not limited to photovoltaic power generation power, dynamic electricity price signals, ambient temperature and rigid load demand), thereby realizing multi-energy coordinated regulation and maximizing economic benefits. Figure 1 As shown in FIG. 1 , a smart home energy management method based on personalized federated reinforcement learning includes: step 1: modeling multiple heterogeneous smart home energy cost minimization problems and designing the environment state, action and reward function corresponding to the Markov decision process; step 2: The edge agents in the heterogeneous smart home environment use the deep reinforcement learning algorithm to optimize the strategy locally, and use the cloud center server for federated learning to obtain a pre-trained global model with stable training performance; Step 3: Post-train and fine-tune the edge agents in each heterogeneous smart home environment to obtain a pre-trained global model suitable for A heterogeneous smart home environment A personalized energy management strategy; Step 4: deploy the fine-tuned personalized energy management strategy to run in the actual environment.

[0019] Further, The expression of the energy cost minimization problem in a heterogeneous smart home environment is as follows: , (1), (2), (3), (4), (5), (6), (7), (8), (9), in: represents the mathematical expectation, Indicates time slot, Indicated in The cost of purchasing electricity from the public grid during the time slot, Representatives in Depreciation cost of time-slot energy storage system; express The charging and discharging power of the time-slot energy storage system, Indicates charging. Indicates discharge; express Input power to the time-slot HVAC system; express The amount of electricity traded with the utility grid during the time slot, Indicates purchasing electricity from the main power grid. Indicates the sale of electricity to the main grid; express The price of purchasing electricity from the public grid during the time slot, express The price of electricity sold to the utility grid during the time slot; represents the depreciation cost coefficient of the energy storage system; express The energy level of the time-slot energy storage system, represents the minimum energy level of the energy storage system, represents the maximum energy level of the energy storage system, Indicates the maximum charging power of the energy storage system, Indicates the maximum discharge power of the energy storage system, express The energy level of the time-slot energy storage system, represents the charging efficiency coefficient of the energy storage system, Represents the discharge efficiency coefficient of the energy storage system; Indicates the maximum input power of the HVAC system; express The photovoltaic power generation power in the time slot, express The non-time-shiftable load power of the time slot; express The indoor temperature of the time slot, express The outdoor temperature of the time slot, express Random thermal perturbations of the time slots, express The indoor temperature of the time slot, Represents the unknown thermal dynamic model of the building; Indicates the lower limit of indoor comfort temperature; Indicates the upper limit of indoor comfort temperature; heterogeneous parameter set .

[0020] Furthermore, the environmental state and actions of the Markov decision process are as follows: (10), (11), (12), in: In heterogeneous smart homes The environmental status of the time slot, yes The relative time slot of the day, , Indicates that heterogeneous smart homes Actions in time slots; represent Rewards obtained from time slots, express Temperature deviation of the time slot, Indicates the conversion factor from temperature deviation to cost.

[0021] Further, in step 2, The edge agent network structure consists of an actor network and a network of critics , both are multi-layer neural networks, where the actor network is a mapping from environment state to action, using Parameterize the number of neurons in the input layer and the environment state Dimension alignment, number of neurons in the output layer and actions Dimension alignment, the activation function used in the hidden layer is the rectified linear unit function, the output layer uses the hyperbolic tangent function for range compression, and the critic network is used to calculate the action value using Parameterized, the input is the environment state and actions The output is in the state Next action The action value of the corresponding input layer neurons is the sum of the state dimension and the action dimension, the number of neurons in the output layer is 1, and the hidden layer also uses the rectified linear unit function. The edge agent also has two target networks: actor target network and critic target network , actor-target network and critic target network The structure of is the same as the corresponding actor and critic networks, and the parameters are regularly cloned from the original networks.

[0022] Further, in step 2, The local training process of the edge agent in a heterogeneous smart home environment is as follows: (1) Edge Agent Execution Strategy Interact with the environment to collect experience tuples , and stored in the experience buffer; (2) Sample small batches of experience from the experience buffer and update the parameters of the critic network and actor network of the edge agent. The loss function expressions of the critic network and actor network are as follows: (13), (14), in: represents the loss of the edge agent critic network, represents the loss of the edge agent actor network, is the edge agent critic network, represents the edge agent critic target network, is the discount factor of the Markov decision process, represents the edge agent actor network, represents the target network of edge agent actors; (3) Soft update the parameters of the edge agent actor target network and the critic target network, expressed as follows: (15), (16), in: , represents a merging factor; (4) Repeat steps (1) to (3) above. Second-rate.

[0023] Furthermore, in the process of federated learning in step 2, The parameter aggregation expression of an edge agent is as follows: (17), in: Represents the network layer index, represents the layer parameter index, Represents the aggregation Edge Agent Layer parameters, Represents the global model Layer The global model parameters are sent as follows: each edge agent model directly clones the network parameters of the global model as the initial weights of its new round of strategy training. The expression is as follows: (18).

[0024] Furthermore, during the post-training fine-tuning process, each node stops uploading edge agent parameters to the central server, stops obtaining global model parameters from the central server, limits the parameter distance between its own edge agent network model and the pre-trained global model, reduces the learning rate, and performs personalized fine-tuning in its own heterogeneous environment until the performance reaches stability. Each node adds a penalty term to its own edge agent network loss function to constrain the parameter distance between the model and the global model. The expression is as follows: (19), in: Represents the new loss function after adding the distance penalty term, represents the original loss function, represents a scaling factor, represents any distance calculation method, Representative The network parameters of the edge agent, Represents the global model parameters.

[0025] Furthermore, the personalized energy management strategy training process can be divided into two parts: In the first part, the training process of the pre-trained global model is as follows: (1) Initialization Network parameters of edge agents in a heterogeneous smart home And experience buffer ; (2) In each heterogeneous smart home, the edge agent interacts with the environment to collect experience tuples and stores them in the experience cache. ; (3) From the experience buffer The edge agent is trained independently using the deep deterministic policy gradient algorithm by sampling experience tuples in the dataset. (4) Edge agent uploads network parameters To the cloud central server, after aggregation, the global model parameters are obtained ; (5) The cloud center server sends global model parameters To the edge agent of each edge node; (6) Repeat steps (2) to (5) above until a pre-trained global model with stable training performance is obtained; In the second part, the post-training fine-tuning process steps are as follows: (1) The edge agent loads the trained pre-trained global model parameters , and stop uploading parameters to the central server; (2) By adding a penalty term to softly constrain the parameter distance between the local model and the pre-trained global model, the loss function calculation method of the edge agent is changed from Updated to ; (3) The edge agent reduces the learning rate to the original , continue to interact with the environment to collect experience, and sample experience tuples to fine-tune the parameters of the edge agent using the deep deterministic policy gradient algorithm until the performance reaches stability; (4) Repeat the above steps (1) to (3) for each heterogeneous smart home environment to obtain A heterogeneous environment A personalized energy management strategy.

[0026] To solve the above problems, the present invention includes a smart home energy management system based on personalized federated reinforcement learning, which includes the following modules: Edge information collection module: used to collect the current status information of the heterogeneous smart home environment for use by the edge learning module and edge reasoning module; Edge reasoning module: used to perform reasoning based on the current state information and the strategy neural network, and output the current actions of the energy storage system and the HVAC system; Edge-end action execution module: used to control the energy storage system and HVAC system and handle abnormal actions according to the current action; Edge learning module: used to calculate the reward information of the heterogeneous smart home environment based on the current state, current action and next time slot state, store experience tuples, and perform local energy management strategy training; Cloud center server: used to aggregate the model parameters of edge nodes and output global model parameters; Communication module: used for sending and receiving information between the edge and the cloud center server, regularly sending the local edge agent model parameters of all edge nodes to the cloud center server, and broadcasting the global model parameters of the cloud center server to all edge nodes.

[0027] Furthermore, the operation process steps are as follows: (1) The edge information collection module collects the current status information of the heterogeneous smart home environment and delivers it to the edge reasoning module and edge learning module; (2) The edge reasoning module infers the action based on the current state information and delivers it to the edge action execution module and the edge learning module; (3) The edge action execution module operates the energy storage system and HVAC system according to the action in the heterogeneous smart home environment. When the environment evolves to the next time slot, the edge information collection module collects the status information of the heterogeneous smart home environment again and delivers it to the learning module. (4) The edge learning module calculates the reward according to the designed reward function based on the current state, current action and next time slot state, stores the experience tuple in the experience buffer, and samples a batch of experience tuples in the experience buffer to update the local edge agent model parameters; (5) Repeat steps (1) to (4). Second-rate; (6) The communication module sends the local edge agent model parameters of all edge nodes to the cloud central server; (7) The cloud center server aggregates the edge agent model parameters from the edge nodes and outputs the global model parameters; (8) The communication module broadcasts the global model parameters to the edge reasoning modules and edge learning modules of all edge nodes; (9) Repeat steps (1) to (8).

[0028] In order to demonstrate the performance of the method of the present invention, three comparison schemes are introduced.

[0029] Comparison scheme 1: The rule-based energy management strategy uses two parts. First, determine the power of the HVAC system at the current moment. When the indoor temperature exceeds the upper limit of the comfortable temperature, turn on the HVAC system at maximum power. When the indoor temperature is lower than the lower limit of the comfortable temperature, turn off the HVAC system, otherwise keep the power of the previous time slot unchanged. Then determine the charging and discharging power of the energy storage system. When the power generation of the photovoltaic new energy system is greater than the power consumption of all loads in the smart home, the excess power is stored in the energy storage system. When the power generation of the photovoltaic new energy system is less than the power consumption of all loads in the smart home (i.e., the HVAC system and conventional loads), the energy storage system releases power.

[0030] Comparison plan 2: Energy management strategy based on deep reinforcement learning, that is, using a deep deterministic policy gradient algorithm in each smart home to independently learn the energy management strategy, without using federated learning for collaborative training.

[0031] Comparison plan 3: Energy management strategy based on traditional federated deep reinforcement learning, that is, using a deep deterministic policy gradient algorithm in each smart home, and introducing traditional federated learning to collaboratively train the energy management strategy. Instead of performing personalized fine-tuning, the energy management strategy corresponding to the global model is directly applied to all smart home environments.

[0032] Figure 3 and Figure 4 The average energy cost (the average value of the total energy cost of all households) and the average temperature deviation (the average value of the total temperature deviation of all households) of the method of the present invention and the comparison scheme in 20 smart home environments are shown. Figure 3 As shown, compared with Scheme 1 and Scheme 2, the method of the present invention can reduce the average energy cost by 6.7%-15.7%. Figure 4 As shown in the figure, compared with all other schemes, the method of the present invention achieves the smallest average temperature deviation and effectively ensures the thermal comfort of the residents. Comparative Scheme 2 independently trains energy management strategies, and each smart home cannot share knowledge. The average energy cost and average temperature deviation indicators are inferior to those of the present invention. Compared with Comparative Scheme 3, the method of the present invention can reduce the average temperature deviation by 93.1% under the premise of increasing the average energy cost by 0.2%.

[0033] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A smart home energy management method based on personalized federated reinforcement learning, characterized in that: The steps include: Step 1: Model the energy cost minimization problem of multiple heterogeneous smart homes and design the environment state, action and reward function corresponding to the Markov decision process; Step 2: The edge agents in a heterogeneous smart home environment use deep reinforcement learning algorithms to optimize strategies locally, and use the cloud central server for federated learning to obtain a pre-trained global model with stable training performance. Step 3: Post-train and fine-tune the edge agents in each heterogeneous smart home environment to obtain A heterogeneous smart home environment A personalized energy management strategy; Step 4: Deploy the fine-tuned personalized energy management strategy in the actual environment.

2. The smart home energy management method based on personalized federated reinforcement learning according to claim 1 is characterized in that: In step 1, The expression of the energy cost minimization problem in a heterogeneous smart home environment is as follows: ， （1）， （2）， （3）， （4）， （5）， （6）， （7）， （8）， （9）， in: represents the mathematical expectation, Indicates time slot, Indicated in The cost of purchasing electricity from the public grid during the time slot, Representatives in Depreciation cost of time-slot energy storage system; express The charging and discharging power of the time-slot energy storage system, Indicates charging. Indicates discharge; express Input power to the time-slot HVAC system; express The amount of electricity traded with the utility grid during the time slot, Indicates purchasing electricity from the main power grid. Indicates the sale of electricity to the main grid; express The price of purchasing electricity from the public grid during the time slot, express The price of electricity sold to the utility grid during the time slot; represents the depreciation cost coefficient of the energy storage system; express The energy level of the time-slot energy storage system, represents the minimum energy level of the energy storage system, represents the maximum energy level of the energy storage system, Indicates the maximum charging power of the energy storage system, Indicates the maximum discharge power of the energy storage system, express The energy level of the time-slot energy storage system, represents the charging efficiency coefficient of the energy storage system, Represents the discharge efficiency coefficient of the energy storage system; Indicates the maximum input power of the HVAC system; express The photovoltaic power generation power of the time slot, express The non-time-shiftable load power of the time slot; express The indoor temperature of the time slot, express The outdoor temperature of the time slot, express Random thermal perturbations of the time slots, express The indoor temperature of the time slot, Represents the unknown thermal dynamic model of the building; Indicates the lower limit of indoor comfort temperature; represents the upper limit of indoor comfort temperature; heterogeneous parameter set .

3. The smart home energy management method based on personalized federated reinforcement learning according to claim 1, characterized in that: In step 1, the environment state, action and reward function of the Markov decision process are The design of a heterogeneous smart home is as follows: （10）， （11）， （12）， in: In heterogeneous smart homes The environmental status of the time slot, yes The relative time slot of the day, , Indicates that heterogeneous smart homes Actions in time slots; represent Rewards obtained from time slots, express Temperature deviation of the time slot, Indicates the conversion factor from temperature deviation to cost.

4. The smart home energy management method based on personalized federated reinforcement learning according to claim 1 is characterized in that: In step 2, The edge agent network structure consists of an actor network and a network of critics , both are multi-layer neural networks, where the actor network is a mapping from environment state to action, using Parameterize the number of neurons in the input layer and the environment state Dimension alignment, number of neurons in the output layer and actions Dimension alignment, the activation function used in the hidden layer is the rectified linear unit function, the output layer uses the hyperbolic tangent function for range compression, and the critic network is used to calculate the action value using Parameterized, the input is the environment state and actions The output is in the state Next action The action value of the corresponding input layer neurons is the sum of the state dimension and the action dimension, the number of output layer neurons is 1, and the hidden layer also uses the rectified linear unit function. The edge agent also has two target networks: actor target network and critic target network , actor-target network and critic target network The structure of is the same as the corresponding actor and critic networks, and the parameters are regularly cloned from the original networks.

5. The smart home energy management method based on personalized federated reinforcement learning according to claim 1 is characterized in that: In step 2, The local training process of the edge agent in a heterogeneous smart home environment is as follows: (1) Edge Agent Execution Strategy Interact with the environment to collect experience tuples , and stored in the experience buffer; (2) Sample small batches of experience from the experience buffer and update the parameters of the critic network and actor network of the edge agent. The loss function expressions of the critic network and actor network are as follows: （13）， （14）， in: represents the loss of the edge agent critic network, represents the loss of the edge agent actor network, is the edge agent critic network, represents the edge agent critic target network, is the discount factor of the Markov decision process, represents the edge agent actor network, represents the target network of edge agent actors; (3) Soft update the parameters of the edge agent actor target network and the critic target network, expressed as follows: （15）， （16）， in: , represents a merging factor; (4) Repeat steps (1) to (3) above. Second-rate.

6. The smart home energy management method based on personalized federated reinforcement learning according to claim 1, characterized in that: In the process of federated learning in step 2, The parameter aggregation expression of an edge agent is as follows: （17）， in: Represents the network layer index, represents the layer parameter index, Represents the aggregation Edge Agent Layer parameters, Represents the global model Layer The global model parameters are sent as follows: each edge agent model directly clones the network parameters of the global model as the initial weights of its new round of strategy training. The expression is as follows: (18).

7. The smart home energy management method based on personalized federated reinforcement learning according to claim 1, characterized in that: In the post-training fine-tuning process of step 3, each node stops uploading edge agent parameters to the central server, stops obtaining global model parameters from the central server, limits the parameter distance between its own edge agent network model and the pre-trained global model, reduces the learning rate, and performs personalized fine-tuning in its own heterogeneous environment until the performance reaches stability. Each node adds a penalty term to its own edge agent network loss function to constrain the parameter distance between the model and the global model. The expression is as follows: （19）， in: Represents the new loss function after adding the distance penalty term, represents the original loss function, represents a scaling factor, represents any distance calculation method, Representative The network parameters of the edge agent, Represents the global model parameters.

8. The smart home energy management method based on personalized federated reinforcement learning according to claim 1, characterized in that: The learning process of personalized energy management strategy is as follows: In the first part, the training process of pre-trained global model is as follows: (1) Initialization Network parameters of edge agents in a heterogeneous smart home And experience buffer ; (2) In each heterogeneous smart home, the edge agent interacts with the environment to collect experience tuples and stores them in the experience cache. ; (3) From the experience buffer The edge agent is trained independently using the deep deterministic policy gradient algorithm by sampling experience tuples in the dataset. (4) Edge agent uploads network parameters To the cloud central server, after aggregation, the global model parameters are obtained ; (5) The cloud center server sends global model parameters To the edge agent of each edge node; (6) Repeat steps (2) to (5) above until a pre-trained global model with stable training performance is obtained; In the second part, the post-training fine-tuning process steps are as follows: (1) The edge agent loads the trained pre-trained global model parameters , and stop uploading parameters to the central server; (2) By adding a penalty term to softly constrain the parameter distance between the local model and the pre-trained global model, the loss function calculation method of the edge agent is changed from Updated to ; (3) The edge agent reduces the learning rate to the original , continue to interact with the environment to collect experience, and sample experience tuples to fine-tune the parameters of the edge agent using the deep deterministic policy gradient algorithm until the performance reaches stability; (4) Repeat the above steps (1) to (3) for each heterogeneous smart home environment to obtain A heterogeneous smart home environment A personalized energy management strategy.

9. The system used in the smart home energy management method based on personalized federated reinforcement learning according to any one of claims 1 to 8, characterized in that: The system includes the following modules: Edge information collection module: used to collect the current status information of the heterogeneous smart home environment for use by the edge learning module and edge reasoning module; Edge reasoning module: used to reason based on current state information and policy neural networks, and output the current actions of the energy storage system and HVAC system; Edge action execution module: used to control the energy storage system and HVAC system and handle abnormal actions according to the current action; Edge learning module: used to calculate the reward information of the heterogeneous smart home environment based on the current state, current action and next time slot state, store experience tuples, and perform local energy management strategy training; Cloud center server: used to aggregate the model parameters of edge nodes and output global model parameters; Communication module: used for sending and receiving information between the edge and the cloud center server, regularly sending the local edge agent model parameters of all edge nodes to the cloud center server, and broadcasting the global model parameters of the cloud center server to all edge nodes.

10. The system according to claim 9, characterized in that The operation process steps are as follows: (1) The edge information collection module collects the current status information of the heterogeneous smart home environment and delivers it to the edge reasoning module and edge learning module; (2) The edge reasoning module infers the action based on the current state information and delivers it to the edge action execution module and the edge learning module; (3) The edge action execution module operates the energy storage system and HVAC system according to the action in the heterogeneous smart home environment. When the environment evolves to the next time slot, the edge information collection module collects the status information of the heterogeneous smart home environment again and delivers it to the learning module. (4) The edge learning module calculates the reward according to the designed reward function based on the current state, current action and next time slot state, stores the experience tuple in the experience buffer, and samples a batch of experience tuples in the experience buffer to update the local edge agent model parameters; (5) Repeat steps (1) to (4). Second-rate; (6) The communication module sends the local edge agent model parameters of all edge nodes to the cloud central server; (7) The cloud center server aggregates the edge agent model parameters from the edge nodes and outputs the global model parameters; (8) The communication module broadcasts the global model parameters to the edge reasoning modules and edge learning modules of all edge nodes; (9) Repeat steps (1) to (8).

Images