A home micro-grid energy optimization control method based on meta-reinforcement learning

Abstract

The application discloses a kind of based on meta-reinforcement learning's family micro-grid energy optimization control method, under the condition of no building thermal power model and guaranteeing user comfort requirement, research includes the joint coordination problem of demand response and alternating current optimal power flow with renewable energy resource consumption, comprising: firstly, the minimum problem of family micro-grid operating cost is modeled as Markov decision process;Secondly, the Soft Actor Critic algorithm integrated with Normalizing Flow is constructed;Thirdly, the method of meta-learning is added in the training of the constructed algorithm.The application has the advantages that it can save more operating cost potential under the premise of guaranteeing the comfort requirement of user, and can quickly adapt to different family micro-grid topological structure and quickly adapt to new tasks.

Description

TECHNICAL FIELD

[0001] The application relates to a meta-reinforcement learning-based energy optimization control method for a household micro-grid, and belongs to the technical field of intelligent buildings. BACKGROUND

[0002] Online energy management of a household micro-grid is a key means for meeting the comfort requirement of energy use of a household and optimizing the operation cost of the household micro-grid. However, due to the high nonlinearity of thermal dynamics, the high heterogeneity of building stocks, and different types and parameters of household appliances, traditional energy management relies on model accuracy and specific case environments, resulting in low reliability and poor generalization ability.

[0003] To solve the above problems, deep reinforcement learning, which is based on the model-free advantage, avoids complex modeling of the household micro-grid energy system and gradually becomes a popular solution. Through the collection of real-time information, online energy management of the household micro-grid can fully utilize renewable energy and demand-side resources at the residential level.

[0004] However, traditional reinforcement learning has insufficient representation ability for complex distribution, and cannot fully represent the complexity and diversity of power grid data in the household micro-grid environment, resulting in inaccurate or biased models, and poor robustness of the model when the environment changes. Therefore, how to improve the online energy management strategy to adapt to the energy optimization of the household micro-grid in complex environments and strong uncertain scenarios is a topic worthy of in-depth study. SUMMARY

[0005] The application aims to overcome the deficiencies in the prior art, and provides a meta-reinforcement learning-based energy optimization control method for a household micro-grid, which improves the scalability of the household micro-grid, speeds up the learning speed of the energy management strategy, and enhances the robustness of the household micro-grid when the environment changes.

[0006] In a first aspect, the application provides a meta-reinforcement learning-based energy optimization control method for a household micro-grid, which is applied to a household micro-grid containing photovoltaic power generation and at least includes the following steps:

[0007] S1: establishing a Markov decision model MDP based on energy management of the household micro-grid.

[0008] According to the mathematical model of the energy management of the household micro-grid, the state of the Markov decision process is determined.

[0009] According to the working principles and characteristics of the energy storage system ESS, electric vehicles, heating, ventilation and air conditioning equipment and the like, the action of the Markov decision process is confirmed.

[0010] According to the goal of the household micro-grid to minimize the residential energy cost while meeting the comfort of the occupants, the reward function of the Markov decision process is determined;

[0011] According to different household micro-grid topologies, a set of Markov decision processes is established.

[0012] S2: Based on the Markov decision model MDP of household micro-grid energy management, a deep reinforcement learning algorithm NFSAC is constructed by introducing a normalized flow, that is, a normalized flow (NormalizingFlow) is introduced in the standard deep reinforcement learning algorithm SAC.

[0013] The SAC algorithm is a kind of off-policy actor critic deep reinforcement learning algorithm under the maximum entropy reinforcement learning framework. In the SAC algorithm, the normalized flow strategy is introduced, so that the policy distribution is closer to the Boltzmann distribution, thereby better capturing the multi-peak behavior of the micro-grid, and the agent has the ability to learn a better scheduling strategy in a complex environment.

[0014] S3: Based on the constructed NFSAC algorithm, a meta-reinforcement learning real-time control algorithm is established, that is, meta-learning is introduced in NFSAC.

[0015] S4: Use the meta-reinforcement learning real-time control algorithm to optimize the energy of the household micro-grid.

[0016] In a second aspect, the present application provides a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method.

[0017] In a third aspect, the present application provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method.

[0018] The building micro-grid online energy management method based on meta-reinforcement learning provided by the present application has the following beneficial effects:

[0019] 1. By simulating the uncertain environment of the building micro-grid, the model is optimized to find a better overall strategy. While maximizing the reward function, the residential energy cost and the occupants' thermal discomfort and mileage anxiety penalty are minimized.

[0020] 2. To solve the problem of sparse rewards in the micro-grid environment and improve the stability of the algorithm, the normalized flow strategy is introduced in the SAC algorithm, so that the policy distribution is closer to the Boltzmann distribution, thereby better capturing the multi-peak behavior of the micro-grid.

[0021] 3. Compared with existing deep reinforcement learning algorithms that are trained on specific topologies, this invention significantly improves training speed when facing entirely new home power grid environments. Attached Figure Description

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0023] Figure 1 This is an energy management framework diagram of the online energy management method for building microgrids based on meta-reinforcement learning used in this invention;

[0024] Figure 2 This is a flowchart of a method according to an embodiment of the present invention;

[0025] Figure 3 This is a diagram showing the scheduling results of the method proposed in this invention on a certain test day, where (a) is a diagram of fixed load and photovoltaic power generation, (b) is a diagram of ESS charging and discharging, (c) is a diagram of EV charging, and (d) is a diagram of indoor and outdoor temperatures when the HVAC system is on or off. Detailed Implementation

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0027] like Figure 1 , Figure 2 As shown in the figure, an embodiment of the present invention provides an energy optimization control method for a home microgrid based on meta-reinforcement learning. The method includes the following steps:

[0028] S1: Establish a Markov decision model (MDP) for energy management in residential microgrids.

[0029] Based on the mathematical model of energy management in a home microgrid, the state of the Markov decision process is determined; based on the working principles and characteristics of energy storage systems (ESS), electric vehicles, HVAC systems, and other equipment, the actions of the Markov decision process are confirmed; based on the goal of minimizing residential energy costs while meeting the comfort needs of residents, the reward function of the Markov decision process is determined; and based on different home microgrid topologies, a set of Markov decision processes is established.

[0030] S2: Based on the Markov decision model MDP for energy management of home microgrids, construct the deep reinforcement learning algorithm NFSAC that introduces normalizing flow, that is, introduce normalizing flow into the standard deep reinforcement learning algorithm SAC.

[0031] The SAC algorithm is a kind of off-policy actor-critic deep reinforcement learning algorithm under the maximum entropy reinforcement learning framework. In the SAC algorithm, a normalized flow policy is introduced to make the policy distribution closer to the Boltzmann distribution, thereby better capturing the multi-peak behavior of the microgrid and enabling the agent to learn a better scheduling strategy in a complex environment.

[0032] S3: Based on the constructed NFSAC algorithm, a meta-reinforcement learning real-time control algorithm is established, that is, meta-learning is introduced into NFSAC.

[0033] The basic assumption of the application of meta-reinforcement learning in considering different device parameters of the household microgrid is that the tasks suitable for meta-reinforcement learning training and the tasks suitable for meta-reinforcement learning testing are subject to the same task distribution.

[0034]

[0035] wherein τ represents a task, that is, a specific household microgrid topology, represents a task set for training, that is, a set of household microgrid topologies used for training parameters; represents a task set for testing, that is, a set of household microgrid topologies used for verifying the training effect. p(τ) represents the probability distribution of the task. The objective function can be defined as follows:

[0036]

[0037] In the formula, represents the updated parameters on the training data; represents the loss function on the training data; represents the loss function on the test data.

[0038] S4: Use the meta-reinforcement learning real-time control algorithm for household microgrid energy optimization.

[0039] Further, in the step S1, the construction of the Markov policy model includes the following processes:

[0040] S1-1: Establish the state and action space.

[0041] All the environmental information elements observed by the household energy management system agent at the t period of time include: the output power of the photovoltaic power generation unit the electricity purchase price the electricity selling price the fixed load demand power the state of charge of the energy storage system ESS the outdoor temperature the state tuple of the electric vehicle the state tuple of the heating, ventilation and air conditioning Further, the state space is obtained, and the state at time t is defined as:

[0042]

[0043] State s t Some elements in the state space have inherent random characteristics and are not affected by the actions of the agent, which are uncontrollable state spaces, including The controllable state space includes

[0044] The scheduling decision variables of the agent include the charging and discharging power of the ESS The exchange power between the ESS and the main grid The power of the heating, ventilation and air conditioning The power of the electric vehicle They satisfy the following power balance constraints:

[0045] Wherein, is the power purchased from the main grid, is the power sold to the main grid, is the discharging power of the ESS at time t, is the charging power of the ESS at time t.

[0046] S1-2: Establish the action space.

[0047] The action space contains all possible actions that the agent can choose in each state, and the action a t taken by the agent at time t is not considered The action a t is defined as:

[0048] S1-3: Set the reward function: according to the expected total energy cost, the basic electricity reward r t of the agent at time t is represented as follows:

[0049]

[0050] Wherein, σ1 and σ2 are used to control the order of magnitude and the trade-off between minimizing the operating cost and the penalty caused by power imbalance, represents the depreciation and degradation cost of the battery, represents the operating cost of the ESS.

[0051] ΔP t corresponds to the power imbalance at time t, and is defined as

[0052]

[0053] The basic reward is added with a household thermal discomfort penalty term The expression is as follows

[0054]

[0055] wherein, represents the set temperature and the indoor temperature difference threshold, and ω1 represents a thermal comfort weight factor;

[0056] S1-4: Establish a set of Markov decision processes.

[0057] For a specific Markov decision process, the state at time t+1 and the reward obtained at time t completely depend on the state at time t and the action taken at time t, and are irrelevant to all states before time t, that is, the following relationship is satisfied:

[0058] (s t+1 ,r t )~p(·|s t ,a t )

[0059] wherein p(·|s t ,a t ) represents a conditional probability distribution given the current state s t and the action a t .

[0060] In the context of a home microgrid, when actions such as switching a load device, adjusting the working mode of an energy storage system, changing the working state of a photovoltaic power generation system, etc. occur, the state transition probability of the system will change. Therefore, each configuration of the home microgrid corresponds to a specific Markov decision process, and finally a set of Markov decision processes is established, and the probability distribution of all possible topological structures of the home microgrid during training is denoted as p(τ).

[0061] Further, in the step S2, the update of the NFSAC includes the following process:

[0062] S2-1: Sample a batch of experience samples (s t ,a t ,r t ,s t+1 ) i＝1,2...,N from the experience replay pool D, wherein s t , a t , r t , s t+1 respectively represent the state, action, reward obtained, and new state transferred after executing the action a t of the agent at time t; N is the number of samples in the batch;

[0063] S2-2: For each experience sample, calculate the target Q value y.

[0064] S2-21: Initial sample generation. The current home micro-grid state s t is input into the Actor network to obtain the parameters of the initial Gaussian distribution, i.e. the mean μ and the log variance logσ. According to the calculated mean μ and standard deviation σ, the initial sample z0 is sampled from the Gaussian distribution.

[0065] z0 ~ N(μ, σ 2 )

[0066] S2-22: Multi-level affine coupling transformation. The multi-level affine coupling transformation transforms the initial sample z0 into a sample z k of a complex distribution through a series of affine coupling layers.

[0067] S2-221: Input segmentation and parameter calculation. The first half dimension of the input sample z0 is denoted as and the second half dimension is denoted as The scaling parameter and the offset parameter

[0068]

[0069] In the formula, W h ,W s ,W s is the weight matrix; b h ,b s ,b t is the bias vector.

[0070] S2-222: Reflection transformation. The reflection transformation is performed on , and the formula is

[0071]

[0072] In the formula, ⊙ represents element multiplication, and exp represents element exponential function. The output after transformation is

[0073]

[0074] S2-223: Multi-layer superposition. The output of S2-222 is taken as the input of the next layer, and multiple affine coupling layers are stacked layer by layer. The process of steps S2-221 and S2-222 is repeated, and the sample is converted into a complex distribution through multi-level transformation to obtain the standardized flow output z k .

[0075] S2-23: The transformed sample z kThe target action is obtained by limiting the range of the action space through a tanh function That is

[0076] S2-24: Calculate the policy density

[0077] The policy density is calculated using the formula where p(z) is the final policy density, p(z0) is the initial density, is the final output vector after multiple layer transformations, is the absolute value of the Jacobian determinant of the total transformation.

[0078] The log Jacobian determinants of each layer are multiplied to obtain the overall log likelihood, and the log likelihood is optimized to maximize the probability of the sample under the model.

[0079]

[0080] where log p(z) is the log of the final policy density, is the cumulative log Jacobian determinant of each layer.

[0081] S2-25: Calculate the target Q value.

[0082]

[0083] where r is the immediate reward; γ is the discount factor, which measures the importance of future rewards; is the future expected return evaluated by the target Q network; α is the temperature parameter, which controls the influence of the entropy term; is the entropy, which is used to encourage the exploration of the policy.

[0084] S2-3: Update the Q network. Update the parameters of the Q network by minimizing the mean square error between the current Q network and the target value y.

[0085]

[0086] where N is the number of samples in this batch; is the Q value output by the current Q network; y is the target Q value.

[0087] S2-4: Update the policy network. The action generation and density calculation are consistent with S2-21 to S2-24. Update the parameters θ of the policy network by minimizing the policy loss function.

[0088]

[0089] where α represents the temperature parameter, and logπ θ (a∣s) represents the entropy, Q value outputted by the current Q network;

[0090] S2-5: update the temperature parameter a. Minimize the loss function of the temperature parameter, so that the entropy of the policy approaches the target entropy Thus, the balance between exploration and utilization is dynamically adjusted.

[0091]

[0092] In the formula, is the target entropy, which is used to control the exploration intensity; a represents the temperature parameter;

[0093] S2-6: update the target Q network. Using a soft update method, the parameters φ' of the target Q network are updated i to the current Q network parameters φ i Gradually approach each other, stabilize the training process.

[0094] φ' i ← τφ i +(1-τ)φ' i

[0095] In the formula, τ is a soft update coefficient.

[0096] Further, in the step S3, the combination of meta-learning and the NFSAC algorithm includes the following processes:

[0097] The goal of meta-learning is to find a set of initial parameters, so that when facing a new task, a small amount of gradient updates can quickly adapt to the new task. Unlike traditional deep reinforcement learning algorithms that train a single MDP, meta-reinforcement learning needs to train a large number of MDPs that follow the same distribution to obtain experience, so as to learn how to quickly adapt to new tasks.

[0098] The interaction process between the agent and the environment in meta-reinforcement learning includes inner loops and outer loops. After initializing the model parameters, in each inner loop, the agent trains for a specific MDP, calculates the task loss J i (θ i ), and in the outer loop, the meta-loss is summarized to update the meta-parameters.

[0099] Specifically, in the inner loop, a task is randomly sampled from the task sample set for testing. For each round in this test, the calculation process is consistent with that described in S2. At the same time, for task i, the loss J i (θ i (θ i on the updated parameters θ

[0100]

[0101] wherein Indicates from strategy π′ i Defined state-action distribution The sampled states s and actions a are defined; r(s,a) represents the reward obtained by taking action a in state s; γ represents the discount factor used to calculate the time discount when calculating the cumulative reward. This represents the value function in state s'; Indicates the next state s ′ value function Expectations; It represents the logarithm of the probability of taking action a in state s;

[0102] Next, the losses from all tasks are summed to obtain the meta-learning loss J. meta (θ):

[0103]

[0104] In the outer loop, gradient descent is used to optimize the parameter θ, minimizing the meta-learning loss:

[0105]

[0106] Where β is the learning rate of meta-learning, which controls the step size of gradient descent updates; This represents the value function parameterized using parameter θ; The loss function J represents the meta-learning loss function. meta (θ) with respect to the value function The gradient;

[0107] In each outer loop iteration, the policy network π is updated simultaneously. θ Q-network and target Q-network φ The parameters enable the model to adapt quickly to new tasks. Once training is complete, the model already possesses a certain degree of generalization ability, allowing it to quickly adapt to new microgrid operation tasks with only a small number of training samples. This enables the microgrid control system to effectively manage and optimize energy resources in the face of different operating conditions and demands.

[0108] For each time period of a selected test day, based on the currently observed initial environmental state s t The optimal policy learned by the trained meta-reinforcement learning real-time control algorithm is used to determine the corresponding control action a. t Subsequently, a reward of r was received. t And observe the environmental state transition to s t+1 This process continues until all test sessions are scheduled online.

[0109] Furthermore, step S4 specifically involves:

[0110] S41: Deploy model: deploy the trained model into the home microgrid agent.

[0111] S42: Initial observation: record the initial environment state and total test period on the test day.

[0112] S43: Strategy application: use the model to formulate the optimal energy management strategy and determine the best scheduling action based on the current environment state.

[0113] S44: Feedback receiving: the agent obtains the reward value and observes the environment state of the next period.

[0114] S45: Iterative optimization: repeat the process of strategy application and feedback receiving until the energy optimization of all test periods is completed.

[0115] From Figure 3 As can be seen from (a) and (b), the ESS absorbs as much excess PV power as possible during the peak PV output period to avoid selling it. At the same time, the ESS discharges during the high electricity consumption period (8:00-9:00 and 17:00-20:00) when the electricity purchase price is high, which helps to reduce the electricity purchase cost and effectively utilize the PV output. In addition, the ESS does not immediately charge a large amount when the SoC is at a low level, but moves the charging period to a period with a lower electricity price (22:00-3:00 the next day). As can be seen from (c), the agent does not immediately charge the electric vehicle after it is connected, but moves the charging period of the electric vehicle to the valley period (22:00-7:00 the next day).

[0116] As can be seen from (d) in the figure, due to the low temperature during 2:00 to 7:00, in order to reduce the energy consumption cost, the heating and air conditioning system does not start immediately, but chooses to start later to save cost. On the other hand, it is worth noting that the heating and air conditioning system always keeps the indoor temperature within the comfortable range to ensure user comfort.

[0117] This charging and discharging strategy and heating and air conditioning operation mode shows an optimized energy management scheme, effectively reducing the overall energy consumption cost while ensuring efficient use of energy and user comfort experience. This strategy not only reasonably utilizes PV power, but also realizes the reduction of comprehensive cost through peak-shifting electricity consumption and reasonable control of heating and air conditioning operation.

[0118] The above embodiments are not a limitation of the present application, and the present application is not limited to the above embodiments, as long as it meets the requirements of the present application, it belongs to the protection scope of the present application.

Claims

1. A method for energy optimization control of a home microgrid based on meta-reinforcement learning, characterized in that Comprise the following steps: Step S1, establish a Markov decision model MDP based on home micro-grid energy management; Step S2, based on the Markov decision model MDP of home micro-grid energy management, a deep reinforcement learning algorithm NFSAC is constructed by introducing standardized flow; Step S3, based on the constructed NFSAC algorithm, a meta-reinforcement learning real-time control algorithm is established; Step S4, using the meta-reinforcement learning real-time control algorithm for home micro-grid energy optimization; Wherein, step S2 is specifically: S2-1. Sample a batch of experience samples from the experience playback pool D. , Let denot t represent the agent's state, action, reward, and action performed at time t, respectively. The new state after transition; N is the number of samples in this batch; S2-2, for each experience sample, calculate the target Q value y; S2-3, update the Q network: The Q network is used to estimate the Q value of a given state and action pair, that is, the expected return; by minimizing the mean square error between the current Q network and the target Q value y, the parameters of the Q network are updated; ； In the formula, N is the number of samples in the batch; is the Q value output by the current Q network; y is the target Q value; respectively represent the state and action; S2-4, update the policy network: The policy network is used to generate actions, with the goal of generating actions that maximize cumulative reward while maintaining sufficient entropy to promote exploration; by minimizing a policy loss function , updating parameters of the policy network ; ； wherein denotes a temperature parameter, denotes an entropy, denotes the Q-value of the current Q-network output; S2-5, update the temperature parameter a: by minimizing a loss function of the temperature parameter such that the entropy of the policy approaches a target entropy to dynamically adjust the balance between exploration and exploitation; ； In the formula, is a target entropy used to control the exploration intensity; denotes a temperature parameter; S2-6, update the target Q network: Using a soft update method, the parameters of the target Q network are towards the current Q network parameters Gradually converge, stabilize the training process; ； In the formula, is a soft update coefficient; Step S2-2 is specifically: S2-21: initial sample generation: Current home microgrid status The input is fed into the Actor network to obtain the parameters of the initial Gaussian distribution, i.e., the mean. and logarithmic variance Based on the calculation and standard deviation Initial samples are sampled from a Gaussian distribution. ; ； S2-22, multi-level affine coupling transformation: An initial sample is transformed into a sample of complex distribution by a series of affine coupling layers Each affine coupling layer implements an affine transformation of part of the input sample.​ S2-23: transform the samples By limiting the range of the action space through the tanh function, the target action is obtained ; S2-24: Calculate the strategy density using the following formula : ； wherein is the initial policy density, is the final output vector after the multi-layer transformation, is the absolute value of the Jacobian determinant of the total transformation; denotes the sample points after the multi-stage transformation, L denotes the number of stages of transformation, denotes the final sample points after the transformation of all stages; Multiply the logarithmic Jacobian determinant of each layer to obtain the overall log-likelihood, and optimize the log-likelihood to maximize the probability of the sample under the model; ； wherein log10 of the final strategy density, is the cumulative log Jacobian determinant of each layer; S2-25: calculate the target Q value: ； where r represents the current state and action yields a reward; is a discount factor that measures the importance of future rewards; is the future expected return evaluated by the target Q network; is a temperature parameter that controls the influence of the entropy term; is the entropy used to encourage exploration of the policy; denotes the next state reached after performing action denotes the next state reached after performing action denotes the action sampled from the policy is a probability distribution function that assigns a probability value to each possible action in state . Step S2-22 is specifically: S2-221, input segmentation and parameter calculation: The initial sample is The first half dimension is denoted as The second half dimension is denoted as The scaling parameter ( ) and the offset parameter ( ) are computed using a fully connected network; S2-222, affine transformation: right The formula for affine transformation is: ； wherein denotes the element-wise multiplication, exp denotes the element-wise exponential function; Final transformed output : ； S2-223, multi-layer superposition; The output of S2-222 is taken as input to the next layer The multiple affine coupling layers are stacked one by one as the input to the next layer; the process of steps S2-221 and S2-222 is repeated to convert the sample into a complex distribution through multi-level transformation to obtain a standardized flow output .

2. The method of claim 1, wherein, Step S1 is specifically: Step S1-1, establish state and action space: All the environment information elements observed by the household energy management system agent at time t, including: photovoltaic power generation unit output power , electricity purchase price , electricity sales price , fixed load demand power , state of charge of energy storage system ESS , outdoor temperature , state tuple of electric vehicles , state tuple of heating, ventilation and air conditioning , and then get the state space, define the state at time t as: ； State space Some elements in the state space have inherent stochastic properties, independent of the agent's actions; the uncontrollable state space includes , the controllable state space includes ; The scheduling decision variables of the agent include the charging and discharging power of the energy storage system ESS the exchange power with the main grid the power of the heating, ventilation and air conditioning the power of the electric vehicle ; which satisfy the following power balance constraints: + + ； wherein, is the power purchased from the main grid, is the power sold to the main grid, is the discharging power of the ESS at time t, is the charging power of the energy storage system ESS at time t; Step S1-2, establish the action space of the agent, define the action taken at time t as: ; Step S1-3, set the reward function: According to the total energy cost desired, the base electricity reward of the agent at time t is is represented as follows: ； wherein, and for controlling the trade-off between the order of magnitude and the minimization of the penalty due to the imbalance of the operating costs and the processing power, represents the depreciation degradation cost of the battery, represents the operating cost of the energy storage system ESS, represents the power imbalance at time t, represents the inhabitant thermal discomfort penalty term; Step S1-4, establish a set of Markov decision processes: For a specific Markov decision process, the state at time t+1 and the reward obtained at time t depend entirely on the state at time t and the action taken at time t, and are independent of all states before time t, that is, satisfy the following relationship: ； wherein represents the conditional probability distribution of a given current state and action ; Each configuration of the home micro-grid corresponds to a Markov decision process, and a set of Markov decision processes is finally established.

3. The method of claim 2, wherein, The power imbalance at time t in step S1 is defined as: 。 4. The method of claim 2, wherein, The household thermal discomfort penalty term in step S1 The expression is as follows: ； wherein, represents a set temperature and the indoor temperature difference a threshold value, represents a thermal comfort weight factor.

5. The method of claim 1, wherein, The basic assumption of the meta-reinforcement learning application in step S3 considers different device parameters of the home micro-grid, and the tasks suitable for meta-reinforcement learning training and meta-reinforcement learning testing conform to the same task distribution; ； ； wherein represents a task, i.e. a home microgrid topology, represents a set of tasks for training, i.e. a set of home microgrid topologies used to train the parameters; represents a set of tasks for testing, i.e. a set of home microgrid topologies used to verify the training; represents a probability distribution over tasks; The objective function is defined as follows: ； wherein denotes the updated parameters on the training data; denotes the loss function on the training data; denotes the loss function on the test data; In the inner loop, for task i, compute updated parameters Loss on this task : ； wherein denotes the state-action distribution defined by the policy denotes the state s and the action a sampled from the state-action distribution defined by the policy denotes the reward obtained by taking action a in state s; denotes a discount factor used for time discounting when calculating cumulative rewards, denotes the value function in state s'; denotes the expected value of the value function for the next state s'; denotes the logarithm of the probability of taking action a in state s. Then the loss of all tasks is aggregated to get the meta-learning loss : ； In outer loop, parameters are optimized using gradient descent method minimizing the meta-learning loss: ； wherein, is a learning rate of meta-learning, controlling the step size of gradient descent update; denotes using parameters to parameterize a value function; denotes a gradient of the meta-learning loss function with respect to the value function . parameters of the policy network, the Q network and the target Q network are simultaneously updated in each outer loop iteration. parameters of the policy network, the Q network and the target Q network are simultaneously updated in each outer loop iteration.​ 6. A computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of any one of claims 1-5.

7. A computing device comprising a memory and a processor, the memory having stored therein executable code, the processor implementing the method of any one of claims 1-5 when executing the executable code.

Images