Metareinforcement learning-based household micro-grid energy optimization control method

Abstract

The invention discloses a household micro-grid energy optimization control method based on meta reinforcement learning, which is used for researching the joint coordination problem of demand response containing renewable energy resource consumption and alternating current optimal power flow under the conditions of no building thermal power model and guarantee of user comfort demand. Modeling the problem of minimizing the operation cost of the household micro-grid as a Markov decision process; secondly, a Soft Actor Critic algorithm which is integrated with Normalization Flow is constructed, and then, a soft Actor Critic algorithm which is integrated with Normalization Flow is constructed; thirdly, adding a meta-learning method into the constructed algorithm training; the method has the advantages that on the premise that the comfort requirement of a user is guaranteed, the method has the larger running cost saving potential, and the method can quickly adapt to different family micro-grid topological structures and quickly adapt to new tasks.

Description

Technical Field

[0001] The invention relates to a household microgrid energy optimization control method based on meta-reinforcement learning, and belongs to the technical field of intelligent buildings. Background Art

[0002] Online energy management of home microgrids is a key means to meet the energy comfort needs of residents and optimize the operating costs of home microgrids. However, due to the high nonlinearity of thermodynamics, the high heterogeneity of building stock, and the different types and parameters of household appliances, traditional energy management relies on model accuracy and specific case environments, resulting in low reliability and poor generalization ability.

[0003] In response to the above problems, deep reinforcement learning has gradually become a popular solution based on its model-free advantage to avoid complex modeling of home microgrid energy systems. By collecting real-time information to perform online energy management on home microgrids, it can make full use of renewable energy and demand-side resources at the residential level.

[0004] However, traditional reinforcement learning is not capable of representing complex distributions and cannot fully represent the complexity and diversity of grid data in a home microgrid environment, resulting in inaccurate or biased models, and poor robustness when the environment changes. Therefore, how to improve online energy management strategies to adapt to energy optimization in home microgrids in complex environments and highly uncertain scenarios is a topic worthy of in-depth research. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a home microgrid energy optimization control method based on meta-reinforcement learning, so as to improve the scalability of the home microgrid, accelerate the learning speed of the energy management strategy, and enhance the robustness of the home microgrid when the environmental uncertainty changes.

[0006] In a first aspect, the present invention provides a method for optimizing energy control of a household microgrid based on meta-reinforcement learning, which is applied to a household microgrid including photovoltaic power generation, and at least comprises the following steps:

[0007] S1: Establish a Markov decision model MDP based on home microgrid energy management.

[0008] Determine the state of the Markov decision process based on the mathematical model of energy management of the home microgrid;

[0009] Confirm the actions of the Markov decision process based on the working principles and characteristics of energy storage systems ESS, electric vehicles, HVAC and other equipment;

[0010] Determine the reward function of the Markov decision process based on the goal of minimizing residential energy costs while meeting the comfort of residents in the home microgrid;

[0011] According to different household microgrid topologies, a collection of Markov decision processes is established.

[0012] S2: Based on the Markov decision model MDP of home microgrid energy management, a deep reinforcement learning algorithm NFSAC that introduces standardized flow is constructed, that is, a standardized flow (NormalizingFlow) is introduced into the standard deep reinforcement learning algorithm SAC.

[0013] The SAC algorithm is an off-policy actor-critic deep reinforcement learning algorithm under the maximum entropy reinforcement learning framework. The normalized flow strategy is introduced into the SAC algorithm to make the policy distribution closer to the Boltzmann distribution, so as to better capture the multi-peak behavior of the microgrid and enable the intelligent agent to learn better scheduling strategies in complex environments.

[0014] S3: Based on the constructed NFSAC algorithm, a meta-reinforcement learning real-time control algorithm is established, that is, meta-learning is introduced in NFSAC.

[0015] S4: Energy optimization of home microgrids using meta-reinforcement learning real-time control algorithm.

[0016] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described.

[0017] In a third aspect, the present invention provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the described method is implemented.

[0018] The invention provides a building microgrid online energy management method based on meta-reinforcement learning, which has the following beneficial effects:

[0019] 1. By simulating the uncertainty environment of building microgrids, the model is optimized to find an overall better strategy. While maximizing the reward function, the residential energy cost and the thermal discomfort and mileage anxiety penalties of residents are minimized.

[0020] 2. In order to solve the sparse reward problem in the microgrid environment and improve the algorithm stability, the normalized flow strategy is introduced into the SAC algorithm to make the strategy distribution closer to the Boltzmann distribution, thereby better capturing the multi-peak behavior of the microgrid.

[0021] 3. Compared with the existing deep reinforcement learning algorithm that is trained for a specific topology, the training speed of the present invention is significantly improved when facing a new home power grid environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0023] Figure 1 It is an energy management framework diagram of the building microgrid online energy management method based on meta-reinforcement learning used in the present invention;

[0024] Figure 2 is a flow chart of a method according to an embodiment of the present invention;

[0025] Figure 3 This is the scheduling result diagram of the method proposed in the present invention on a certain test day, where (a) is the fixed load and photovoltaic power generation diagram, (b) is the ESS charging and discharging, (c) is the EV charging, and (d) is the indoor and outdoor temperature when the HVAC is on and off. DETAILED DESCRIPTION

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0027] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a method for optimizing energy control of a household microgrid based on meta-reinforcement learning, and the method includes the following steps:

[0028] S1: Establish a Markov decision model MDP based on home microgrid energy management.

[0029] According to the mathematical model of home microgrid energy management, the state of the Markov decision process is determined; according to the working principles and characteristics of energy storage systems ESS, electric vehicles, HVAC and other equipment, the actions of the Markov decision process are confirmed; according to the goal of minimizing residential energy costs while satisfying the comfort of residents in the home microgrid, the reward function of the Markov decision process is determined; according to different home microgrid topologies, a collection of Markov decision processes is established.

[0030] S2: Based on the Markov decision model MDP of home microgrid energy management, a deep reinforcement learning algorithm NFSAC that introduces standardized flow is constructed, that is, a standardized flow (NormalizingFlow) is introduced into the standard deep reinforcement learning algorithm SAC.

[0031] The SAC algorithm is an off-policy actor-critic deep reinforcement learning algorithm under the maximum entropy reinforcement learning framework. The normalized flow strategy is introduced into the SAC algorithm to make the policy distribution closer to the Boltzmann distribution, so as to better capture the multi-peak behavior of the microgrid and enable the intelligent agent to learn better scheduling strategies in complex environments.

[0032] S3: Based on the constructed NFSAC algorithm, a meta-reinforcement learning real-time control algorithm is established, that is, meta-learning is introduced in NFSAC.

[0033] Meta-reinforcement learning is applied in the basic assumption that different device parameters of the home microgrid are considered, and the tasks suitable for meta-reinforcement learning training and meta-reinforcement learning testing follow the same task distribution.

[0034]

[0035] where τ represents the task, i.e., a specific home microgrid topology, represents the set of tasks used for training, i.e., the set of home microgrid topologies used to train parameters; represents the set of tasks for testing, that is, the set of home microgrid topologies used to test the training effect. p(τ) represents the probability distribution of the task. Its objective function can be defined as follows:

[0036]

[0037] In the formula, Represents the parameters updated on the training data; represents the loss function on the training data; Represents the loss function on the test data.

[0038] S4: Energy optimization of home microgrids using meta-reinforcement learning real-time control algorithm.

[0039] Furthermore, in step S1, the construction of the Markov strategy model includes the following process:

[0040] S1-1: Establish state and action space.

[0041] All environmental information elements observed by the home energy management system agent during period t include: the output power of the photovoltaic power generation unit Electricity purchase price Electricity price Fixed load power demand State of charge of energy storage system ESS Outdoor temperature The state tuple of the electric car HVAC status tuple Then we get the state space and define the state at time t as:

[0042]

[0043] Status t Some elements in have inherent random characteristics and are not affected by the actions of the agent, which is an uncontrollable state space, including The controllable state space includes

[0044] The scheduling decision variables of the intelligent agent include the charging and discharging power of the ESS Power exchange with the main grid HVAC power Electric car power The following power balance constraints are satisfied between them:

[0045] in, To purchase electricity from the main grid, To sell electricity to the main grid, is the discharge power of ESS at time t, is the charging power of ESS at time t.

[0046] S1-2: Establish action space.

[0047] The action space contains all possible actions that the agent can choose in each state. t Not considered The action a taken by the agent at time t t Defined as:

[0048] S1-3: Set the reward function: According to the expected total energy cost, the basic electricity reward r of the agent in period t is t It is expressed as follows:

[0049]

[0050] Among them, σ 1 and σ 2 Used to control the order of magnitude and minimize the trade-off between running cost and the penalty caused by handling power imbalance, represents the depreciation cost of the battery, Indicates the operating cost of ESS.

[0051] ΔP t The power imbalance at time t is defined as

[0052]

[0053] Added a resident heat discomfort penalty to the base reward The expression is as follows

[0054]

[0055] in, Indicates the set temperature With indoor temperature Difference The threshold value, ω 1 represents the thermal comfort weight factor;

[0056] S1-4: Build an ensemble of Markov decision processes.

[0057] For a specific Markov decision process, the state at time t+1 and the reward at time t depend entirely on the state at time t and the action taken at time t, and have nothing to do with all the states before time t, that is, they satisfy the following relationship:

[0058] (s t+1 ,r t )~p(·|s t ,a t )

[0059] where p(·|s t ,a t ) indicates that given the current state s t and action a t The conditional probability distribution of .

[0060] In the context of home microgrids, when actions such as switching load devices, adjusting the working mode of the energy storage system, and changing the working state of the photovoltaic power generation system occur, the state transition probability of the system will change. Therefore, each configuration of a home microgrid corresponds to a specific Markov decision process, and finally a set of Markov decision processes is established, and the probability distribution of all possible topological structures during home microgrid training is recorded as p(τ).

[0061] Furthermore, in step S2, the updating of NFSAC includes the following process:

[0062] S2-1: Sample a batch of experience samples from the experience replay pool D (s t ,a t ,r t ,s t+1 ) i＝1,2...,N ,s t 、a t 、r t 、s t+1 Respectively represent the state, action, reward and action a of the agent at time t. tThe new state transferred to after that; N is the number of samples in this batch;

[0063] S2-2: For each experience sample, calculate the target Q value y.

[0064] S2-21: Initial sample generation. The current home microgrid state s t Input to the Actor network to obtain the parameters of the initial Gaussian distribution, namely the mean μ and logarithmic variance logσ. Based on the calculated mean μ and standard deviation σ, sample the initial sample z from the Gaussian distribution 0 .

[0065] z 0 ~N(μ,σ 2 )

[0066] S2-22: Multi-level affine coupling transformation. The multi-level affine coupling transformation transforms the initial sample z through a series of affine coupling layers. 0 Transformed into a sample z of complex distribution k , each affine coupling layer implements an affine transformation of part of the input samples.

[0067] S2-221: Input segmentation and parameter calculation. 0 The first half dimension is denoted as The second half dimension is denoted as Use a fully connected network to compute the scaling parameters and offset parameters

[0068]

[0069] Where W h ,W s ,W s is the weight matrix; b h ,b s ,b t is the bias vector.

[0070] S2-222: Reflection transformation. Perform reflection transformation, the formula is

[0071]

[0072] In the formula, ⊙ represents the element-wise product, and exp represents the element-wise exponential function. The transformed output is

[0073]

[0074] S2-223: Multi-layer stacking. Use the output of S2-222 as the input of the next layer, and stack multiple affine coupling layers layer by layer. Repeat the process of steps S2-221 and S2-222, and transform the sample into a complex distribution through multi-level transformation to obtain the standardized flow output z k .

[0075] S2-23: The transformed sample z k The target action is obtained by limiting the range of the action space through the tanh function. Right now

[0076] S2-24: Calculating Strategy Density

[0077] Using the formula Calculate the strategy density. In the formula, p(z) is the final strategy density, p(z 0 ) is the initial density, is the final output vector after multi-layer transformation, is the absolute value of the Jacobian of the total transformation.

[0078] The log-Jacobian determinant of each layer is accumulated to obtain the overall log-likelihood, and the log-likelihood is optimized to maximize the probability of the sample under the model.

[0079]

[0080] Where log p(z) is the logarithm of the final strategy density, is the cumulative log-Jacobian of each layer.

[0081] S2-25: Calculate the target Q value.

[0082]

[0083] Where r is the immediate reward; γ is the discount factor, which is used to measure the importance of future rewards; is the expected future return of the target Q network evaluation; α is the temperature parameter that controls the influence of the entropy term; is entropy, which is used to encourage the exploratory nature of the strategy.

[0084] S2-3: Update the Q network. Update the parameters of the Q network by minimizing the mean square error between the current Q network and the target value y.

[0085]

[0086] In the formula, N is the number of samples in this batch; is the Q value output by the current Q network; y is the target Q value.

[0087] S2-4: Update the policy network. The action generation and density calculation are consistent with S2-21 to S2-24. Minimize the policy loss function and update the parameters θ of the policy network.

[0088]

[0089] Where α represents the temperature parameter, logπ θ (a|s) represents entropy, Indicates the Q value of the current Q network output;

[0090] S2-5: Update the temperature parameter α. Minimize the loss function of the temperature parameter so that the entropy of the strategy is close to the target entropy Thereby dynamically adjusting the balance between exploration and exploitation.

[0091]

[0092] In the formula, is the target entropy, which is used to control the exploration intensity; α represents the temperature parameter;

[0093] S2-6: Update the target Q network. Use the soft update method to update the parameters φ' of the target Q network. i To the current Q network parameter φ i Gradually move closer and stabilize the training process.

[0094] φ′ i ←τφ i +(1-τ)φ′ i

[0095] Where τ is the soft update coefficient.

[0096] Furthermore, in step S3, the combination of meta-learning and NFSAC algorithm includes the following process:

[0097] The goal of meta-learning is to find a set of initial parameters that can quickly adapt to new tasks through a small amount of gradient updates. Unlike traditional deep reinforcement learning algorithms that train a single MDP, meta-reinforcement learning requires training a large number of MDPs that follow the same distribution to gain experience and learn how to quickly adapt to new tasks.

[0098] The interaction process between the agent and the environment in meta-reinforcement learning includes inner loops and outer loops. After initializing the model parameters, in each inner loop, the agent trains for a specific MDP and calculates the task loss J i (θ i ), and the meta-loss is summarized in the outer loop to update the meta-parameters.

[0099] Specifically, in the inner loop, a task is randomly sampled from the task sample set for testing. For each round in this test, the calculation process is consistent with that described in S2. At the same time, for task i, the updated parameter θ is calculated i The loss on this task J i (θ i ):

[0100]

[0101] in Represents the strategy π′ i The state-action distribution defined The sample state s and action a are in the state s; r(s,a) represents the reward obtained by taking action a in state s; γ represents the discount factor, which is used to calculate the time discount when accumulating rewards. represents the value function in state s'; Indicates the next state s ′ The value function expectations; represents the logarithm of the probability of taking action a in state s;

[0102] Then sum up the losses of all tasks to get the meta-learning loss J meta (θ):

[0103]

[0104] In the outer loop, gradient descent is used to optimize the parameters θ to minimize the meta-learning loss:

[0105]

[0106] Among them, β is the learning rate of meta-learning, which controls the step size of gradient descent update; represents the value function parameterized by parameter θ; Represents the meta-learning loss function J meta (θ) About the value function The gradient of

[0107] In each outer loop iteration, the policy network π is updated simultaneously θ , Q network and target Q network Q φ The parameters of the model are adjusted to enable the model to quickly adapt to new tasks. When the training is completed, the model has a certain generalization ability and can quickly adapt to new microgrid operation tasks through a small amount of sample training. This enables the microgrid control system to effectively manage and optimize energy resources when facing different operating conditions and requirements.

[0108] For each period of the selected test day, according to the currently observed initial state of the environment s t , using the optimal strategy learned by the trained meta-reinforcement learning real-time control algorithm to determine the corresponding control action a t . Then, get the reward r t And observe that the environment state transfers to s t+1 This cycle continues until the scheduling tasks for all test periods are completed online.

[0109] Further, the step S4 specifically comprises:

[0110] S41: Deploy model: Deploy the trained model to the home microgrid agent.

[0111] S42: Initial Observations: Record the initial environmental conditions on the test day and the total test period.

[0112] S43: Strategy application: Use the model to develop an optimal energy management strategy and determine the best scheduling action based on the current environmental state.

[0113] S44: Feedback reception: The agent obtains the reward value and observes the environment state in the next period.

[0114] S45: Iterative optimization: Repeat the process of applying strategies and receiving feedback until the energy optimization of all test periods is completed.

[0115] from Figure 3 As can be seen from (a) and (b), the ESS absorbs as much excess PV power as possible during the peak period of PV output to avoid selling it. At the same time, during the peak hours (8:00-9:00 and 17:00-20:00) when the purchase price is high, the ESS chooses to discharge, which helps to reduce the purchase cost and effectively utilize the PV output. In addition, the ESS does not immediately charge a large amount when the SoC is at a low level, but moves the charging period to a period with lower electricity prices (22:00-3:00 the next day). As can be seen from Figure (c), the intelligent agent does not immediately charge the electric vehicle after it is connected, but moves the charging period of the electric vehicle to the valley period (22:00-7:00 the next day).

[0116] Combined with Figure (d), it can be seen that due to the low temperature between 2:00 and 7:00, in order to reduce energy costs, the HVAC system did not start immediately, but chose to delay the start to save costs. On the other hand, it is worth noting that the HVAC system always keeps the indoor temperature within the comfortable range to ensure user comfort.

[0117] This charging and discharging strategy and HVAC operation mode show an optimized energy management solution, which effectively reduces the overall energy consumption cost while ensuring efficient use of energy and user comfort experience. This strategy not only makes rational use of PV power generation, but also reduces the overall cost by shifting peak power consumption and reasonably controlling HVAC operation.

[0118] The above embodiments are not limitations of the present invention, and the present invention is not limited to the above embodiments. As long as the requirements of the present invention are met, they belong to the protection scope of the present invention.

Claims

1. A method for optimizing energy control of a household microgrid based on meta-reinforcement learning, characterized in that The steps include: Step S1, establishing a Markov decision model MDP based on home microgrid energy management; Step S2, based on the Markov decision model MDP of home microgrid energy management, construct a deep reinforcement learning algorithm NFSAC that introduces standardized flow; Step S3: establishing a meta-reinforcement learning real-time control algorithm based on the constructed NFSAC algorithm; Step S4: Utilize the meta-reinforcement learning real-time control algorithm to optimize the energy of the home microgrid.

2. The method according to claim 1, characterized in that: Step S1 specifically includes: Step S1-1, establish state and action space: All environmental information elements observed by the home energy management system agent at time t, including: the output power of the photovoltaic power generation unit Electricity purchase price Electricity price Fixed load power demand State of charge of energy storage system ESS Outdoor temperature The state tuple of the electric car HVAC status tuple Then we get the state space and define the state at time t as: State space s t Some elements in have inherent random properties and are not affected by the agent's actions; the uncontrollable state space includes The controllable state space includes The scheduling decision variables of the intelligent agent include the charging and discharging power of the energy storage system ESS Power exchange with the main grid HVAC power Electric car power The following power balance constraints are satisfied between them: in, To purchase electricity from the main grid, To sell electricity to the main grid, is the discharge power of ESS at time t, is the charging power of the energy storage system ESS at time t; Step S1-2: Establish the action space of the agent and define the action taken at time t as: Step S1-3, set reward function: According to the expected total energy cost, the basic electricity reward r of the agent at time t is t It is expressed as follows: Among them, σ1 and σ2 are used to control the order of magnitude and minimize the trade-off between the running cost and the penalty caused by the processing power imbalance, represents the depreciation cost of the battery, Represents the operating cost of the energy storage system ESS, ΔP t represents the power imbalance at time t, It indicates the penalty item for thermal discomfort of residents; Step S1-4: Establish a set of Markov decision processes: For a specific Markov decision process, the state at time t+1 and the reward at time t depend entirely on the state at time t and the action taken at time t, and have nothing to do with all the states before time t, that is, they satisfy the following relationship: (s t+1 ,r t )~p(·|s t ,a t ) where p(·|s t , a t ) indicates that given the current state s t and action a t The conditional probability distribution of ; Each configuration of a household microgrid corresponds to a Markov decision process, and finally a collection of Markov decision processes is established.

3. The method according to claim 2, characterized in that: The power imbalance ΔP at time t in step S1 t Defined as:

4. The method according to claim 2, characterized in that: Resident thermal discomfort penalty item in step S1 The expression is as follows: in, Indicates the set temperature With indoor temperature Difference The threshold of ω1 represents the thermal comfort weight factor.

5. The method according to claim 1, characterized in that: Step S2 specifically includes: S2-1. Sample a batch of experience samples (s t , a t , r t ,s t+1 ) i＝1，2...，N ,s t 、a t 、r t 、s t+1 Respectively represent the state, action, reward and action a of the agent at time t. t The new state transferred to after that; N is the number of samples in this batch; S2-2, for each experience sample, calculate the target Q value y; S2-3: Update Q network: The Q network is used to estimate the Q value, i.e. the expected reward, for a given state and action pair. The parameters of the Q network are updated by minimizing the mean square error between the current Q network and the target Q value y. In the formula, N is the number of samples in this batch; is the Q value output by the current Q network; y is the target Q value; s and a represent the state and action respectively; S2-4: Update strategy network: The policy network is used to generate actions. Its goal is to generate actions that maximize the cumulative reward while maintaining sufficient entropy to promote exploration. π (θ), update the parameters θ of the policy network; Where α represents the temperature parameter, logπ θ (a|s) represents entropy, Indicates the Q value of the current Q network output; S2-5: Update temperature parameter α: By minimizing the loss function J(α) of the temperature parameter, the entropy of the strategy is made close to the target entropy Thus dynamically adjusting the balance between exploration and exploitation; In the formula, is the target entropy, which is used to control the exploration intensity; α represents the temperature parameter; S2-6: Update target Q network: Using the soft update method, the parameters φ' of the target Q network are i To the current Q network parameter φ i Gradually move closer and stabilize the training process; f′ i ←tf i +(1-τ)φ′ i Where τ is the soft update coefficient.

6. The method according to claim 5, characterized in that: Step S2-2 specifically includes: S2-21: Initial sample generation: The current state of the household microgrid s t Input to the Actor network to obtain the parameters of the initial Gaussian distribution, namely the mean μ and logarithmic variance logσ; based on the calculated μ and standard deviation σ, sample the initial sample z0 from the Gaussian distribution; z0~N(μ,σ 2 ) S2-22, Multi-level affine coupling transformation: The initial sample z0 is transformed into a complex distributed sample z through a series of affine coupling layers k , each affine coupling layer implements the affine transformation of part of the input samples; S2-23: The transformed sample z k The target action is obtained by limiting the range of the action space through the tanh function. S2-24: Use the following formula to calculate the strategy density p(z): Where p(z0) is the initial strategy density, is the final output vector after multi-layer transformation, is the absolute value of the Jacobian of the total transformation; z L represents the sample point after multi-level transformation, L represents the number of transformation levels, and z represents the final sample point after all levels of transformation; Multiply the log-Jacobian determinant of each layer to obtain the overall log-likelihood, and optimize the log-likelihood to maximize the probability of the sample under the model; Where log p(z) is the logarithm of the final strategy density, To accumulate the logarithmic Jacobian of each layer; S2-25: Calculate the target Q value: In the formula, r represents the current state s′ and action The reward obtained under ; γ is the discount factor, which is used to measure the importance of future rewards; is the expected future return of the target Q network evaluation; α is the temperature parameter that controls the influence of the entropy term; Entropy is used to encourage the exploratory nature of the strategy; s′ indicates execution action The next state reached after represents the actions sampled from the policy π, π θ Is the probability distribution function, indicating that for each possible action under a given state s′ Assign a possible value.

7. The method according to claim 5, characterized in that: Step S2-22 is specifically: S2-221, Input segmentation and parameter calculation: The first half dimension of the input sample z0 is recorded as The second half dimension is denoted as Computing scaling parameters using a fully connected network and offset parameters S2-222, reflection transformation: right Perform reflection transformation, the formula is: In the formula, ⊙ represents the element-wise product, and exp represents the element-wise exponential function; Finally, the transformed output z1 is obtained: S2-223, multi-layer stacking; The output z1 of S2-222 is used as the input of the next layer, and multiple affine coupling layers are stacked layer by layer; the process of steps S2-221 and S2-222 is repeated, and the samples are converted into complex distributions through multi-level transformation to obtain the standardized flow output z k .

8. The method according to claim 1, characterized in that: In step S3, the meta-reinforcement learning is applied to the basic assumption that the parameters of different devices in the home microgrid are taken into consideration, and the tasks suitable for meta-reinforcement learning training and meta-reinforcement learning testing follow the same task distribution; where τ represents the task, i.e., the home microgrid topology, represents the set of tasks used for training, i.e., the set of home microgrid topologies used to train parameters; represents the set of tasks used for testing, that is, the set of home microgrid topologies used to test the training effect; p(τ) represents the probability distribution of the task; The objective function is defined as follows: In the formula, Represents the parameters updated on the training data; Represents the loss function on the training data; Represents the loss function on the test data; In the inner loop, for task i, the updated parameter θ is calculated i The loss on this task J i (θ i ): in Represents the strategy π′ i The state-action distribution defined The sample state s and action a are in the state s; r(s, a) represents the reward obtained by taking action a in state s; γ represents the discount factor, which is used to calculate the time discount when accumulating rewards. represents the value function in state s'; Represents the value function for the next state s′ expectations; represents the logarithm of the probability of taking action a in state s; Then sum up the losses of all tasks to get the meta-learning loss J meta (θ): In the outer loop, gradient descent is used to optimize the parameters θ to minimize the meta-learning loss: Among them, β is the learning rate of meta-learning, which controls the step size of gradient descent update; represents the value function parameterized by parameter θ; Represents the meta-learning loss function J meta (θ) About the value function The gradient of In each outer loop iteration, the policy network π is updated simultaneously θ , Q network and target Q network Q φ Parameters.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

10. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 8 is implemented.

Images