Multi-family micro-grid energy regulation and control method, system, equipment and medium

Abstract

The invention discloses a community family micro-grid fast adaptation energy regulation and control method based on element reinforcement learning, and the method comprises the steps: abstracting different family micro-grids into element tasks, and constructing a unified state space, motion space and physical constraint model; the method comprises the following steps: constructing a double-layer optimization structure containing a meta-policy network and a meta-value network based on multi-task distribution, and carrying out meta-training through internal loop rapid adaptation and external loop cross-task optimization to obtain initialization parameters with generalization ability; when a new family micro-grid is accessed, based on the initialization parameters, strategy rapid adaptation can be completed only through a small amount of real-time data, and energy regulation and control actions are output. According to the method, the multi-microgrid common characteristics can be effectively extracted, the strategy cross-scene adaptation capability is improved, the new microgrid deployment period is shortened, the training cost is reduced, meanwhile, the stability and the performability of the regulation and control strategy are guaranteed, the optimal electricity cost and the maximum consumption of renewable energy sources are achieved, and the method is suitable for community diversified family microgrid scenes.

Description

Technical Field

[0001] This invention belongs to the field of energy regulation technology for home microgrids, specifically relating to an energy regulation method, system, device, and medium for multi-home microgrids. Background Technology

[0002] With the rapid adoption of distributed photovoltaic (PV) systems, home energy storage, and electric vehicles in communities, home microgrids have become a key terminal unit in the new power distribution system. Currently, achieving optimal electricity costs for home microgrids while maximizing the absorption of renewable energy, while ensuring user comfort, is a core technological requirement for community energy management.

[0003] Existing energy optimization methods for residential microgrids largely rely on deterministic models or single-task reinforcement learning methods. Deterministic models require precise environmental modeling and load forecasting capabilities, making them ill-suited to the dynamic characteristics of residential microgrids, which include diverse equipment types, highly real-time and stochastic operational states, and are influenced by multiple external factors such as weather, electricity prices, and user behavior. While single-task reinforcement learning can learn strategies through interaction in complex environments, it suffers from large sample requirements, high training costs, and weak transferability. It cannot adapt to the significant differences among households in load composition, energy storage scale, electric vehicle integration modes, and photovoltaic configurations, resulting in poor generalization ability of strategies across microgrid scenarios. When a new residential microgrid is connected, traditional deep reinforcement learning requires retraining or lengthy fine-tuning, hindering rapid strategy deployment. Furthermore, existing methods struggle to effectively integrate microgrid physical constraints, leading to insufficient policy executability and poor stability.

[0004] Therefore, there is an urgent need for an energy regulation scheme that can systematically learn the commonalities of multiple microgrid structures, generalize strategies across household scenarios, and quickly adapt to the operating characteristics of new household microgrids under limited sample conditions, in order to solve existing technical bottlenecks and meet the needs of efficient operation and maintenance of community household microgrids. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the technical problem that deterministic models or single-task reinforcement learning methods in the prior art cannot quickly adapt to the operating characteristics of new home microgrids, thereby providing an energy regulation method, system, device and medium for multi-home microgrids.

[0006] A method for energy regulation in a multi-family microgrid includes the following steps: Each home microgrid defines a task, which includes a task dataset, a uniform state vector, a uniform action vector, and an immediate reward; Each task collects high-dimensional time-series data to form a task dataset; the task dataset includes the current operating state vector of the home microgrid, energy regulation actions, immediate rewards, and the state at the next moment; Construct a unified state space, which includes device profile vectors and operating state vectors; Construct a unified action space, which includes energy regulation actions; The reward function includes electricity cost and grid power purchase items, and the task calculates the immediate reward based on the reward function; Construct a meta-policy network, which includes meta-policy parameters, a unified state vector and a task embedding vector as inputs, and an energy regulation action as output. Construct a value network, which includes a value function for evaluating the expected returns of the meta-policy network; For each task, the meta-strategy parameters are updated and fine-tuned through an inner loop, and the task adaptation parameters are updated accordingly. Based on the results of the inner loop update, the meta-strategy parameters are updated through the outer loop. When a new home microgrid is connected, a new task is defined, and the task adaptation parameters are obtained through an inner loop update based on the meta-policy parameters. Based on the task adaptation parameters, the energy regulation action is obtained through the policy network to perform energy regulation of the home microgrid.

[0007] Furthermore, the high-dimensional time-series data includes the status of the home microgrid, energy regulation actions, and instant rewards.

[0008] Furthermore, the device profile vector of the unified state space includes the energy storage capacity of the household microgrid, the energy storage charging and discharging efficiency, the electric vehicle battery capacity, and the maximum available photovoltaic power; the operating state vector of the unified state space includes the total household load power, the energy storage state of charge, the electric vehicle battery state, the photovoltaic output, and the real-time electricity price.

[0009] Furthermore, the energy regulation actions include energy storage charging and discharging power adjustment, electric vehicle charging and discharging adjustment, and load time shift adjustment.

[0010] Furthermore, the meta-policy network structure includes an input layer, a hidden layer, and an output layer; the dimension of the input layer is the sum of the number of features of the unified state vector and the task embedding vector; the hidden layer includes two fully connected layers; and the output layer outputs action distribution parameters.

[0011] Furthermore, in the inner loop update, the meta-policy parameters are updated and fine-tuned based on the loss function to obtain the task adaptation parameters; the loss function includes the product of the logarithm of the action selection probability and the advantage function; the advantage function is expressed as the difference between the action value and the state value.

[0012] Furthermore, in the outer loop update, the task loss is calculated based on the updated task adaptation parameters; and the meta-policy parameters are optimized and updated based on the task loss corresponding to the task adaptation parameters.

[0013] A multi-family microgrid energy regulation system, used to achieve multi-family microgrid energy regulation according to the above-mentioned multi-family microgrid energy regulation method, includes: The meta-policy network takes a unified state vector and task embedding vector as input and outputs energy regulation actions. The task embedding module is used to obtain the device profile vector and running status vector of the task and form the task embedding vector. The inner loop parameter update module is used to fine-tune the meta-strategy parameters and update the task adaptation parameters for each task through an inner loop. The outer loop meta-update module updates the meta-strategy parameters through the outer loop based on the result of the inner loop update.

[0014] A computer device includes a processor and a memory connected to the processor. The memory stores one or more programs that are executed by the processor to implement the steps in the above-described multi-home microgrid energy regulation method.

[0015] A computer-readable storage medium storing one or more programs, which are executed by a processor to implement the steps in the multi-home microgrid energy regulation method described above.

[0016] Beneficial effects: This invention improves the generalization ability of policies to adapt to diverse home microgrid scenarios; by constructing a unified state space, action space and physical constraint model, different home microgrids are abstracted into meta-tasks and formed into task distributions, and combined with the meta-training mechanism to extract common structural features of multiple microgrids, it solves the problem of weak generalization ability of traditional single-task reinforcement learning strategies across home microgrid scenarios, and can adapt to the needs of home microgrids with different load compositions, energy storage scales, electric vehicle access modes and photovoltaic configurations.

[0017] This invention obtains generalizable initial parameters through meta-training. When a new home microgrid is connected, only a small amount of real-time running data is needed. Policy adaptation can be completed through fine-tuning in the inner loop without retraining the model. This significantly shortens the policy deployment time and solves the shortcomings of traditional deep reinforcement learning, which requires retraining and has a long fine-tuning cycle when connecting to new scenarios. It also reduces the system operation and maintenance and training costs.

[0018] This invention, through a two-layer optimization structure of meta-policy network and value network, ensures that the learned regulation strategy conforms to the physical laws of microgrid operation, avoids the problems of insufficient policy executability and poor stability caused by neglecting physical constraints in existing methods, and improves the robustness of energy regulation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic block diagram illustrating the main method steps of the present invention. Detailed Implementation

[0021] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0022] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0023] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to direct connection or indirect connection through an intermediate medium, or to the internal connection of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0024] Reference Figure 1 As shown, this embodiment provides a method for energy regulation of multi-family microgrids, including the following steps: Step S1: Construct a community-based family micro-network element task set and a unified environment model; Step S1.1: Each home microgrid defines an independent task, which includes a task dataset, a uniform state vector, a uniform action vector, and an immediate reward; Each task collects high-dimensional time-series data to form a task dataset; the task dataset includes the current operating state vector of the home microgrid, energy regulation actions, immediate rewards, and the state at the next moment; Step S1.2: Construct a unified state space, which includes device profile vectors and operating state vectors; Step S1.3: Construct a unified action space, which includes energy regulation actions; Step S1.4: Construct a physical constraint model for the home microgrid; Step S1.5: The reward function includes electricity cost and grid power purchase items. The task calculates the immediate reward based on the reward function. Step S2: Construct the meta-reinforcement learning agent structure and a two-layer optimization mechanism; Step S2.1: Construct a meta-policy network, which includes meta-policy parameters, input task unified state vector and task embedding vector, and output energy regulation action; Step S2.2: Construct a value network, which includes a value function for evaluating the expected returns of the meta-policy network; Step S2.3: Construct the task embedding module; Step S2.4: For each task, update and fine-tune the meta-strategy parameters through an inner loop, and update the task adaptation parameters; Step S2.5: Based on the result of the inner loop update, update the meta-policy parameters through the outer loop; Step S3: When a new home micronet is connected, a new task is defined, and the task adaptation parameters are obtained by updating the meta-policy parameters through an inner loop. Based on the task adaptation parameters, the energy regulation action is obtained through the policy network to perform energy regulation of the home micronet.

[0025] Specifically, in step S1.1, each home microgrid defines an independent task. The task includes a task dataset, a unified state vector, a unified action vector, and an immediate reward. The high-dimensional time-series data includes the state of the home microgrid, energy regulation actions, and immediate rewards.

[0026] Each task collects high-dimensional time-series data including electrical load, equipment, energy storage, photovoltaics, electricity prices, and user behavior, defined as: ; in, Indicates a family microgrid At any moment state, Indicates an energy regulation action. Indicates an immediate reward. Indicates task Total number of time steps.

[0027] In this embodiment, the device profile vector of the unified state space includes the energy storage capacity of the household microgrid, the energy storage charging and discharging efficiency, the electric vehicle battery capacity, and the maximum available photovoltaic power; the operating state vector of the unified state space includes the total household load power, the energy storage state of charge, the electric vehicle battery state, the photovoltaic output, and the real-time electricity price.

[0028] Unified state space Represented as: ; Specifically, device profile vectors Represented as: ; in, Indicates the energy storage capacity of a household microgrid. Indicates the energy storage charging / discharging efficiency. Indicates the battery capacity of an electric vehicle. This indicates the maximum available power of photovoltaics.

[0029] Running state vector Represented as: ; in, This refers to the total household load power (including adjustable load and rigid load). In the state of energy storage charge, Status of electric vehicle battery Contribute to photovoltaic power This refers to the real-time electricity price.

[0030] In this embodiment, the energy regulation action includes energy storage charging and discharging power adjustment, electric vehicle charging and discharging adjustment, and load time shift adjustment.

[0031] Specifically, the action space is represented as: ; in, Indicates the amount of energy storage charging and discharging power adjustment. This indicates the charging and discharging regulation of an electric vehicle. This indicates the load shift adjustment amount, which is the aggregated adjustable load change.

[0032] The constraints on the motion quantity are expressed as follows: ; ; ; Among them, the energy storage charging and discharging power adjustment is constrained between the maximum and minimum energy storage capacity, the electric vehicle charging and discharging adjustment is constrained between the maximum and minimum electric vehicle capacity, and the load time shift adjustment is constrained between 0 and 1 of the adjustable amount.

[0033] The physical constraint model for home microgrids includes energy storage state constraints and total home power balance; The energy storage state change equation is expressed as: ; The energy storage state change equation is used to calculate the remaining energy of the energy storage battery after each time step.

[0034] The energy storage state constraint is expressed as follows: ; The energy storage state constraint refers to the state of charge of the energy storage battery between being completely discharged and being completely fully charged. The battery status of an electric vehicle is represented as follows: ; The electric vehicle battery state is used to calculate the change in battery charge at each time step.

[0035] The total power balance of a household is expressed as: ; Household total power balance, that is, the power that the grid needs to provide = load - power change of energy storage - power change of electric vehicles - photovoltaic output.

[0036] Ultimately, the meta-task is represented as: ; in, Represents the task dataset, Indicates a unified state. Indicates a unified action. Represents a set of physical constraints. This indicates an immediate reward.

[0037] In this embodiment, factors such as electricity cost, renewable energy utilization, and user comfort are combined into the immediate reward for reinforcement learning. Let the electricity cost be expressed as:

[0038] Among them, home microgrids At any moment The power purchased by the power grid is The corresponding electricity price is .

[0039] To characterize the utilization of renewable energy by residential microgrids, a photovoltaic curtailment penalty is introduced. Let the residential microgrid at time [time value missing]... The available photovoltaic output is The actual absorption capacity is Then the power wasted is expressed as: ; For home microgrids at all times The actual photovoltaic power absorbed is determined by both the energy balance and energy regulation actions. The penalty for renewable energy utilization is defined as follows: ; in This is a weighting coefficient used to balance economic efficiency with renewable energy consumption targets.

[0040] Meanwhile, to reflect user comfort constraints, a load time shift and energy storage operation deviation penalty are introduced. Assume the adjustable load at time... The delay is Its allowed range is [0, The user comfort penalty term is defined as follows: ; Taking into account the impact of insufficient renewable energy absorption and load shifting on user comfort, a penalty term is defined. , represented as: + ; Considering factors such as deviation of energy storage state of charge from the expected range and excessive delay of adjustable load, a penalty term is introduced. Instant rewards are defined as: .

[0041] In step S2, the meta-policy network is defined as follows: ; in, This represents the set of learnable parameters for the meta-policy network. Indicates at time Output energy regulation action, Represents a unified state vector. Represents the task embedding vector, used to represent the task. Structural characteristics.

[0042] Specifically, the meta-policy network structure includes an input layer, a hidden layer, and an output layer; the dimension of the input layer is the sum of the number of features of the unified state vector and the task embedding vector; the hidden layer includes two fully connected layers; and the output layer outputs action distribution parameters.

[0043] In this embodiment, the dimension of the input layer is The hidden layer consists of two fully connected layers with the following numbers of neurons: The output layer outputs motion distribution parameters, including the mean. With variance .

[0044] The formula for calculating the policy network is expressed as follows: ; ; ; in, For trainable weights, Indicates the bias term. This represents a non-linear activation function.

[0045] Value networks are used to evaluate strategies in a task. The state value in the equation is represented as ;in, Represents the set of value network parameters. This represents the value function, used to estimate the state in a task. Expected returns The value network employs a multi-layer neural network and uses mean squared error as the optimization objective, as follows: ; .

[0046] Different residential microgrids differ in equipment configuration, load patterns, and photovoltaic characteristics. This invention introduces task embedding vectors. This serves as an important characteristic of the meta-task.

[0047] The task embedding vector is defined as: ; in, This indicates that the task is embedded in the network. Its network parameters; Indicates task Device profile vectors corresponding to home microgrids; Indicates task The state statistics during operation are represented by a set of time-series running states of the task. Obtained through time aggregation or feature encoding, it is used to reflect the typical operating characteristics of the home microgrid.

[0048] In this embodiment, during the inner loop update, the meta-policy parameters are updated and fine-tuned based on the loss function to obtain the task adaptation parameters; the loss function includes the product of the logarithm of the action selection probability and the advantage function; the advantage function is represented as the difference between the action value and the state value.

[0049] For the task Define the inner loop update as follows: ; in, Indicates the task The adapted parameters surface This indicates the learning rate of the inner loop.

[0050] The loss function adopts the advantage objective and is expressed as: ; The advantage is defined as follows: ; Updated This refers to the customized strategy for this task.

[0051] In this embodiment, during the outer loop update, the task loss is calculated based on the updated task adaptation parameters; and the meta-policy parameters are optimized and updated based on the task loss corresponding to the task adaptation parameters.

[0052] The outer loop update is defined as: ; in, This represents the outer loop learning rate. This indicates that the adapted parameters are used in the task. The loss calculated above.

[0053] In step S3, the meta-parameters are evaluated on multiple home microgrid tasks. An outer loop training process is performed to enable it to quickly adapt across tasks. The training process includes four stages: task sampling, environment interaction, inner loop update, and meta-gradient update.

[0054] Specifically, from the task set According to distribution Select a batch of tasks, and for each task In its corresponding home micronet environment, using meta-policy Perform energy regulation, combine the physical constraint model to obtain state transitions, and collect trajectory data. .

[0055] Based on the in-step iterative update formula, for each task Calculate loss And obtain the adaptation parameters for this task. ; The adaptation results of each task are summarized, the outer loop is invoked to update the rules, and the meta-policy parameters are adjusted. Performing gradient descent once improves its overall few-sample adaptability across all sampling tasks.

[0056] Repeatedly perform task sampling, inner loop and outer loop updates until the average reward of the meta-policy converges on each home microgrid task, thus obtaining the meta-policy initialization parameters with generalization ability.

[0057] When a new home micronet is connected to the community, based on the meta-policy parameters It can quickly adapt to new tasks by combining data from a small sample size.

[0058] During the normal operation of the new home microgrid, the state-action-reward-next state sequence is collected within a short time window. ; in It is far less than the number of steps required for a complete training session.

[0059] Meta-policy parameters obtained from meta-training As initial parameters, based on the inner loop update rules, for the new task Perform several gradient updates to obtain the adapted policy parameters. .

[0060] This embodiment also provides a multi-family microgrid energy regulation system for realizing multi-family microgrid energy regulation according to the above-described multi-family microgrid energy regulation method, including: The meta-policy network takes a unified state vector and task embedding vector as input and outputs energy regulation actions. The task embedding module is used to obtain the device profile vector and running status vector of the task and form the task embedding vector. The inner loop parameter update module is used to fine-tune the meta-strategy parameters and update the task adaptation parameters for each task through an inner loop. The outer loop meta-update module updates the meta-strategy parameters through the outer loop based on the result of the inner loop update.

[0061] This embodiment also provides a computer device, which includes a processor and a memory. The memory is connected to the processor and stores one or more programs. The one or more programs are executed by the processor to implement the steps in the above-described multi-family microgrid energy regulation method.

[0062] This embodiment also provides a computer-readable storage medium storing one or more programs that are executed by a processor to implement the steps in the multi-family microgrid energy regulation method described above.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for energy regulation in a multi-family microgrid, characterized in that, Includes the following steps: Each home microgrid defines a task, which includes a task dataset, a unified state vector, a unified action vector, and an immediate reward; each task collects high-dimensional time-series data to form a task dataset; the task dataset includes the current operating state vector, energy regulation action, immediate reward, and the state at the next moment of the home microgrid; A unified state space is constructed, which includes device profile vectors and operating state vectors; a unified action space is constructed, which includes energy regulation actions; the reward function includes electricity cost and grid power purchase items, and the task calculates the immediate reward based on the reward function; Construct a meta-policy network, which includes meta-policy parameters, a unified state vector and a task embedding vector as inputs, and an energy regulation action as output; construct a value network, which includes a value function to evaluate the expected return of the meta-policy network. For each task, the meta-policy parameters are fine-tuned through an inner loop, and the task adaptation parameters are updated. Based on the results of the inner loop update, the meta-policy parameters are updated through an outer loop. When a new home micronet is connected, a new task is defined, and the task adaptation parameters are obtained through an inner loop update based on the meta-policy parameters. Based on the task adaptation parameters, the energy regulation action is obtained through the policy network to perform energy regulation of the home micronet.

2. The energy regulation method for a multi-family microgrid according to claim 1, characterized in that, The high-dimensional time-series data includes the status of the home microgrid, energy regulation actions, and instant rewards.

3. The energy regulation method for a multi-family microgrid according to claim 1, characterized in that, The device profile vector of the unified state space includes the energy storage capacity of the household microgrid, the energy storage charging and discharging efficiency, the electric vehicle battery capacity, and the maximum available photovoltaic power; the operating state vector of the unified state space includes the total household load power, the energy storage state of charge, the electric vehicle battery state, the photovoltaic output, and the real-time electricity price.

4. The energy regulation method for a multi-family microgrid according to claim 1, characterized in that, The energy regulation actions include energy storage charging and discharging power adjustment, electric vehicle charging and discharging adjustment, and load time shift adjustment.

5. The energy regulation method for a multi-family microgrid according to claim 1, characterized in that, The meta-policy network structure includes an input layer, a hidden layer, and an output layer; the dimension of the input layer is the sum of the number of features of the unified state vector and the task embedding vector; the hidden layer includes two fully connected layers; and the output layer outputs action distribution parameters.

6. The energy regulation method for a multi-family microgrid according to claim 1, characterized in that, In the inner loop update, the meta-policy parameters are updated and fine-tuned based on the loss function to obtain the task adaptation parameters; the loss function includes the product of the logarithm of the action selection probability and the advantage function; the advantage function is expressed as the difference between the action value and the state value.

7. The energy regulation method for a multi-family microgrid according to claim 1, characterized in that, In the outer loop update, the task loss is calculated based on the updated task adaptation parameters; and the meta-policy parameters are optimized and updated based on the task loss corresponding to the task adaptation parameters.

8. A multi-family microgrid energy regulation system, characterized in that, The method for implementing energy regulation of a multi-family microgrid according to any one of claims 1-7 includes: The meta-policy network takes a unified state vector and task embedding vector as input and outputs energy regulation actions. The task embedding module is used to obtain the device profile vector and running status vector of the task and form the task embedding vector. The inner loop parameter update module is used to fine-tune the meta-strategy parameters and update the task adaptation parameters for each task through an inner loop. The outer loop meta-update module updates the meta-strategy parameters through the outer loop based on the result of the inner loop update.

9. A computer device, characterized in that, The computer device includes a processor and a memory connected to the processor. The memory stores one or more programs, which are executed by the processor to implement the steps in the multi-home microgrid energy regulation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which are executed by a processor to implement the steps in the multi-home microgrid energy regulation method as described in any one of claims 1-7.

Images