Regional power grid self-organizing group decision-making method and system based on multi-target staged learning

Abstract

The invention discloses a regional power grid self-organizing group decision-making method and system based on multi-target staged learning, and the method comprises the steps: building a plurality of group strategy networks through regional power grid resources, enabling the group strategy networks to carry out the forward calculation according to the observed quantity of the group strategy networks, outputting the group action quantity in a current environment state, and outputting the group power according to the group action quantity. Node injection power of the hitching group is counted into group power; calculating a single-round reward value according to environment states and actions before and after decision making, and storing single-round decision making experience into an experience pool; performing neural network parameter optimization, calculating a loss function value of the value network, and judging whether convergence occurs or not according to the variable quantity of the loss function value until all objective function subitems are reckoned in and a loss function is converged; and the group strategy network performs forward calculation and outputs the group action quantity in the current environment state, and the group resources execute operation according to the group action quantity, so that the flexible resource power is output to the power grid hitching node. According to the invention, autonomous intelligent decision-making of resource groups can be realized.

Description

Technical Field

[0001] The present invention belongs to the field of source-grid-load-storage coordinated dispatching, and specifically relates to a regional power grid self-organizing group decision-making method and system for multi-objective phased learning. Background Art

[0002] With the orderly development of the construction of new power systems, distributed flexible resources are rich in variety, high in penetration, and large in adjustable capacity. The granularity of adjustable loads has been reduced from megawatts to kilowatts. Taking a certain regional power grid as an example, the source-grid-load-storage control system is connected to more than 100,000 micro-loads such as a university, a company's headquarters base, a heat pump energy station, and a photovoltaic storage and charging building. In addition, the types of flexible resources are diverse, including energy storage, non-industrial air conditioning, commercial buildings, residential loads, electric vehicles, electric boilers, industrial parks, microgrids, etc., which poses a huge challenge to the power balance of regional power grids taking into account new energy, adjustable loads and other resources.

[0003] As the proportion of distributed renewable energy in regional power grids gradually increases, regional power grids are transforming from one-way energy distribution to a two-way balance of power generation and consumption. The existing provincial power grid centralized optimization is difficult to cope with the model complexity and power volatility of a large number of groups. Summary of the invention

[0004] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a regional power grid self-organizing group decision-making method and system with multi-objective phased learning, which takes the regional power grid resource group as the scheduling object, adopts phased neural network parameter optimization, realizes autonomous intelligent decision-making of the resource group, and effectively improves the level of intelligent interaction of massive flexible resources.

[0005] In order to achieve the above object, the present invention has the following technical solutions: In the first aspect, a method for self-organizing group decision-making of a regional power grid with multi-objective phased learning is provided, comprising: Establish several group strategy networks through regional power grid resources. i The group strategy network performs forward calculations based on its own observations and outputs the current environment state The amount of group action under , according to the group movement Output group power , the power injected by the node connected to the group is included in the group power , use grid interactive simulation to calculate power flow and output the active power of power system nodes , reactive power And voltage ; Based on the environmental conditions before and after the decision and And actions Calculate the single round reward value , the single-round decision experience ( , , , ) is stored in the experience pool; Optimize the neural network parameters, extract experience data from the experience pool, and set the network value according to ( , , )Calculate the value network gradient; according to ( , ) Calculate the full-cycle benefit valuation of group actions in the current environment through the value network , based on full-cycle earnings valuation and group movement volume Calculate the gradient of the group strategy network; calculate the loss function value of the value network, and determine whether it has converged based on the change in the loss function value. If it has converged, add a new stage objective function item, otherwise continue to update the value network and the group strategy network; until all objective function items are taken into account and the loss function converges; The swarm strategy network performs forward calculations based on the power grid measurement information and user-side measurement information to output the current environmental status. The amount of group action under , the regional power grid group resources are based on the group action volume Execute operations to enable flexible resource power to be exported to regional power grid connection nodes.

[0006] As a preferred solution, the step of establishing a plurality of group strategy networks through regional power grid resources includes: Self-organizing group observation space design:

[0007] Self-organizing group action space design:

[0008] Design of multi-objective reward function for self-organizing groups:

[0009] In the formula, is the observation space of the ith agent in period t, is the mean room temperature within the group, is the ambient temperature, is the energy storage power, is the photovoltaic power, is the wind power, Inject power into the node, is the node voltage; is the agent action space, Temperature adjustment, is the discharge power regulation value, is the charging power adjustment amount, is the photovoltaic power regulation, is the wind power regulation value.

[0010] As a preferred solution, the multi-objective phased learning regional power grid self-organizing group decision-making method also includes multi-agent reinforcement learning of several group strategy networks established through regional power grid resources, including: Multi-agent strategy network design; Multi-agent target strategy network design; Multi-agent global value network design; Multi-agent global objective value network design; Perform multi-agent global value network and target global value network updates; Perform multi-agent policy network and target policy network updates; Add stage targets and end training after stage-by-stage training converges.

[0011] As a preferred solution, the steps of designing the multi-agent strategy network include: Input layer dimension calculation:

[0012] Output layer dimension calculation:

[0013] Hidden layer dimension calculation:

[0014] In the formula, is the policy network input layer dimension, is the observation vector dimension, is the dimension of the policy network output layer, is the action vector dimension, is the hidden layer dimension of the policy network, is the number of training samples, Adjust parameters for diversity;

[0015]

[0016] Based on the hidden layer dimension calculation formula , bringing in the above two dimensional conditions determined by the input and output layer dimensions, when If the conditions are met, the original value can be used; otherwise, the boundary value of the above conditions can be used; The number of hidden layers is set to be greater than 2. By comparing the results, the number of layers is increased for underfitting and decreased for overfitting. Overfitting is judged based on the fact that the reward obtained by using the training set data for decision-making reaches the upper limit, but the reward obtained by using the test set data for decision-making reaches the lower limit. Underfitting is judged based on the fact that the reward values ​​obtained by making decisions on both the training set and the test set data reach the lower limit.

[0017] As a preferred solution, the steps of designing the multi-agent global value network include: Input layer dimension calculation:

[0018] Output layer dimension calculation:

[0019] Hidden layer dimension calculation:

[0020]

[0021]

[0022] In the formula, Input dimensions for the value network, is the value network output dimension, is the hidden layer dimension of the network.

[0023] As a preferred solution, the step of updating the multi-agent global value network and the target global value network includes: Target Global Value Network Using the Global Value Network Initialize a copy of Global Value Network In the current target global value network Add random disturbance on the basis of Extract multiple experiences ( , , , ) to update the target global value network, including: Calculate the action in period t+1:

[0024] In the formula, is the strategy network at time t+1 The decision results, Represents the forward calculation result of the policy network; Calculate the target value network estimate:

[0025] in, is the target global value network at time t+1 The valuation results, Representing the global value network The forward calculation result of Revenue calculation:

[0026] In the formula, is the total global revenue of multi-agent decision-making under the action and state at time t calculated according to the Bellman equation, is the reward function value at time t, is the discount factor for future earnings; Action estimation calculation:

[0027] In the formula, Global value network An estimate of the global payoff of multi-agent decision making under actions and states at time t, Global value network The forward calculation result of Loss function calculation:

[0028] in, Representing the global value network The loss function of Gradient calculation:

[0029] In the formula, Represents the target global value network The gradient of Representing the global value network The gradients of all parameters with respect to their estimates, It is calculated using the back-propagation method; Target value network update:

[0030] in, The target global value network The learning rate.

[0031] As a preferred solution, the step of updating the multi-agent strategy network and the target strategy network includes: Calculate action estimates: Take the global value network An estimate of the global payoff of multi-agent decision making for actions and states at time t ; Calculate the gradient of the action with respect to the loss function: The policy network loss function takes the global value network The valuation is:

[0032] In the formula, is the forward calculation result of the global value network; Therefore, the gradient of action to valuation is expressed as:

[0033] In the formula, is the gradient of the action of the ith agent with respect to the global value network valuation; Calculate the gradient of the policy network parameters with respect to the action:

[0034] In the formula, is the gradient of the policy network parameters of the ith agent at time period t to the action; The gradient of the policy network parameters to the action is calculated by the policy network back propagation method; Calculate the gradient of the policy network parameters with respect to the estimate:

[0035] In the formula, The gradient of the strategy network of the ith agent to the valuation is equal to the product of the gradient of the strategy network parameters to the action and the gradient of the action to the valuation network of the corresponding agent in the same period; Targeted Policy Network Updates:

[0036] In the formula, is the target strategy network parameter, is the target policy network learning rate.

[0037] As a preferred solution, the step of adding a stage target and ending the training after the staged training converges includes: Determine whether the loss function converges:

[0038] Calculate whether the change in the loss function meets the convergence condition ; If it is not satisfied, continue to perform the strategy network and global value network under the current reward function And the corresponding target global value network Parameter update of Weighting coefficients and stage mapping: Phase 1: Adding Balance Rewards and cost incentives , so that the self-organizing group regulation can meet the power balance constraints of the regional power grid under the minimum cost target; Phase 2: Adding line voltage rewards and node voltage reward , so that the regulation target of the self-organizing group meets the safety constraints of power grid operation; Phase 3: Adding incentives for new energy consumption and group response bias rewards , so that the self-organized group adjustment target is conducive to the consumption of new energy and the completion of group resources; The objective function is adjusted in stages. The weighted coefficient of the objective function is initialized to 0. After each stage of training reaches the convergence condition, the weighted coefficient is modified to 1 in sequence until the training ends after the last stage of training converges.

[0039] As a preferred solution, the group strategy network performs forward calculation based on the power grid measurement information and the user side measurement information to output the current environmental status The amount of group action under In the steps of i Based on the statistical information of electrical quantities and resource measurements observed by the group Perform forward calculation of the strategy network to adjust the group action volume :

[0040] In the formula, Represents the forward calculation result of the policy network; The group resources of the regional power grid are based on the group action volume In the step of executing the operation to output the power of flexible resources to the regional power grid connection node, the internal resources of the regional power grid group perform charging and discharging, temperature regulation and distributed new energy power control according to the published group action strategy.

[0041] Secondly, a regional power grid self-organizing group decision-making system with multi-objective phased learning is provided, including: The decision-making process establishment module is used to establish several group strategy networks through regional power grid resources. i The group strategy network performs forward calculations based on its own observations and outputs the current environment state The amount of group action under , according to the group movement Output group power , the power injected by the node connected to the group is included in the group power , use grid interactive simulation to calculate power flow and output the active power of power system nodes , reactive power And voltage ; Based on the environmental conditions before and after the decision and And actions Calculate the single round reward value , the single-round decision experience ( , , , ) is stored in the experience pool; The neural network parameter optimization module is used to optimize the neural network parameters, extract experience data from the experience pool, and set the network according to ( , , )Calculate the value network gradient; according to ( , ) Calculate the full-cycle benefit valuation of group actions in the current environment through the value network , based on full-cycle earnings valuation and group movement Calculate the gradient of the group strategy network; calculate the loss function value of the value network, and determine whether it has converged based on the change in the loss function value. If it has converged, add a new stage objective function item, otherwise continue to update the value network and the group strategy network; until all objective function items are taken into account and the loss function converges; The decision output module is used by the group strategy network to perform forward calculations based on the power grid measurement information and user-side measurement information, and output the current environmental status The amount of group action under , the regional power grid group resources are based on the group action volume Execute operations to enable flexible resource power to be exported to regional power grid connection nodes.

[0042] As a preferred solution, the steps of the decision-making process establishment module establishing a plurality of group strategy networks through regional power grid resources include: Self-organizing group observation space design:

[0043] Self-organizing group action space design:

[0044] Design of multi-objective reward function for self-organizing groups:

[0045] In the formula, is the observation space of the ith agent in period t, is the mean room temperature within the group, is the ambient temperature, is the energy storage power, is the photovoltaic power, is the wind power, Inject power into the node, is the node voltage; is the agent action space, Temperature adjustment, is the discharge power regulation value, is the charging power adjustment amount, is the photovoltaic power regulation, is the wind power regulation; Multi-agent reinforcement learning is performed on several swarm strategy networks established through regional power grid resources, including: Multi-agent strategy network design; Multi-agent target strategy network design; Multi-agent global value network design; Multi-agent global objective value network design; Perform multi-agent global value network and target global value network updates; Perform multi-agent policy network and target policy network updates; Add stage targets and end training after stage-by-stage training converges.

[0046] As a preferred solution, the decision output module will self-organize the group i Based on the statistical information of electrical quantities and resource measurements observed by the group Perform forward calculation of the strategy network to adjust the group action volume :

[0047] In the formula, Represents the forward calculation result of the policy network; The internal resources of the regional power grid group perform charging and discharging, temperature regulation, and distributed new energy power control based on the published group action strategy.

[0048] In a third aspect, an electronic device is provided, comprising a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the multi-objective phased learning regional power grid self-organizing group decision-making method.

[0049] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the regional power grid self-organizing group decision-making method with multi-objective staged learning is implemented.

[0050] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects: A self-organizing group multi-agent decision-making intelligent body training framework has been constructed to achieve autonomous intelligent decision-making of resource groups and realize intelligent interaction between sources, networks, loads and storage. A self-organizing group is a massive resource aggregate. The aggregate is constructed through spontaneous interaction, and the group allows users to join and exit if the access conditions are met. Massive device-level resources are aggregated layer by layer to form an adjustable resource group. The present invention realizes the coordinated interaction of sources, networks, loads and storage through a centralized decision-making-distributed autonomous hybrid mode, reducing the model complexity and power volatility. The present invention is suitable for intelligent decision-making of massive flexible resource groups in regional power grids, including but not limited to interactive regulation of sources, networks, loads and storage in regional power grid zones, and can effectively improve the level of intelligent interaction of massive flexible resources.

[0051] Furthermore, the embodiment of the present invention performs multi-agent reinforcement learning on several group strategy networks established through regional power grid resources. The proposed training method adopts a staged neural network parameter optimization technology to improve the security of intelligent decision-making, as well as the learning efficiency and convergence of the neural network.

[0052] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 The training phase architecture diagram of the regional power grid self-organizing group decision-making method for multi-objective phased learning of the present invention; Figure 2 Schematic diagram of intelligent decision-making process of regional power grid self-organizing group decision-making method with multi-objective phased learning of the present invention; Figure 3 The multi-agent global value network and target global value network update flow chart of the present invention; Figure 4 The multi-agent strategy network and target strategy network update flow chart of the present invention. DETAILED DESCRIPTION

[0055] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0056] See also Figure 1 and Figure 2 The embodiment of the present invention proposes a multi-objective phased learning regional power grid self-organizing group decision-making method, which is mainly divided into two parts: an intelligent decision-making process and a neural network parameter optimization process, specifically including: S1. Establish several group strategy networks through regional power grid resources. i The group strategy network performs forward calculations based on its own observations and outputs the current environment state The amount of group action under , according to the group movement Output group power , the power injected by the node connected to the group is included in the group power , use grid interactive simulation to calculate power flow and output the active power of power system nodes , reactive power And voltage ; Based on the environmental conditions before and after the decision and And actions Calculate the single round reward value , the single-round decision experience ( , , , ) is stored in the experience pool; S2, optimize the neural network parameters, extract experience data from the experience pool, and value the network according to ( , , ) The network gradient is calculated by the minimum mean square root; according to ( , ) Calculate the full-cycle benefit valuation of group actions in the current environment through the value network , based on full-cycle earnings valuation and group movement Calculate the gradient of the group strategy network; calculate the loss function value of the value network, and determine whether it has converged based on the change in the loss function value. If it has converged, add a new stage objective function item, otherwise continue to update the value network and the group strategy network; until all objective function items are taken into account and the loss function converges; S3, the group strategy network performs forward calculations based on the power grid measurement information and user-side measurement information, and outputs the current environmental status The amount of group action under , the regional power grid group resources are based on the group action volume Execute operations to enable flexible resource power to be exported to regional power grid connection nodes.

[0057] Among them, forward calculation is a basic process in neural networks, which involves passing input data through each layer of the network until the output is obtained. Massive device-level resources are aggregated layer by layer to form adjustable resource groups. The coordinated interaction of source, grid, load and storage through a centralized decision-making-distributed autonomous hybrid mode has become an important task in the construction of new power systems.

[0058] In a possible implementation, the steps of establishing a plurality of group strategy networks through regional power grid resources include: Self-organizing group observation space design:

[0059] Self-organizing group action space design:

[0060] Design of multi-objective reward function for self-organizing groups:

[0061] In the formula, is the observation space of the ith agent in period t, is the mean room temperature within the group, is the ambient temperature, is the energy storage power, is the photovoltaic power, is the wind power, Inject power into the node, is the node voltage; is the agent action space, Temperature adjustment, is the discharge power regulation value, is the charging power adjustment amount, is the photovoltaic power regulation, is the wind power regulation value.

[0062] In a possible implementation, multi-agent reinforcement learning is performed on several swarm strategy networks established through regional power grid resources, including: Multi-agent strategy network design; Multi-agent target strategy network design; Multi-agent global value network design; Multi-agent global objective value network design; Perform multi-agent global value network and target global value network updates; Perform multi-agent policy network and target policy network updates; Add stage targets and end training after stage-by-stage training converges.

[0063] In a possible implementation, the steps of designing the multi-agent strategy network include: Input layer dimension calculation:

[0064] Output layer dimension calculation:

[0065] Hidden layer dimension calculation:

[0066] In the formula, is the policy network input layer dimension, is the observation vector dimension, is the dimension of the policy network output layer, is the action vector dimension, is the hidden layer dimension of the policy network, is the number of training samples, The diversity adjustment parameter is added. The number of samples determines the model complexity of the agent, but the diversity of the samples will affect the actual amount of information in the samples. , usually 2 to 10. In general, the more training samples there are, the higher the diversity is, and the more neurons that can support the neural network.

[0067]

[0068]

[0069] The more complex the decision neural network is, the better it is. If the neural network is too complex, it will make the training difficult to converge or cause overfitting. , bringing in the above two dimensional conditions determined by the input and output layer dimensions, when When the conditions are met, just take the original value; otherwise, take the boundary value of the above conditions.

[0070] The decision problem contains complex mapping relationships and a large number of decision boundaries. The number of hidden layers is set to be greater than 2. By comparing the results, the number of layers is increased for underfitting and decreased for overfitting. Among them, the basis for judging overfitting is that when using the training set data, the reward for decision-making is extremely high, but when using the test set, the reward for decision-making is low; the basis for judging underfitting is that the reward values ​​for decision-making on both the training set and the test set data are low.

[0071] In a possible implementation, the multi-agent target policy network design is the same as the multi-agent policy network design.

[0072] In a possible implementation, the step of designing a multi-agent global value network includes: Input layer dimension calculation:

[0073] Output layer dimension calculation:

[0074] Hidden layer dimension calculation:

[0075]

[0076]

[0077] In the formula, Input dimensions for the value network, is the value network output dimension, is the hidden layer dimension of the network.

[0078] In one possible implementation, the multi-agent global objective value network design is the same as the multi-agent global value network design.

[0079] See also Figure 3 In a possible implementation, the step of updating the multi-agent global value network and the target global value network uses one global value network (referred to as value network) and one target global value network (referred to as target value network).

[0080] The specific steps include: Target Global Value Network Using the Global Value Network Initialize a copy of Global Value Network In the current target global value network Add random disturbance on the basis of Extract multiple experiences ( , , , ) to update the target global value network, including: Calculate the action in period t+1:

[0081] In the formula, is the strategy network at time t+1 The decision results, Represents the forward calculation result of the policy network; Calculate the target value network estimate:

[0082] in, is the target global value network at time t+1 The valuation results, Representing the global value network The forward calculation result of Revenue calculation:

[0083] In the formula, is the total global revenue of multi-agent decision-making under the action and state at time t calculated according to the Bellman equation, is the reward function value at time t, is the discount factor for future earnings; Action estimation calculation:

[0084] In the formula, Global value network An estimate of the global payoff of multi-agent decision making under actions and states at time t, Global value network The forward calculation result of Loss function calculation:

[0085] in, Representing the global value network The loss function of Gradient calculation:

[0086] In the formula, Represents the target global value network The gradient of Representing the global value network The gradients of all parameters with respect to their estimates, It is calculated using the back-propagation method; back-propagation is a common method for training artificial neural networks. It is an algorithm used in conjunction with optimization methods such as gradient descent. The back-propagation algorithm uses the chain rule to start from the output layer of the network and calculate the gradient backwards layer by layer along the connections of the network.

[0087] Target value network update:

[0088] in, The target global value network The learning rate.

[0089] See also Figure 4 In a possible implementation, the step of updating the multi-agent policy network and the target policy network includes: Calculate action estimates: Take the global value network in the previous step An estimate of the global payoff of multi-agent decision making for actions and states at time t ; Calculate the gradient of the action with respect to the loss function: The policy network loss function takes the global value network The valuation is:

[0090] In the formula, is the forward calculation result of the global value network; Therefore, the gradient of action to valuation is expressed as:

[0091] In the formula, is the gradient of the action of the ith agent with respect to the global value network valuation; Calculate the gradient of the policy network parameters with respect to the action:

[0092] In the formula, is the gradient of the policy network parameters of the ith agent at time period t to the action; The gradient of the policy network parameters to the action is calculated by the policy network back propagation method; Calculate the gradient of the policy network parameters with respect to the estimate:

[0093] In the formula, The gradient of the strategy network of the ith agent to the valuation is equal to the product of the gradient of the strategy network parameters to the action and the gradient of the action to the valuation network of the corresponding agent in the same period; Target Policy Network Updates:

[0094] In the formula, is the target strategy network parameter, is the target policy network learning rate.

[0095] In a possible implementation, the step of increasing the stage target and ending the training after the staged training converges includes: Determine whether the loss function converges:

[0096] Calculate whether the change in the loss function meets the convergence condition ; If not satisfied, continue to perform policy network and global value network under the current reward function And the corresponding target global value network Parameter update of Weighting coefficients and stage mapping: Phase 1: Adding Balance Rewards and cost incentives , so that the self-organizing group regulation can meet the power balance constraints of the regional power grid under the minimum cost target; Phase 2: Adding line voltage rewards and node voltage reward , so that the regulation target of the self-organizing group meets the safety constraints of power grid operation; Phase 3: Adding incentives for new energy consumption and group response bias rewards , so that the self-organized group adjustment target is conducive to the consumption of new energy and the completion of group resources; The objective function is adjusted in stages. The weighted coefficient of the objective function is initialized to 0. After each stage of training reaches the convergence condition, the weighted coefficient is modified to 1 in sequence until the training ends after the last stage of training converges.

[0097] The self-organizing group makes autonomous decisions. The group strategy network performs forward calculations based on the power grid measurement information and user-side measurement information to output the current environmental status The amount of group action under In the steps of i Based on the statistical information of electrical quantities and resource measurements observed by the group Perform forward calculation of the strategy network to adjust the group action volume :

[0098] In the formula, Represents the forward calculation result of the policy network; Self-organizing flexible resource response, the regional power grid group resources are based on the group action volume In the step of executing the operation to output the power of flexible resources to the regional power grid connection node, the internal resources of the regional power grid group perform charging and discharging, temperature regulation and distributed new energy power control according to the published group action strategy.

[0099] The method for self-organizing group decision-making of regional power grids with multi-objective phased learning in the embodiment of the present invention completes the construction of a training framework for self-organizing group multi-agent decision-making intelligent agents, realizes autonomous intelligent decision-making of resource groups, and realizes intelligent interaction between sources, grids, loads and storage. The training method proposed in the present invention adopts phased neural network parameter optimization technology to improve the learning efficiency and convergence of neural networks.

[0100] Another embodiment of the present invention further provides a regional power grid self-organizing group decision-making system with multi-objective phased learning, comprising: The decision-making process establishment module is used to establish several group strategy networks through regional power grid resources. i The group strategy network performs forward calculations based on its own observations and outputs the current environment state The amount of group action under , according to the group movement Output group power , the power injected by the node connected to the group is included in the group power , use grid interactive simulation to calculate power flow and output the active power of power system nodes , reactive power And voltage ; Based on the environmental conditions before and after the decision and And actions Calculate the single round reward value , the single-round decision experience ( , , , ) is stored in the experience pool; The neural network parameter optimization module is used to optimize the neural network parameters, extract experience data from the experience pool, and set the network according to ( , , ) The network gradient is calculated by the minimum mean square root; according to ( , ) Calculate the full-cycle benefit valuation of group actions in the current environment through the value network , based on full-cycle earnings valuation and group movement Calculate the gradient of the group strategy network; calculate the loss function value of the value network, and determine whether it has converged based on the change in the loss function value. If it has converged, add a new stage objective function item, otherwise continue to update the value network and the group strategy network; until all objective function items are taken into account and the loss function converges; The decision output module is used by the group strategy network to perform forward calculations based on the power grid measurement information and user-side measurement information, and output the current environmental status The amount of group action under , the regional power grid group resources are based on the group action volume Execute operations to enable flexible resource power to be exported to regional power grid connection nodes.

[0101] In a possible implementation, the step of the decision process establishment module establishing a plurality of group strategy networks through regional power grid resources includes: Self-organizing group observation space design:

[0102] Self-organizing group action space design:

[0103] Design of multi-objective reward function for self-organizing groups:

[0104] In the formula, is the observation space of the ith agent in period t, is the mean room temperature within the group, is the ambient temperature, is the energy storage power, is the photovoltaic power, is the wind power, Inject power into the node, is the node voltage; is the agent action space, Temperature adjustment, is the discharge power regulation value, is the charging power adjustment amount, is the photovoltaic power regulation, is the wind power regulation; Multi-agent reinforcement learning is performed on several swarm strategy networks established through regional power grid resources, including: Multi-agent strategy network design; Multi-agent target strategy network design; Multi-agent global value network design; Multi-agent global objective value network design; Perform multi-agent global value network and target global value network updates; Perform multi-agent policy network and target policy network updates; Add stage targets and end training after stage-by-stage training converges.

[0105] In a possible implementation, the self-organizing group makes its own decision, and the decision output module sends the self-organizing group i Based on the statistical information of electrical quantities and resource measurements observed by the group Perform forward calculation of the strategy network to adjust the group action volume :

[0106] In the formula, Represents the forward calculation result of the policy network; Self-organizing flexible resource response, the internal resources of the regional power grid group perform charging and discharging, temperature regulation and distributed new energy power control based on the published group action strategy.

[0107] Another embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the multi-objective staged learning regional power grid self-organizing group decision-making method.

[0108] Another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the regional power grid self-organizing group decision-making method with multi-objective staged learning is implemented.

[0109] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals. For ease of explanation, the above content only shows the part related to the embodiment of the present invention. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium is non-temporary and can be stored in a storage device formed by various electronic devices, which can realize the execution process recorded in the method of the embodiment of the present invention.

[0110] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-objective phased learning regional power grid self-organizing group decision-making method, characterized in that: include: Establish several group strategy networks through regional power grid resources. i The group strategy network performs forward calculations based on its own observations and outputs the current environment state The amount of group action under , according to the group movement Output group power , the power injected by the node connected to the group is included in the group power ; Based on the environmental conditions before and after the decision and And actions Calculate the single round reward value , the single-round decision experience ( , , , ) is stored in the experience pool; Optimize the neural network parameters, extract experience data from the experience pool, and set the network value according to ( , , )Calculate the value network gradient; according to ( , ) Calculate the full-cycle benefit valuation of group actions in the current environment through the value network , based on full-cycle earnings valuation and group movement Calculate the gradient of the swarm strategy network; calculate the loss function value of the value network, and determine whether it has converged based on the change in the loss function value, until all objective function items are taken into account and the loss function converges; The swarm strategy network performs forward calculations and outputs the current environment state The amount of group action under , the regional power grid group resources are based on the group action volume Execute operations to enable flexible resource power to be exported to regional power grid connection nodes.

2. The method for regional power grid self-organization group decision-making based on multi-objective phased learning according to claim 1 is characterized in that: The steps of establishing a plurality of group strategy networks through regional power grid resources include: Self-organizing group observation space design: Self-organizing group action space design: Design of multi-objective reward function for self-organizing groups: In the formula, is the observation space of the ith agent in period t, is the mean room temperature within the group, is the ambient temperature, is the energy storage power, is the photovoltaic power, is the wind power, Inject power into the node, is the node voltage; is the agent action space, Temperature adjustment, is the discharge power regulation value, is the charging power adjustment amount, is the photovoltaic power regulation, is the wind power regulation quantity.

3. The method for regional power grid self-organization group decision-making with multi-objective phased learning according to claim 1 is characterized in that: It also includes multi-agent reinforcement learning of several swarm strategy networks established through regional power grid resources, including: Multi-agent strategy network design; Multi-agent target strategy network design; Multi-agent global value network design; Multi-agent global objective value network design; Perform multi-agent global value network and target global value network updates; Perform multi-agent policy network and target policy network updates; Add stage targets and end training after stage-by-stage training converges.

4. The method for regional power grid self-organization group decision-making with multi-objective phased learning according to claim 3 is characterized in that: The steps of multi-agent strategy network design include: Input layer dimension calculation: Output layer dimension calculation: Hidden layer dimension calculation: In the formula, is the policy network input layer dimension, is the observation vector dimension, is the dimension of the policy network output layer, is the action vector dimension, is the hidden layer dimension of the policy network, is the number of training samples, Adjust parameters for diversity; Based on the hidden layer dimension calculation formula , bringing in the above two dimensional conditions determined by the input and output layer dimensions, when If the conditions are met, the original value can be used; otherwise, the boundary value of the above conditions can be used; The number of hidden layers is set to be greater than 2. By comparing the results, the number of layers is increased for underfitting and decreased for overfitting. Overfitting is judged based on the fact that the reward obtained by using the training set data for decision-making reaches the upper limit, but the reward obtained by using the test set data for decision-making reaches the lower limit. Underfitting is judged based on the fact that the reward values ​​obtained by making decisions on both the training set and the test set data reach the lower limit.

5. The method for regional power grid self-organization group decision-making with multi-objective phased learning according to claim 3 is characterized in that: The steps of multi-agent global value network design include: Input layer dimension calculation: Output layer dimension calculation: Hidden layer dimension calculation: In the formula, Input dimensions for the value network, is the value network output dimension, is the hidden layer dimension of the network.

6. The method for regional power grid self-organization group decision-making with multi-objective phased learning according to claim 3 is characterized in that: The step of updating the multi-agent global value network and the target global value network comprises: Target Global Value Network Using the Global Value Network Initialize a copy of Global Value Network In the current target global value network Add random disturbance on the basis of Extract multiple experiences ( , , , ) to update the target global value network, including: Calculate the action in period t+1: In the formula, is the strategy network at time t+1 The decision results, Represents the forward calculation result of the policy network; Calculate the target value network estimate: in, is the target global value network at time t+1 The valuation results, Representing the global value network The forward calculation result of Revenue calculation: In the formula, is the total global revenue of multi-agent decision-making under the action and state at time t, is the reward function value at time t, is the discount factor for future earnings; Action estimation calculation: In the formula, Global value network An estimate of the global payoff of multi-agent decision making under actions and states at time t, Global value network The forward calculation result of Loss function calculation: in, Representing the global value network The loss function of Gradient calculation: In the formula, Represents the target global value network The gradient of Representing the global value network The gradients of all parameters with respect to their estimates, It is calculated using the back-propagation method; Target value network update: in, The target global value network The learning rate.

7. The method for regional power grid self-organization group decision-making with multi-objective phased learning according to claim 6 is characterized in that: The steps of updating the multi-agent strategy network and the target strategy network include: Calculate action estimates: Take the global value network An estimate of the global payoff of multi-agent decision making for actions and states at time t ; Calculate the gradient of the action with respect to the loss function: The policy network loss function takes the global value network The valuation is: In the formula, is the forward calculation result of the global value network; Therefore, the gradient of action to valuation is expressed as: In the formula, is the gradient of the action of the ith agent with respect to the global value network valuation; Calculate the gradient of the policy network parameters with respect to the action: In the formula, is the gradient of the policy network parameters of the ith agent at time period t to the action; The gradient of the policy network parameters to the action is calculated by the policy network back propagation method; Calculate the gradient of the policy network parameters with respect to the estimate: In the formula, The gradient of the strategy network of the ith agent to the valuation is equal to the product of the gradient of the strategy network parameters to the action and the gradient of the action to the valuation network of the corresponding agent in the same period; Targeted Policy Network Updates: In the formula, is the target strategy network parameter, is the target policy network learning rate.

8. The method for regional power grid self-organization group decision-making with multi-objective phased learning according to claim 3 is characterized in that: The step of increasing the stage target and ending the training after the staged training converges comprises: Determine whether the loss function converges: Calculate whether the change in the loss function meets the convergence condition ; If it is not satisfied, continue to perform the strategy network and global value network under the current reward function And the corresponding target global value network Parameter update of Weighting coefficients and stage mapping: Phase 1: Adding Balance Rewards and cost incentives , so that the self-organizing group regulation can meet the power balance constraints of the regional power grid under the minimum cost target; Phase 2: Adding line voltage rewards and node voltage reward , so that the regulation target of the self-organizing group meets the safety constraints of power grid operation; Phase 3: Adding incentives for new energy consumption and group response bias rewards , so that the self-organized group adjustment target is conducive to the consumption of new energy and the completion of group resources; The objective function is adjusted in stages. The weighted coefficient of the objective function is initialized to 0. After each stage of training reaches the convergence condition, the weighted coefficient is modified to 1 in sequence until the training ends after the last stage of training converges.

9. The method for regional power grid self-organization group decision-making with multi-objective phased learning according to claim 1 is characterized in that: In the step of calculating the loss function value of the value network and determining whether convergence has occurred based on the change in the loss function value, if convergence has occurred, a new stage objective function item is added, otherwise, the value network and the group strategy network are continued to be updated; In the step of forward calculation of the group strategy network, the group strategy network performs forward calculation based on the power grid measurement information and the user side measurement information to output the current environment state The amount of group action under ; Self-organizing groups i Based on the statistical information of electrical quantities and resource measurements observed by the group Perform forward calculation of the strategy network to adjust the group action volume : In the formula, Represents the forward calculation result of the policy network; The group resources of the regional power grid are based on the group action volume In the step of executing the operation to output the power of flexible resources to the regional power grid connection node, the internal resources of the regional power grid group perform charging and discharging, temperature regulation and distributed new energy power control according to the published group action strategy.

10. A multi-objective phased learning regional power grid self-organizing group decision-making system, characterized in that: include: The decision-making process establishment module is used to establish several group strategy networks through regional power grid resources. i The group strategy network performs forward calculations based on its own observations and outputs the current environment state The amount of group action under , according to the group action volume Output group power , the power injected by the node connected to the group is included in the group power ; Based on the environmental conditions before and after the decision and And actions Calculate the single round reward value , the single-round decision experience ( , , , ) is stored in the experience pool; The neural network parameter optimization module is used to optimize the neural network parameters, extract experience data from the experience pool, and set the network according to ( , , )Calculate the value network gradient; according to ( , ) Calculate the full-cycle benefit valuation of group actions in the current environment through the value network , based on full-cycle earnings valuation and group movement Calculate the gradient of the swarm strategy network; calculate the loss function value of the value network, and determine whether it has converged based on the change in the loss function value, until all objective function items are taken into account and the loss function converges; The decision output module is used by the group strategy network to perform forward calculations based on the power grid measurement information and user-side measurement information, and output the current environmental status The amount of group action under , the regional power grid group resources are based on the group action volume Execute operations to enable flexible resource power to be exported to regional power grid connection nodes.

11. The regional power grid self-organizing group decision-making system with multi-objective phased learning according to claim 10, characterized in that: The steps of the decision process establishment module establishing a plurality of group strategy networks through regional power grid resources include: Self-organizing group observation space design: Self-organizing group action space design: Design of multi-objective reward function for self-organizing groups: In the formula, is the observation space of the ith agent in period t, is the mean room temperature within the group, is the ambient temperature, is the energy storage power, is the photovoltaic power, is the wind power, Inject power into the node, is the node voltage; is the agent action space, Temperature adjustment, is the discharge power regulation value, is the charging power adjustment amount, is the photovoltaic power regulation, is the wind power regulation; Multi-agent reinforcement learning is performed on several swarm strategy networks established through regional power grid resources, including: Multi-agent strategy network design; Multi-agent target strategy network design; Multi-agent global value network design; Multi-agent global objective value network design; Perform multi-agent global value network and target global value network updates; Perform multi-agent policy network and target policy network updates; Add stage targets and end training after stage-by-stage training converges.

12. The regional power grid self-organizing group decision-making system with multi-objective phased learning according to claim 10, characterized in that: The decision output module will self-organize the group i Based on the statistical information of electrical quantities and resource measurements observed by the group Perform forward calculation of the strategy network to adjust the group action volume : In the formula, Represents the forward calculation result of the policy network; The internal resources of the regional power grid group perform charging and discharging, temperature regulation, and distributed new energy power control based on the published group action strategy.

13. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the regional power grid self-organizing group decision-making method with multi-objective staged learning as claimed in any one of claims 1 to 9.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the method for regional power grid self-organization group decision-making with multi-objective staged learning as described in any one of claims 1 to 9 is implemented.

Images