Power distribution network source network load storage low-carbon optimization scheduling increment reinforcement learning method and system

Abstract

The invention belongs to the field of power distribution network low-carbon economic optimization scheduling, and provides a power distribution network source network load storage low-carbon optimization scheduling increment reinforcement learning method and system, and the method comprises the steps: carrying out the intelligent agent division of a power grid, obtaining a plurality of intelligent agent regions, and collecting the observation data of each intelligent agent region; on the basis of the observation data, constructing a low-carbon economic dispatching model of source-grid-load-storage cooperation by taking the operation cost and carbon emission of each agent regional power grid as objective functions; designing the source-network-load-storage coordinated low-carbon economic dispatching model into a reinforcement learning model in a multi-agent environment according to a state space, an action space, an environment and a reward function; and solving the reinforcement learning model in the multi-agent environment to obtain an optimal scheduling solution. According to the incremental optimization method, the environment can help the intelligent agent to detect environment changes in training, excessive exploration risks are avoided, the training time is shortened, and meanwhile the flexibility of the system is improved.

Description

Technical Field

[0001] The present invention relates to the field of low-carbon economic optimization dispatching of distribution networks, and in particular to an incremental reinforcement learning method and system for low-carbon optimization dispatching of distribution network sources, grids, loads and storage. Background Art

[0002] With the advancement of source-grid-load-storage integration and the increase in the penetration rate of renewable energy, the traditional power system faces tremendous transformation pressure. The previous power system relied on coal-fired power units with strong controllability and stable output, while the new power system has gradually shifted to renewable energy with strong uncertainty such as wind power and photovoltaics. This transformation has made it more difficult to balance the power and electricity in the power grid, especially in the case of load fluctuations and unstable power generation output, the safety and stability of power grid operation are facing challenges.

[0003] At present, the coordinated dispatch of source-grid-load-storage systems is the key to meeting these challenges. Through reinforcement learning technology, the source-grid-load-storage system can realize the linkage of power sources, power grids, loads and energy storage, effectively balance supply and demand, improve the flexibility and stability of the power grid, and reduce dependence on traditional fossil energy backup units, thereby achieving the goal of a low-carbon economy. However, with the increase in the proportion of distributed energy and renewable energy generation, the "streaming data" generated by the power system has greatly increased the difficulty of dispatching. Traditional dispatching methods based on reinforcement learning are only suitable for the collaborative optimization of fixed tasks. When environmental data changes, they need to re-collect data globally and retrain, resulting in a huge consumption of communication resources and time costs. It is difficult to respond to the rapidly changing state of the power grid in real time, resulting in low efficiency and high cost when dealing with uncertainties in the source-grid-load-storage system, and it is difficult to meet the needs of new power systems. Summary of the invention

[0004] The purpose of the present invention is to provide a distribution network source-grid-load-storage low-carbon optimization scheduling incremental reinforcement learning method and system to solve the problems of low efficiency and high cost in traditional technologies.

[0005] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides an incremental reinforcement learning method for low-carbon optimal dispatching of distribution network source, grid, load and storage, comprising: Divide the power grid into intelligent agents to obtain multiple intelligent agent regions, and collect observation data for each intelligent agent region; Based on the observed data, a low-carbon economic dispatch model with source-grid-load-storage coordination is constructed with the operating cost and carbon emissions of each intelligent regional power grid as the objective function; The low-carbon economic dispatch model of source-grid-load-storage coordination is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function; Solve the reinforcement learning model in a multi-agent environment and obtain the optimal scheduling solution.

[0006] Furthermore, the power grid is divided into intelligent agents to obtain multiple intelligent agent regions, including: With the goal of making the connection between regions a single loop, the power flow between regions a single direction, and the power grid within the group unconstrained, the power grid is divided into different regions, and different intelligent agents are set according to the regions, obtaining multiple intelligent agent regions.

[0007] Furthermore, the observation data includes: Load data , Actual output power of distributed power supply , reactive power , Actual output power of energy storage device , Intelligent soft switch active output , reactive power , Power loss cost coefficient , Switching cost coefficient of on-load tap changer , the switching cost coefficient of the capacitor bank and carbon emission cost coefficient .

[0008] Furthermore, based on the observed data, a low-carbon economic dispatch model with source-grid-load-storage coordination is constructed with the operating cost and carbon emissions of each intelligent regional power grid as the objective function, including: The objective function is as follows:

[0009] in, and is the weight coefficient, is the number of partitioned regions, For Region Power loss cost, For Region Switch operating costs, For Region Carbon emission costs, For Region Voltage deviation degree; The constraints of the low-carbon economic dispatch model with coordinated source, grid, load and storage include the constraints of the active distribution network flow, the constraints of on-load tap changers and capacitor banks, and the operation constraints of other regulating resources.

[0010] Furthermore, the low-carbon economic dispatch model for coordinated source-grid-load-storage is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function, including: State Space The state space of the system includes the load data of each agent , Actual output power of distributed power supply , reactive power , Actual output power of energy storage device , Intelligent soft switch active output , reactive power , Power loss cost coefficient , Switching cost coefficient of on-load tap changer , the switching cost coefficient of the capacitor bank , Carbon emission cost coefficient and node voltage ,Right now

[0011] Action Space The action space variables include the actual output power of distributed generation , Intelligent soft switch active output , reactive power and the actual output power of the energy storage device ,Right now

[0012] Environmental Design On the basis of the completion of the reinforcement learning agent training, each agent evaluates the environment after the data change according to the new data received, and evaluates the current environmental drift state by using the reward value obtained by using the original optimal strategy decision in the new and old environments. If environmental drift is detected, incremental reinforcement learning is started, and the drift parameters are used to initialize the parameters of the agent in the new environment in the form of a convex combination of the random initialization strategy and the original optimal strategy. The low-carbon economic dispatch model with coordinated source, grid, load and storage is used as the environment. When training the Actor-Critic network of the agent, importance sampling is added, and the weights of the data samples generated by the strategy are adjusted. When updating the strategy, elastic weight consolidation is added. While the agent is learning new tasks, the parameters of the constraint model are updated.

[0013] Reward Function The inverse of the objective function of the multi-regional power grid model is used as the immediate reward for each agent F , according to the constraints, if the corresponding variable does not meet the constraints, then set the penalty value , together with the immediate reward, is used as the agent's final reward function, calculated as follows: .

[0014] Furthermore, solving the reinforcement learning model in the multi-agent environment to obtain the optimal scheduling solution includes: The MAIRL-EEPP method is used for solving, and the interface files with the states and actions in the MAIRL-EEPP method are set up so that the power flow calculation can be performed in real time according to the state actions and the corresponding environmental state quantities can be fed back; Based on the trained agents, each agent is trained according to the formula:

[0015] Detect the degree of environmental drift and synchronize and initialize parameters, where: is the old optimal strategy in the environment Rewards for execution, is the old optimal strategy in the environment Rewards for execution, is the defined drift boundary value; Set the training schedule period T after the environment changes and set the number of training rounds M for each agent. The initial values ​​are all set to 1. Initially, the agent network parameters are randomly set. ; Each agent interacts with the environment and executes strategies in the current environment , select actions based on the actions output by the policy network , and observe the next state and rewards , repeat until the maximum number of training rounds is reached, and record the state, action, and reward of each time step; Update each network parameter: For each time step, calculate the cumulative reward from the current time to the terminal state , according to the formula

[0016] Calculate the policy gradient and update the parameters, where The objective function of the policy network after adding elastic consolidation weights Strategy The gradient of is the policy network objective function Strategy The gradient of To adjust the hyperparameters of the regularization weights, is the Fisher information matrix, are the parameters of the current policy network, is the optimal strategy parameter in the original environment, For strategy Download all tracks Perform expectation calculations, For a given state When you select Action The gradient of the log probability of the policy, From the time step Discounts on future rewards starting and, is the importance weight, is the learning rate; Determine whether the current number of training rounds m reaches the set value M. If it reaches the set value, end the training, output the results and save them. If it does not reach the set value, start a new round of training.

[0017] In a second aspect, the present invention provides a distribution network source-grid-load-storage low-carbon optimization dispatching system, comprising The data acquisition module is used to divide the power grid into intelligent agents, obtain multiple intelligent agent areas, and collect observation data for each intelligent agent area; The model building module is used to build a low-carbon economic dispatch model with source-grid-load-storage coordination based on observation data and the operating cost and carbon emissions of each intelligent regional power grid as the objective function; The model design module is used to design the low-carbon economic dispatch model of source-grid-load-storage coordination into a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function; The optimization solution module is used to solve the reinforcement learning model in a multi-agent environment and obtain the optimal scheduling solution.

[0018] Furthermore, the observation data includes: Load data , Actual output power of distributed power supply , reactive power , Actual output power of energy storage device , Intelligent soft switch active output , reactive power , Power loss cost coefficient , Switching cost coefficient of on-load tap changer , the switching cost coefficient of the capacitor bank and carbon emission cost coefficient .

[0019] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the incremental reinforcement learning method for low-carbon optimized scheduling of distribution network source, grid, load and storage are implemented.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the incremental reinforcement learning method for low-carbon optimized scheduling of distribution network source, grid, load and storage.

[0021] Compared with the prior art, the present invention has the following technical effects: The present invention improves the distributed autonomy and flexibility of the system by dividing the power grid into multiple intelligent agent regions, each of which makes decisions and updates independently. This division enables the system to better adapt to local changes without requiring global adjustments to the entire power grid.

[0022] The present invention constructs a low-carbon economic dispatch model that takes into account the coordination of source, grid, load and storage, taking the operating cost and carbon emissions of all regional power grids in the system as the optimization target, and taking the safe and stable operation of the system as a constraint. This model takes into account both economic benefits and environmental protection requirements, and is in line with the current green and low-carbon development trend.

[0023] The model contains detailed equality and inequality constraints, such as active distribution network power flow constraints, soft switch power flow constraints, power balance constraints, and operation constraints of various equipment, etc. These constraints ensure the safety and stability of the system during the optimization process.

[0024] The multi-agent incremental reinforcement learning method based on environment detection and policy perturbation (MAIRL-EEPP) enables the agent to quickly adapt to the new environment through incremental updates after detecting environmental drift. This method avoids the large amount of time and resources required to retrain the entire model, and improves the system's responsiveness and adaptability.

[0025] This method effectively improves the training efficiency by detecting environmental drift and then conducting a "centralized training, distributed execution" model. At the same time, the importance weighting and elastic weight consolidation (EWC) technology are used to make the training process more adaptable to non-stationary environments and avoid catastrophic forgetting problems.

[0026] Thanks to the incremental reinforcement learning method, the agent can quickly update its decision after receiving new data, realizing the system's real-time decision-making ability. This is of great significance for dealing with emergencies and uncertainties in power systems.

[0027] By optimizing the dispatching strategy, the operating cost and carbon emissions of the power grid are reduced, achieving a dual improvement in economic and environmental benefits.

[0028] In summary, this technical solution uses the multi-agent incremental reinforcement learning method to build a low-carbon economic dispatch model that takes into account the coordination of source, grid, load and storage, and achieves technical effects such as rapid adaptation to new environments, optimization of dispatch strategies, and improvement of training efficiency and decision-making capabilities. These technical effects not only improve the operating efficiency and stability of the power system, but also provide strong support for the realization of a green and low-carbon power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The grid topology used in the incremental reinforcement learning optimization method for low-carbon economic dispatch of source-grid-load-storage distribution networks.

[0030] Figure 2 It is a flow chart of the present invention.

[0031] Figure 3 Flowchart for solving the overall algorithm for low-carbon economic dispatch optimization of sources, grids, loads and storage based on MAIRL-EEPP. DETAILED DESCRIPTION

[0032] The present invention is further described below in conjunction with the accompanying drawings: Example 1, please refer to Figure 2 The present invention provides a distribution network source-grid-load-storage low-carbon optimization scheduling incremental reinforcement learning method, comprising: Divide the power grid into intelligent agents to obtain multiple intelligent agent regions, and collect observation data for each intelligent agent region; Based on the observed data, a low-carbon economic dispatch model with source-grid-load-storage coordination is constructed with the operating cost and carbon emissions of each intelligent regional power grid as the objective function; The low-carbon economic dispatch model of source-grid-load-storage coordination is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function; Solve the reinforcement learning model in a multi-agent environment and obtain the optimal scheduling solution.

[0033] By dividing the power grid into intelligent agents, the complex distribution network system is decomposed into multiple manageable intelligent agent areas. This partition management not only reduces the complexity of the problem, but also enables each intelligent agent to focus more on the optimal dispatch within its jurisdiction. At the same time, the observation data collected for each intelligent agent area provides a solid foundation for subsequent model construction and solution. It improves the pertinence and efficiency of data processing and helps to achieve more refined management.

[0034] Based on the collected observation data, the model is constructed with operating costs and carbon emissions as the objective function, achieving dual considerations of economy and environmental protection. This collaborative model can comprehensively consider the interaction between source (power generation side), network (grid transmission), load (user side demand) and storage (energy storage equipment) to find the optimal scheduling strategy. The mathematical model abstracts complex practical problems, facilitating quantitative analysis and optimization solutions.

[0035] The low-carbon economic dispatch model is transformed into a reinforcement learning model in a multi-agent environment. The self-learning and adaptive capabilities of the reinforcement learning algorithm are used to enable the model to find the optimal solution through continuous trial and error. The design of the state space, action space, environment, and reward function provides clear optimization goals and constraints for the reinforcement learning algorithm. The reinforcement learning algorithm can handle complex uncertainty and dynamic problems and is suitable for systems such as distribution networks that are variable and uncertain.

[0036] By solving the reinforcement learning model in a multi-agent environment, the optimal scheduling solution is obtained. This solution not only takes into account economic and environmental performance, but also fully considers the synergy between various agents to achieve global optimality. Compared with traditional optimization methods, reinforcement learning algorithms can find global optimal solutions more efficiently and have better adaptability to the scale and complexity of problems.

[0037] In summary, the incremental reinforcement learning method for low-carbon optimal dispatching of distribution network sources, grids, loads and storage described in this paragraph has significant technical effects. It decomposes the complex distribution network system into multiple intelligent regions for refined management, and uses the self-learning and adaptive capabilities of the reinforcement learning algorithm to solve the optimal dispatching solution, thus achieving dual optimization of economy and environmental protection. This method not only improves the operating efficiency of the distribution network, but also helps promote the low-carbon transformation and sustainable development of the energy industry.

[0038] Embodiment 2, the present invention provides a distribution network source-grid-load-storage low-carbon optimization scheduling incremental reinforcement learning method, specifically comprising: In view of the problem that the original reinforcement learning agent model cannot adapt to the new environment and has reduced performance due to the continuous generation of streaming data during the operation of the power system, this paper collects the observation data of each agent area under normal conditions and the data that may change during the operation process, and builds a low-carbon economic dispatch model with source-grid-load-storage coordination based on this data. The operation cost and carbon emissions of all regional power grids in the system are taken as the optimization goals, and the safe and stable operation of the system is taken as the constraint. The multi-agent incremental reinforcement learning method based on environment detection and policy perturbation (Multi-Agent Incremental Reinforcement Learning Via Environment Evaluation And Policy Perturbation, MAIRL-EEPP) method is adopted, such as Figure 3As shown in the figure, the model is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function; the above model is programmed and solved using simulation software, and then the results obtained are compared with other methods to verify that the trained model not only has a good ability to cope with uncertainty, but also can realize real-time decision-making of the system. At the same time, it has a faster convergence speed and better training results during the training process, and has obvious advantages in dealing with complex environmental problems.

[0039] 1. Divide the power grid into intelligent entities: Take the IEEE123 node standard system as an example, and use the following standards as the basis for division: 1) The connection between regions is as single-circuit as possible; 2) The structure between regions is clear, and the flow direction is single; 3) The power grid within the group is basically unconstrained and has strong structure. Divide the IEEE123 node standard system into different regions, and set different intelligent entities according to the regions. Use the regional intelligent entity as the decision-making center to achieve incremental collaborative optimization operation of multi-regional power grids. The regional division is as follows: Figure 1 shown.

[0040] 2. A low-carbon economic dispatching method for source, grid, load and storage based on multi-agent incremental reinforcement learning includes the following steps: The observation data of each intelligent agent area is collected. The observation data of each intelligent agent involved in the present invention includes: load data, actual output power and reactive power of distributed power sources, actual output power of energy storage devices, active output and reactive power of intelligent soft switches, and cost coefficients including power loss cost coefficient, switching cost coefficient of on-load tap changer, switching cost coefficient of capacitor bank, and carbon emission cost coefficient; each regional intelligent agent incrementally meets the load demand in the region, improves voltage deviation and minimizes carbon emissions through generator sets on the one hand and energy interaction between regions on the other hand, and realizes autonomous incremental update of regional distribution to a certain extent. The model is as follows:

[0041] Taking the carbon emission cost, operation cost and improvement of grid voltage deviation of all regional grids in the system as the optimization objectives, the cost of each regional grid is set as power loss cost (including network loss cost and power loss cost of smart soft switch (SOP)), switch operation cost (adjustment cost of on-load tap changer (OLTC) and switching cost of capacitor banks (CBs)) and carbon emission cost. Among them, and is the weight coefficient, is the number of partitioned areas. For Region Power loss cost, For Region Switch operating costs, For Region Carbon emission costs, For Region Voltage deviation degree.

[0042] According to the constructed low-carbon economic dispatch model considering the coordination of source, grid, load and storage, the constraints of the optimization model are determined: The system's equality constraints mainly include active distribution network power flow constraints, soft switch power flow constraints, and power balance constraints. The system's inequality constraints are mainly aimed at ensuring the safe and stable operation of the system, mainly node voltage upper and lower limit constraints; on-load voltage tap changer tap changer step change upper and lower limit constraints; capacitor bank change upper and lower limit constraints; renewable energy unit output power upper and lower limit constraints; flexible load operation upper and lower limit constraints; energy storage system power output and state upper and lower limit constraints; carbon emission constraints.

[0043] Based on the basic elements of multi-agent deep reinforcement learning, the low-carbon economic dispatch model considering the coordination of source, grid, load and storage is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function. After training, MAIRL-EEPP is used for incremental update solution.

[0044] The solution is achieved using computer software. By comparing it with the centralized optimization solution, it is verified that the trained model not only has a good ability to cope with uncertainty, but also can realize real-time decision-making of the system, which is conducive to online application. By comparing it with the retrained multi-agent deep reinforcement learning method, it is verified that the MAIRL-EEPP method has a faster convergence speed and better training results during the training process, and has obvious advantages in dealing with complex environmental problems.

[0045] 3. The MAIRL-EEPP method adopts the mode of "centralized training and distributed execution" after detecting environmental drift. Each agent is set up with a separate actor and critic network. Different from the traditional retrained multi-agent reinforcement learning, the agent's strategy is initialized by the optimal strategy and random strategy in the original environment after weight balancing. During training, the utilization efficiency of different strategy samples is improved by importance weighting, making the training process more adaptable to non-stationary environments. The Actor of each agent takes actions according to its own state, and then the Critic evaluates the Actor's actions. The Actor updates its own strategy based on feedback. The Critic of each agent obtains more accurate evaluation information by estimating the strategies of other agents. When updating the strategy, elastic weight consolidation (EWC) is introduced to impose constraints on some parameters in the neural network, so that when learning new tasks, the knowledge of previous tasks is effectively retained, thereby avoiding conflicts between the new and old data environments of the model and solving the problem of catastrophic forgetting. After training, each agent does not need to obtain information from other agents, and each agent independently completes the decision update. MAIRL-EEPP obtains the optimal strategy through incremental training after environmental changes. When applied, only local new information is needed, which can effectively deal with system uncertainty problems in complex environments.

[0046] The specific implementation steps are as follows: Step 1: Import observation data for each agent region.

[0047] First collect Figure 2 The observation data of each intelligent agent area shown in the figure, the observation data of each intelligent agent involved in the present invention includes load data , Actual output power of distributed power supply , reactive power ; Actual output power of energy storage device , Intelligent soft switch active output , reactive power , the cost factor includes the power loss cost factor , Switching cost coefficient of on-load tap changer , the switching cost coefficient of the capacitor bank , Carbon emission cost coefficient .

[0048]

[0049] Step 2: Construct a multi-regional power grid collaborative optimization model with a high proportion of renewable energy.

[0050] Step 2.1: Establish the objective function.

[0051] A low-carbon economic dispatch model considering the coordination of source, grid, load and storage is constructed. Taking the IEEE123 node standard system as an example, it is divided into three regional power grids. The carbon emission cost, operation cost and improvement of grid voltage deviation of all regional power grids in the system are taken as the optimization targets. The cost of each regional power grid is set as the power loss cost (including the cost of network loss and the power loss cost of smart soft switch (SOP)), switch operation cost (adjustment cost of on-load voltage tap changer (OLTC) and switching cost of capacitor bank (CBs)) and carbon emission cost. The objective function of the system is as follows:

[0052] in, and is the weight coefficient, is the number of partitioned areas. For Region Power loss cost, For Region Switch operating costs, For Region Carbon emission costs, For Region Voltage deviation degree.

[0053] In formula (3),

[0054] in, For Region Power loss cost, is the power loss cost coefficient, To calculate the duration, is the branch resistance, is the branch current; is the active power loss of soft switching (SOP), is the power loss interval. is the set of lines associated with line loss, is the line set with soft switching losses.

[0055]

[0056] in, For Region Switch operating costs, To calculate the duration, is the switching cost coefficient of the on-load tap changer, is the switching cost coefficient of the capacitor bank. and They are Moment and The number of positions of the on-load tap-changer at any given time. and They are Moment and The number of gears of the capacitor bank at any time. A set of transformers controlled by on-load tapchangers. A related set of capacitor banks.

[0057]

[0058] in, For Region Carbon emission costs, To calculate the duration, For all carbon emission equipment collection, is the carbon emission cost coefficient, For equipment The amount of carbon emissions per unit time.

[0059]

[0060] in, For Region The voltage deviation degree, To calculate the duration, for node The voltage amplitude at the moment, is the voltage amplitude reference value, is the voltage amplitude range.

[0061] Step 2.2: Establish constraints for the optimization model.

[0062] The constraints of the above-mentioned low-carbon economic dispatch model considering the coordination of source, grid, load and storage include the constraints of active distribution network flow, the constraints of on-load tap changers and capacitor banks, and the operation constraints of other regulating resources.

[0063] 1. Active distribution network flow constraints:

[0064] In the formula, For Node To Node In time The active power, For Node To Node The line resistance between For Node To Node In time The current, For Node The distributed generators at time The active power, For Node The soft switch at time The active power, For Node The energy storage system at time The active power, For Node In time The load active power demand, For the node The set of all directly connected nodes, For Node To Node In time The amount of active power transferred. For Node To Node In time The reactive power, For Node To Node The line reactance between For Node The distributed generators at time The reactive power, For Node The soft switch at time The reactive power, For Node The capacitor bank at time The reactive power, For Node In time The reactive power demand of the load, For Node To Node In time The reactive power transfer amount, and For Node and nodes In time The voltage amplitude, and are the lower and upper limits of the voltage amplitude respectively.

[0065] 2. Constraints on on-load tap-changers and capacitor banks: The constraints of the on-load tap-changer are as follows:

[0066] in, and For Node and nodes In time The voltage amplitude, For Node and nodes The initial voltage ratio coefficient between For Node and nodes In time A coefficient of change, indicating the voltage transfer ratio between the two nodes in the initial state, is the incremental change in voltage ratio, To calculate the duration, For Node and nodes In time The coefficient of Adjust the total limit for the maximum voltage ratio allowed for the on-load tapchanger, and For Node and nodes The voltage between the upper and lower bounds is Represents an integer.

[0067] The constraints for the capacitor bank are as follows:

[0068] in, For time Time Node The reactive power output of the capacitor bank, For time Time Node The number of capacitor banks switched on and off, The reactive power provided for each group of capacitors, To calculate the duration, For time Time Node The number of capacitor banks switched on and off, is the upper limit of the number of times the capacitor is switched on and off in a unit time period, is the maximum switching quantity of capacitor bank, Represents an integer.

[0069] 3. The operating constraints of other adjustment resources are as follows: The distributed generator constraints are as follows:

[0070] in, For Node The distributed generators at time The active power, For Node The distributed generators at time Active power of renewable energy generation, Represents photovoltaic power generation, Represents wind power generation, For Node The distributed generators at time The maximum active power of the output, For Node The distributed generators at time The reactive power, For Node Rated apparent power capacity of distributed generators.

[0071] The soft switching power flow constraints are as follows:

[0072] in, For Node The soft switch at time The active power, For Node The soft switch at time The active power, For Node The soft switch at time The active power loss is For Node The soft switch at time The active power loss is is the node correlation coefficient of the soft switch, For Node The soft switch at time The reactive power, and For point The upper and lower limits of the reactive power of the soft switch at For point The apparent power capacity of the soft switch at.

[0073] Operation constraints of flexible loads:

[0074] in, For Node The power variation of the flexible load at For Node The maximum allowable range of power variation of the flexible load at To calculate the duration, For Node The flexible load at time The initial power, is the minimum efficiency factor.

[0075] Energy storage system charging and discharging control constraints:

[0076] in, , is a binary variable, indicating Whether the energy storage device is in the state of discharge or charging at all times, for The energy storage device discharge power at any moment, is the maximum discharge power of the energy storage system, for The charging power of the energy storage device at all times, is the maximum charging power of the energy storage system, for The energy storage capacity of the energy storage system at any moment, for The energy storage capacity of the energy storage system at any moment, for The charging power of the energy storage device at all times, for The energy storage device discharge power at any moment, is the charging efficiency coefficient, is the discharge efficiency coefficient, The total capacity of the energy storage system, is the minimum available capacity of the energy storage system, The maximum available capacity of the energy storage system.

[0077] Carbon emission constraints:

[0078] in, is the carbon emission cost, for Time Node The output power, is the coefficient of the quadratic term, is the linear term coefficient, represents distributed generation, Represents energy storage system.

[0079] Step 3: Design an incremental reinforcement learning model in a multi-agent environment.

[0080] Based on the basic elements of multi-agent deep reinforcement learning, the low-carbon economic dispatch model considering the coordination of source, grid, load and storage is designed as an incremental reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function, as follows: 1. State Space The state space of the system mainly includes the load data of each agent , Actual output power of distributed power supply , reactive power ; Actual output power of energy storage device , Intelligent soft switch active output , reactive power , the cost factor includes the power loss cost factor , Switching cost coefficient of on-load tap changer , the switching cost coefficient of the capacitor bank , Carbon emission cost coefficient , node voltage ,Right now

[0081] 2. Action Space The action space variables correspond to the control variables of the system under study. In the system, each regional power grid acts as an intelligent agent. According to the constraints of the intelligent agent, its action space variables include the actual output power of the distributed generation. , intelligent soft switch active output , reactive power , the actual output power of the energy storage device ,Right now

[0082] 3. Environmental design On the basis of the completion of the reinforcement learning agent training, each agent will evaluate the environment after the data change according to the new data received, and evaluate the current environmental drift state by using the reward value obtained by using the original optimal strategy decision in the new and old environments. If environmental drift is detected, incremental reinforcement learning is started, and the drift parameters are used to initialize the parameters of the agent in the new environment in the form of a convex combination of the random initialization strategy and the original optimal strategy. The low-carbon economic dispatch model with source, grid, load and storage coordination is used as the environment. When training the Actor-Critic network of the agent, the importance sampling method is added to adjust the weights of the data samples generated by the strategy to effectively cope with the impact of non-stationary environments and speed up training. When updating the strategy, the elastic weight consolidation method is added. While the agent is learning new tasks, the parameter update of the constraint model is used to help the agent maintain its memory of previous tasks when facing new environmental tasks to avoid catastrophic forgetting, and the cycle is repeated.

[0083] 4. Reward Function The reward function will affect the convergence of the algorithm to a certain extent, so the setting of the reward signal needs to be able to convey the goal that the agent wants to achieve, so as to guide the agent to improve its actions in the direction of maximizing the reward function. The opposite of the objective function of the multi-regional power grid model is used as the immediate reward for each agent. In the optimization problem, the corresponding constraints must also be met. According to the constraints described in the present invention, if the corresponding variable does not meet the constraints, a penalty value is set. , together with the immediate reward, is used as the agent's final reward function, calculated as follows:

[0084] Step 4: Use the MAIRL-EEPP method to solve.

[0085] Based on the low-carbon economic dispatch model considering the coordination of source, grid, load and storage constructed above, the MAIRL-EEPP method is used to solve it. Actor obtains its own status information , For intelligent agents Actions taken, To obtain the reward, is its own weight parameter, For the original environment, is the new environment after the data changes. After that, the agent first uses The optimal strategy learned from Perform multiple evaluation experiments and receive rewards , then, in the environment Perform the same action in , get reward , by comparison and , the agent detects whether the environment is drifting. The detection rules are:

[0086] in, is the old optimal strategy in the environment Rewards for execution, is the old optimal strategy in the environment Rewards for execution, is the defined drift boundary value.

[0087] If the environment drifts, incremental learning is started, and the strategy applicability rate is calculated , the rules are:

[0088] in, is the set of states in all evaluation experiments The average number of states in is the average number of drift states identified across all evaluation experiments.

[0089] In environmental changes Finally, when learning the optimal strategy, in order to prevent local optimality, the strategy in the new environment is initialized by combining the degree of environmental drift and using convex combination. The initialization rules are as follows:

[0090] in, is the initialization strategy in the new environment, For strategic applicability, According to the strategy suitability The calculated parameter factors, is the optimal strategy for the old environment, The random initialization strategy Parameters, is a hyperparameter that determines the degree of perturbation.

[0091] In the reinforcement learning process, Agents, observation set Status information for all agents. Actor continuously updates its own parameters To maximize the expected value of its own reward, in order to speed up the strategy update in a non-stationary environment, the importance weight is added, and the Actor's strategy update rule is:

[0092] in, is the policy network objective function Strategy The gradient of For strategy Download all tracks Perform expectation calculations, For a given state When you select Action The gradient of the log probability of the policy, From the time step Discounts on future rewards starting and, is the importance weight, is the normalization parameter, is a smoothing parameter that prevents small return events from having a weight of 0.

[0093] During the training process, when the strategy is iterated, in order to protect the important parameters in the old tasks, the elastic weight consolidation method (EWC) is introduced in the objective function to avoid catastrophic forgetting, thereby achieving the purpose of incremental reinforcement learning. The rules introduced are as follows:

[0094] in, The objective function of the policy network after the elastic weight consolidation method is introduced. is the original policy network objective function, To adjust the hyperparameters of the regularization weights, is the Fisher information matrix, are the parameters of the current policy network, is the optimal strategy parameter in the original environment.

[0095] Therefore, the Actor's final strategy update is:

[0096] in, The objective function of the policy network after adding elastic consolidation weights Strategy The gradient of is the policy network objective function Strategy The gradient of To adjust the hyperparameters of the regularization weights, is the Fisher information matrix, are the parameters of the current policy network, is the optimal strategy parameter in the original environment, For strategy Download all tracks Perform expectation calculations, For a given state When you select Action The gradient of the log probability of the policy, From the time step Discounts on future rewards starting and, is the importance weight, is the learning rate.

[0097] According to the solution principle of MAIRL-EEPP, the overall algorithm flow of the low-carbon economic dispatch model based on MAIRL-EEPP considering the coordination of source, grid, load and storage is as follows: Step 1: Initialize the environment. Load the low-carbon economic dispatch model that considers the coordination of source, grid, load and storage into the environment, and set the interface file with the state and action in the MAIRL-EEPP algorithm so that the power flow calculation can be performed in real time according to the state action and the corresponding environmental state quantity can be fed back.

[0098] Step 2: Based on the trained agents, each agent:

[0099] Detect the degree of environmental drift and synchronize and initialize parameters. Set the training schedule period T after the environment changes and set the number of training rounds M for each agent. The initial values ​​are all set to 1. Initially, the agent network parameters are randomly set. .

[0100] Step 3: Each agent interacts with the environment. Each agent executes a strategy in the current environment. , select actions based on the actions output by the policy network , and observe the next state and rewards , repeat until the maximum number of training rounds is reached, and record the state, action, and reward of each time step. Step 4: Update the network parameters. For each time step, calculate the cumulative reward from the current time to the terminal state ,according to: Calculate policy gradients and perform parameter updates.

[0101] Step 5: Determine whether the current number of training rounds m reaches the set value M. If it reaches the set value, end the training, output the results and save them. If it does not reach the set value, return to step 2 and start a new round of training.

[0102] The above model is programmed and solved using simulation software. By comparing it with the centralized optimization solution, it is verified that the trained model not only has a good ability to deal with uncertainty, but also can realize real-time decision-making of the system, which is conducive to online application. By comparing it with the traditional multi-agent deep reinforcement learning method of retraining after data changes, it is verified that the MAIRL-EEPP method has a faster convergence speed and better training results in the training process, effectively avoiding catastrophic forgetting, and has obvious advantages in dealing with complex environmental problems.

[0103] The incremental reinforcement learning optimization method for low-carbon economic dispatch of source-grid-load-storage distribution network proposed in the present invention is mainly aimed at the fact that the current data-driven intelligent collaborative optimization model in the power grid cannot effectively cope with the dynamically changing and gradually increasing massive data in the power system. Therefore, the present invention constructs a low-carbon economic dispatch model considering the collaboration of source-grid-load-storage, adopts a method based on multi-agent incremental reinforcement learning, takes environmental reward evaluation as the incremental start criterion, and adds importance sampling and elastic consolidation weight methods to adapt to the ever-changing environment, solving the optimization problem of online update of the current data-driven optimization model. It has the following advantages: The difference between the present invention and the traditional power grid optimization dispatching method lies in: (1) It adopts the framework of multi-agent incremental reinforcement learning, which can respond to changes in power grid data online while avoiding catastrophic forgetting, quickly respond to needs and adapt to non-stationary environments. (2) In the proposed incremental optimization method, the environment can help the agent detect environmental changes during training, avoiding excessive exploration risks, speeding up training time and improving system flexibility. (3) It adopts the convex combination of the previous optimal strategy and the random strategy to quickly initialize the training parameters, retain important parameters during the training process to avoid catastrophic forgetting, ensure the ability to process past data, reduce the risk of operational fluctuations, and improve the robustness of decision-making.

[0104] In yet another embodiment of the present invention, a distribution network source-grid-load-storage low-carbon optimal scheduling incremental reinforcement learning system is provided, which can be used to implement the above-mentioned distribution network source-grid-load-storage low-carbon optimal scheduling incremental reinforcement learning method. Specifically, the system includes: The data acquisition module is used to divide the power grid into intelligent agents, obtain multiple intelligent agent areas, and collect observation data for each intelligent agent area; The model building module is used to build a low-carbon economic dispatch model with source-grid-load-storage coordination based on observation data and the operating cost and carbon emissions of each intelligent regional power grid as the objective function; The model design module is used to design the low-carbon economic dispatch model of source-grid-load-storage coordination into a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function; The optimization solution module is used to solve the reinforcement learning model in a multi-agent environment and obtain the optimal scheduling solution.

[0105] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0106] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the incremental reinforcement learning method for low-carbon optimization scheduling of distribution network source network load storage.

[0107] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the incremental reinforcement learning method for low-carbon optimization scheduling of distribution network source network load storage in the above embodiment.

[0108] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An incremental reinforcement learning method for low-carbon optimal dispatching of distribution network sources, grids, loads and storage, characterized in that: include: Divide the power grid into intelligent agents to obtain multiple intelligent agent regions, and collect observation data for each intelligent agent region; Based on the observed data, a low-carbon economic dispatch model with source-grid-load-storage coordination is constructed with the operating cost and carbon emissions of each intelligent regional power grid as the objective function; The low-carbon economic dispatch model of source-grid-load-storage coordination is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function; Solve the reinforcement learning model in a multi-agent environment and obtain the optimal scheduling solution.

2. The incremental reinforcement learning method for low-carbon optimal scheduling of distribution network source, grid, load and storage according to claim 1 is characterized in that: The power grid is divided into multiple intelligent agent areas, including: With the goal of making the connection between regions a single loop, the power flow between regions a single direction, and the power grid within the group unconstrained, the power grid is divided into different regions, and different intelligent agents are set according to the regions, obtaining multiple intelligent agent regions.

3. The incremental reinforcement learning method for low-carbon optimal scheduling of distribution network source, grid, load and storage according to claim 1 is characterized in that: The observation data include: Load data , Actual output power of distributed power supply , reactive power , Actual output power of energy storage device , Intelligent soft switch active output , reactive power , Power loss cost coefficient , Switching cost coefficient of on-load tap changer , the switching cost coefficient of the capacitor bank and carbon emission cost coefficient .

4. The incremental reinforcement learning method for low-carbon optimal scheduling of distribution network source, grid, load and storage according to claim 3 is characterized in that: Based on the observed data, a low-carbon economic dispatch model with source-grid-load-storage coordination is constructed with the operating cost and carbon emissions of each intelligent regional power grid as the objective function, including: The objective function is as follows: in, and is the weight coefficient, is the number of partitioned regions, For Region Power loss cost, For Region Switch operating costs, For Region Carbon emission costs, For Region Voltage deviation degree; The constraints of the low-carbon economic dispatch model with coordinated source, grid, load and storage include the constraints of the active distribution network flow, the constraints of on-load tap changers and capacitor banks, and the operation constraints of other regulating resources.

5. The incremental reinforcement learning method for low-carbon optimal scheduling of distribution network source, grid, load and storage according to claim 4 is characterized in that: The low-carbon economic dispatch model for source-grid-load-storage coordination is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function, including: State Space The state space of the system includes the load data of each agent , Actual output power of distributed power supply , reactive power , Actual output power of energy storage device , Intelligent soft switch active output , reactive power , Power loss cost coefficient , Switching cost coefficient of on-load tap changer , the switching cost coefficient of the capacitor bank , Carbon emission cost coefficient and node voltage ,Right now Action Space The action space variables include the actual output power of distributed power sources, the active output of intelligent soft switches, , reactive power and the actual output power of the energy storage device ,Right now Environmental Design After the training of reinforcement learning agents is completed, each agent evaluates the environment after the data changes according to the new data received, and evaluates the current environment drift state by using the reward value obtained by using the original optimal strategy decision in the new and old environments. If environmental drift is detected, incremental reinforcement learning is started, and the drift parameters are used to initialize the parameters of the agent in the new environment in the form of a convex combination of the random initialization strategy and the original optimal strategy. The low-carbon economic dispatch model with source-grid-load-storage coordination is used as the environment. When training the Actor-Critic network of the agent, importance sampling is added, and the weights of the data samples generated by the strategy are adjusted. When updating the strategy, elastic weight consolidation is added. While the agent is learning new tasks, the parameters of the constraint model are updated; Reward Function The inverse of the objective function of the multi-regional power grid model is used as the immediate reward for each agent F , according to the constraints, if the corresponding variable does not meet the constraints, then set the penalty value , together with the immediate reward, is used as the agent's final reward function, calculated as follows: 。 6. The incremental reinforcement learning method for low-carbon optimal dispatching of distribution network source, grid, load and storage according to claim 5 is characterized in that: Solving the reinforcement learning model in a multi-agent environment to obtain an optimal scheduling solution includes: The MAIRL-EEPP method is used for solving, and the interface files with the states and actions in the MAIRL-EEPP method are set up so that the power flow calculation can be performed in real time according to the state actions and the corresponding environmental state quantities can be fed back; Based on the trained agents, each agent is trained according to the formula: Detect the degree of environmental drift and synchronize and initialize parameters, where: is the old optimal strategy in the environment Rewards for execution, is the old optimal strategy in the environment Rewards for execution, is the defined drift boundary value; Set the training schedule period T after the environment changes and set the number of training rounds M for each agent. The initial values ​​are all set to 1. Initially, the agent network parameters are randomly set. ; Each agent interacts with the environment and executes strategies in the current environment , select actions based on the actions output by the policy network , and observe the next state and rewards , repeat until the maximum number of training rounds is reached, and record the state, action, and reward of each time step; Update each network parameter: For each time step, calculate the cumulative reward from the current time to the terminal state , according to the formula Calculate the policy gradient and update the parameters, where The objective function of the policy network after adding elastic consolidation weights Strategy The gradient of is the policy network objective function Strategy The gradient of To adjust the hyperparameters of the regularization weights, is the Fisher information matrix, are the parameters of the current policy network, is the optimal strategy parameter in the original environment, For strategy Download all tracks Perform expectation calculations, For a given state When you select Action The gradient of the log probability of the policy, From the time step Discounts on future rewards starting and, is the importance weight, is the learning rate; Determine whether the current number of training rounds m reaches the set value M. If it reaches the set value, end the training, output the results and save them. If it does not reach the set value, start a new round of training.

7. The incremental reinforcement learning system for low-carbon optimization scheduling of distribution network source, grid, load and storage is characterized by: include The data acquisition module is used to divide the power grid into intelligent agents, obtain multiple intelligent agent areas, and collect observation data for each intelligent agent area; The model building module is used to build a low-carbon economic dispatch model with source-grid-load-storage coordination based on observation data and the operating cost and carbon emissions of each intelligent regional power grid as the objective function; The model design module is used to design the low-carbon economic dispatch model of source-grid-load-storage coordination into a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function; The optimization solution module is used to solve the reinforcement learning model in a multi-agent environment and obtain the optimal scheduling solution.

8. The distribution network source-grid-load-storage low-carbon optimization dispatching system according to claim 7 is characterized in that: The observation data include: Load data , Actual output power of distributed power supply , reactive power , Actual output power of energy storage device , Intelligent soft switch active output , reactive power , Power loss cost coefficient , Switching cost coefficient of on-load tap changer , the switching cost coefficient of the capacitor bank and carbon emission cost coefficient .

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the incremental reinforcement learning method for low-carbon optimized scheduling of distribution network source, grid, load and storage are implemented as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the incremental reinforcement learning method for low-carbon optimized scheduling of distribution network source, grid, load and storage are implemented as described in any one of claims 1 to 6.

Images