A multi-objective coordinated optimization transformer area energy storage control method and system

Abstract

The application discloses a kind of multi-target coordination optimization's transformer area energy storage control method and system, method includes: the reinforcement learning state space including photovoltaic accessible capacity margin is constructed;Define active and reactive power control action, state transition process synchronously updates energy storage state of charge and photovoltaic accessible capacity margin;Establish six target coordination optimization function covering voltage deviation, voltage out of limit, reverse power flow, three-phase imbalance, energy storage life loss and photovoltaic carrying capacity improvement;Build dynamic weight adjustment mechanism driven by operation risk and target deviation;According to target time scale division, build three-level reward function including instant operation, photovoltaic carrying and energy storage life;Using reinforcement learning algorithm trains energy storage control strategy and on-line deployment closed-loop control.The application can realize transformer area voltage optimization, reverse power flow suppression, three-phase imbalance governance, photovoltaic access space release and energy storage life protection of synergistic optimization.

Description

Technical Field

[0001] This invention belongs to the field of power distribution network operation control and energy storage optimization technology, and particularly relates to a multi-objective coordinated optimization method and system for transformer substation energy storage control. Background Technology

[0002] The installed capacity of distributed photovoltaic (PV) power in rural low-voltage distribution substations is continuously increasing, and high-proportion distributed PV grid integration has become an important trend in the development of rural distribution networks in my country. However, due to the long lines, large load fluctuations, and relatively weak network structure of rural distribution networks, centralized integration of distributed PV can easily lead to problems such as node voltage exceeding limits, increased reverse power flow, exacerbated three-phase imbalance, and insufficient local absorption capacity, thus limiting the further integration capacity of renewable energy in the substations. Therefore, how to utilize energy storage systems to achieve coordinated operation of distribution substations under high-proportion distributed PV grid integration has become an important research direction in the field of distribution network operation and control.

[0003] Existing technologies typically employ rule-based control, model predictive control, or optimized scheduling methods to achieve energy storage charging and discharging control, aiming to reduce voltage deviation, suppress reverse power flow, or improve the absorption capacity of new energy sources. Among these, with the development of artificial intelligence technology, reinforcement learning, capable of autonomously learning control strategies through environmental interaction, is increasingly being applied to the field of energy storage optimization control. Reinforcement learning methods do not require the establishment of precise mathematical models, possess good adaptability and online decision-making capabilities, and can adapt to the complex operating environment brought about by random fluctuations and load changes in distributed photovoltaic systems.

[0004] However, existing reinforcement learning-based energy storage control methods still have the following shortcomings: (1) The existing reinforcement learning state space mainly includes the energy storage charge state, load power, photovoltaic output, voltage and other operating states. The control objective is usually limited to the optimization of the current operating state. There is a lack of state variables that can reflect the impact of energy storage regulation on the future new energy access capacity, which makes it difficult for the control strategy to take into account both the current operating safety and the future new energy carrying capacity improvement.

[0005] (2) Existing multi-objective optimization methods typically use fixed weights to comprehensively consider multiple optimization objectives such as voltage, current, power flow, and energy storage operation. However, in actual operation, the operational risks faced by the transformer substation at different times exhibit significant dynamic changes. For example, voltage overruns and reverse power flow problems are more prominent during the midday peak photovoltaic period, while the energy storage operation status is of greater concern at night. Fixed weights make it difficult to dynamically adjust the importance of each optimization objective based on the real-time operating status, thus affecting the control effect.

[0006] (3) Existing reinforcement learning reward functions are mainly constructed based on the current control effect. They are usually designed around voltage control, power flow optimization or economic operation. They lack effective evaluation of the contribution of energy storage control actions to improving the future photovoltaic access capability, and do not fully consider the life loss problem caused by frequent charging and discharging of energy storage. This makes reinforcement learning strategies prone to short-sighted optimization, which is not conducive to the long-term stable operation of energy storage systems.

[0007] (4) Most existing energy storage control strategies are optimized on a single time scale and lack a unified evaluation mechanism that takes into account immediate operational safety, future new energy carrying capacity improvement and long-term energy storage life protection. This makes it difficult to meet the multi-objective and multi-time scale collaborative optimization control needs of distribution radio stations under the condition of high proportion of distributed photovoltaic access.

[0008] Therefore, it is necessary to propose a multi-objective coordinated optimization method for distribution area energy storage that can sense changes in photovoltaic carrying capacity, dynamically adjust and optimize target weights, and take into account the optimization of current operating status, the improvement of future new energy access capacity, and the protection of long-term energy storage life, so as to improve the comprehensive operating performance and new energy absorption capacity of high-penetration distributed photovoltaic access distribution areas. Summary of the Invention

[0009] This invention aims to solve the technical problems of existing reinforcement learning energy storage control methods, such as focusing only on the optimization of the current operating state, difficulty in taking into account the future improvement of new energy carrying capacity, fixed optimization target weights, and insufficient energy storage life protection. It provides a multi-objective coordinated optimization method and system for transformer substation energy storage control.

[0010] In a first aspect, the present invention provides a multi-objective coordinated optimization method for transformer substation energy storage control, comprising: Collect the current operating status information of the transformer area, and construct the reinforcement learning state space for the current moment based on the operating status information; The energy storage system is treated as an intelligent agent. Energy storage control actions are defined and input into a preset state transition model based on the current reinforcement learning state space and the energy storage control actions. The state transition model updates the energy storage state of charge and photovoltaic access capacity margin at the next moment and outputs the reinforcement learning state space at the next moment. Using the reinforcement learning state space at the current moment and the reinforcement learning state space at the next moment as input, a multi-objective coordinated optimization function is constructed, which contains multiple optimization objective values; The multiple optimization target values ​​are input into a preset dynamic weight adjustment mechanism to calculate the operational risk index and target deviation degree corresponding to each optimization target, and the weight of each optimization target is dynamically updated according to the operational risk index and target deviation degree. The multiple optimization target values ​​and the updated weights of each optimization target are input into a preset hierarchical reinforcement learning reward function to construct immediate operation reward, photovoltaic carrying capacity reward and energy storage life reward respectively. The rewards of each layer are weighted and summed to obtain a comprehensive reward function. The current reinforcement learning state space, the energy storage control action, the next reinforcement learning state space, and the comprehensive reward function are used as reinforcement learning training samples. The energy storage control strategy is trained using a reinforcement learning algorithm, and the policy network is iteratively updated to obtain the optimal control strategy. The optimal control strategy is deployed to the energy storage management system. In each control cycle, the real-time state space is collected to input the optimal control strategy, and the energy storage control action is output and executed.

[0011] Secondly, the present invention provides a multi-objective coordinated optimization system for transformer substation energy storage, comprising: The first construction module is configured to collect the current operating status information of the transformer area and construct the reinforcement learning state space at the current moment based on the operating status information. The first output module is configured to treat the energy storage system as an intelligent agent, define energy storage control actions, and input the current reinforcement learning state space and the energy storage control actions into a preset state transition model. The state transition model updates the energy storage state of charge and photovoltaic access capacity margin at the next moment, and outputs the reinforcement learning state space at the next moment. The solution module is configured to construct a multi-objective coordinated optimization function with the reinforcement learning state space at the current time and the reinforcement learning state space at the next time as input. The multi-objective coordinated optimization function contains multiple optimization objective values. The update module is configured to input the multiple optimization target values ​​into a preset dynamic weight adjustment mechanism, calculate the operational risk indicators and target deviation degree corresponding to each optimization target, and dynamically update the weight of each optimization target according to the operational risk indicators and target deviation degree; The second construction module is configured to input the multiple optimization target values ​​and the updated weights of each optimization target into a preset hierarchical reinforcement learning reward function to construct an instant operation reward, a photovoltaic carrying capacity reward, and an energy storage life reward, respectively, and to sum the rewards of each layer by weight to obtain a comprehensive reward function; The training module is configured to use the current reinforcement learning state space, the energy storage control action, the next reinforcement learning state space, and the comprehensive reward function as reinforcement learning training samples, train the energy storage control strategy using a reinforcement learning algorithm, iteratively update the policy network, and obtain the optimal control strategy. The second output module is configured to deploy the optimal control strategy to the energy storage management system, and to collect the real-time state space input of the optimal control strategy in each control cycle, output the energy storage control action and execute it.

[0012] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the multi-objective coordinated optimization distribution area energy storage control method according to any embodiment of the present invention.

[0013] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor performs the steps of the multi-objective coordinated optimization distribution area energy storage control method according to any embodiment of the present invention.

[0014] The multi-objective coordinated optimization method and system for transformer area energy storage control in this application has the following beneficial effects: The weights of each optimization target are dynamically adjusted based on real-time operational risks and the degree of target deviation, enabling the reinforcement learning control strategy to automatically adjust the control focus for different operating conditions. Compared with the fixed weight method, it has better environmental adaptability and operational effect. When the photovoltaic output increases and the voltage rises, the weights of voltage deviation and voltage over-limit targets are automatically increased; when the reverse power flow increases, the weights of reverse power flow and photovoltaic carrying capacity targets are automatically increased.

[0015] Based on the time scale of different optimization objectives, the reward function is divided into immediate operation reward, future new energy carrying capacity reward, and long-term energy storage lifetime reward, and is evaluated uniformly by combining dynamic weights. This effectively alleviates the reward conflict problem in reinforcement learning and improves the efficiency and convergence stability of policy learning.

[0016] While improving the photovoltaic absorption and grid connection capabilities, the control strategy fully considers the full life cycle operation characteristics of the energy storage system. By introducing energy storage state of charge and life loss factors into the state space, objective function and reward function, the control strategy can avoid frequent deep charging and discharging and high power surges, delay the battery aging process and reduce operation and maintenance costs.

[0017] It has good engineering applicability and promotion value, especially suitable for rural areas, remote areas and distribution networks with high new energy penetration. Through the intelligent coordination and control of the energy storage system, it can effectively alleviate problems such as voltage fluctuations, reverse power flow and three-phase imbalance caused by the access of new energy, and improve the carrying capacity and operational reliability of the distribution network for distributed new energy. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a multi-objective coordinated optimization method for transformer substation energy storage control according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a multi-objective coordinated optimization substation energy storage control system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 The diagram shows a flowchart of a multi-objective coordinated optimization method for transformer area energy storage control according to this application.

[0022] like Figure 1 As shown, the multi-objective coordinated optimization method for transformer substation energy storage control specifically includes the following steps: Step S101: Collect the current operating status information of the transformer area and construct the reinforcement learning state space for the current moment based on the operating status information.

[0023] In this step, the current state of charge of energy storage, output power of distributed photovoltaic, load power of the transformer area, state of node voltage, reverse power flow, three-phase imbalance rate, and photovoltaic access capacity margin are obtained. The photovoltaic access capacity margin is the remaining capacity of the transformer area to further connect distributed photovoltaic under the current operating conditions, while meeting the voltage constraints, power flow constraints, and energy storage regulation capacity constraints. The state of charge of energy storage, the output power of distributed photovoltaics, the load power of the transformer area, the node voltage state, the reverse power flow, the three-phase imbalance rate, and the available photovoltaic capacity margin are combined to form the reinforcement learning state space at the current moment, which is expressed as: , In the formula, Let be the reinforcement learning state space at time t. Let t be the state of charge of the stored energy. Let be the output power of the distributed photovoltaic system at time t. Let t be the load power of the transformer area. Let t represent the voltage state of the transformer substation nodes. Let be the reverse power flow power at time t. Let be the three-phase imbalance rate at time t. Let t be the available photovoltaic capacity margin at time t.

[0024] Step S102: Treat the energy storage system as an intelligent agent, define energy storage control actions, and input the current reinforcement learning state space and the energy storage control actions into a preset state transition model. The state transition model updates the energy storage state of charge and photovoltaic access capacity margin at the next moment, and outputs the reinforcement learning state space at the next moment.

[0025] In this step, the energy storage control actions are defined. ,in, Let be the active power regulation of energy storage at time t, and satisfy . , To maximize the active power regulation capability of the energy storage system, Let the energy storage reactive power regulation power at time t be, and satisfy the following conditions: , This represents the maximum reactive power regulation capacity of the energy storage system. Update the energy storage state of charge based on the energy storage charging and discharging process: , In the formula, The energy storage state of charge at time t+1, Let t be the state of charge of the stored energy. To control the time interval, This refers to the rated capacity of the energy storage. Updated solar grid connection capacity margin: , , , In the formula, This represents the available photovoltaic capacity margin at time t+1. This represents the maximum allowable photovoltaic (PV) capacity margin for the distribution network under current operating conditions. This is the capacity margin adjustment factor. This refers to the change in the safety margin of the node voltage before and after energy storage regulation. This serves as the baseline value for the node voltage safety margin. For energy storage active power regulation, For energy storage reactive power regulation, This is the maximum reverse current allowed in the area. This is the proportional coefficient for active and reactive power regulation. For the coordinated gain of active and reactive power regulation, The preset adjustment coefficient, Let be the output power of the distributed photovoltaic system at time t. , , , , All are preset weighting coefficients; The updated energy storage state of charge at time t+1 and the photovoltaic access capacity margin at time t+1 are combined with the distributed photovoltaic output power at time t+1, the transformer load power at time t+1, the transformer node voltage state at time t+1, the reverse power flow power at time t+1, and the three-phase imbalance rate at time t+1 to form the reinforcement learning state space at time t+1.

[0026] Step S103: Using the reinforcement learning state space at the current moment and the reinforcement learning state space at the next moment as input, construct a multi-objective coordinated optimization function, which contains multiple optimization objective values.

[0027] In this step, the multi-objective coordinated optimization function includes voltage deviation optimization objective, voltage limit overrun optimization objective, reverse power flow optimization objective, three-phase imbalance optimization objective, energy storage lifetime loss optimization objective, and photovoltaic carrying capacity improvement objective; The expression for the voltage deviation optimization objective is as follows: , , In the formula, Let t be the target for optimizing the voltage deviation. This represents the number of nodes in the distribution area. Let be the actual voltage of the k-th node at time t. Rated voltage, The voltage sensitivity weight of the k-th node, Let be the photovoltaic output power of the k-th node. Let k be the load power of the k-th node. The photovoltaic impact coefficient, To prevent the division of zero by a small constant; The expression for the voltage over-limit optimization objective is: , In the formula, The optimization objective for voltage exceeding the limit at time t is... These are the maximum and minimum allowable voltages, respectively. This is the margin influence coefficient. Let t be the available photovoltaic capacity margin. This represents the maximum allowable photovoltaic (PV) capacity margin for the distribution network under current operating conditions. The expression for the reverse power flow optimization objective is: , In the formula, The objective for optimizing the reverse power flow at time t is... Let be the reverse power flow power at time t. The photovoltaic load matching factor is... The current adjustable capacity of energy storage at time t. To maximize the adjustable capacity of energy storage, Let be the output power of the distributed photovoltaic system at time t. Let t be the load power of the transformer area. The expression for the three-phase imbalance optimization objective is: , In the formula, The objective of the three-phase imbalance optimization at time t is... , , These are the currents of phase A, phase B, and phase C, respectively. , , These are the photovoltaic power connected to phase A, phase B, and phase C, respectively. This is the interphase power influence coefficient; The expression for the energy storage lifetime loss optimization objective is: , In the formula, The optimization objective is the energy storage lifetime loss at time t. This represents the increased photovoltaic utilization after energy storage control. As a weighting factor for photovoltaic power consumption revenue, The energy storage state of charge at time t-1, Let t be the state of charge of the stored energy. To control the time interval; The expression for the photovoltaic carrying capacity improvement target is: , In the formula, The target for improving photovoltaic carrying capacity at time t is... This represents the available photovoltaic capacity margin at time t+1. Let t be the available photovoltaic capacity margin at time t.

[0028] Step S104: Input the multiple optimization target values ​​into a preset dynamic weight adjustment mechanism, calculate the operation risk index and target deviation degree corresponding to each optimization target, and dynamically update the weight of each optimization target according to the operation risk index and target deviation degree.

[0029] In this step, the operational risk index is calculated, and the expression is: , In the formula, Let be the operational risk indicator for the i-th optimization objective at time t. Let be the i-th optimization objective at time t. Let be the expected value of the i-th optimization objective. To prevent the division of zero by a small constant; The degree of deviation from the target is calculated using the following expression: , In the formula, Let be the degree of deviation of the i-th optimization objective at time t. Let be the allowable deviation threshold for the i-th optimization objective. Let be the deviation sensitivity index for the i-th optimization objective; A comprehensive evaluation index is determined based on the operational risk indicators and the degree of deviation from the target, expressed as follows: , In the formula, The comprehensive evaluation index at time t To implement the weighting of risk indicators, Weights are assigned to the degree of deviation from the target.

[0030] Step S105: Input the multiple optimization target values ​​and the updated weights of each optimization target into a preset hierarchical reinforcement learning reward function to construct the instant operation reward, photovoltaic carrying capacity reward and energy storage life reward respectively. Then, sum the rewards of each layer by weight to obtain the comprehensive reward function.

[0031] In this step, build instant running rewards. The expression is: , In the formula, Let be the dynamic weights of the i-th optimization objective at time t. Let i be the optimization objective at time t; Building a photovoltaic carrying capacity incentive system The expression is: , In the formula, Let be the dynamic weight of the photovoltaic carrying capacity improvement target at time t. The target for improving photovoltaic carrying capacity at time t; Build energy storage lifetime rewards The expression is: , In the formula, The dynamic weights are used to optimize the energy storage lifetime loss target at time t. The target value for optimizing the energy storage lifetime loss at time t; The comprehensive reward function is calculated as follows: , In the formula, This is the comprehensive reward function.

[0032] Step S106: The reinforcement learning state space at the current moment, the energy storage control action, the reinforcement learning state space at the next moment, and the comprehensive reward function are used as reinforcement learning training samples. The energy storage control strategy is trained using a reinforcement learning algorithm, and the policy network is iteratively updated to obtain the optimal control strategy.

[0033] In this step, the reinforcement learning state space at the current moment is used. As input to the policy network and value network, energy storage control actions are used. As the output of the policy network, the comprehensive reward function As a strategy optimization target; Initialize the policy network parameters and value network parameters; In each training step, energy storage control actions are generated based on the current policy network. The environment transitions to the next state according to the state transition model. And calculate the comprehensive reward. , to use empirical samples Store in the experience replay pool; Randomly sample small batches of samples from the experience replay pool, update the value network parameters by minimizing the temporal difference error, and update the policy network parameters by maximizing the expected cumulative reward; Repeat the above training steps until the policy network converges and output the optimal policy.

[0034] Step S107: Deploy the optimal control strategy to the energy storage management system. In each control cycle, collect the real-time state space input of the optimal control strategy, output the energy storage control action and execute it.

[0035] In summary, the method of this application collects the operating status information of the transformer substation and constructs a reinforcement learning state space that includes the capacity margin for photovoltaic (PV) grid connection. It establishes an extended Markov decision model, defines the active and reactive power control actions for energy storage, and synchronously updates the energy storage state of charge and the PV grid connection capacity margin during the state transition process. It establishes a six-objective coordinated optimization function covering voltage deviation, voltage over-limit, reverse power flow, three-phase imbalance, energy storage lifetime loss, and PV carrying capacity improvement. It constructs a dynamic weight adjustment mechanism driven by operational risk and objective deviation. Based on the objective time scale, it constructs a three-level reward function including real-time operation, PV carrying capacity, and energy storage lifetime. It uses a reinforcement learning algorithm to train the energy storage control strategy and deploys closed-loop control online. This invention can achieve coordinated optimization of transformer substation voltage optimization, reverse power flow suppression, three-phase imbalance management, PV grid connection space release, and energy storage lifetime protection.

[0036] Please see Figure 2 The diagram shows a structural block diagram of a multi-objective coordinated optimization substation energy storage control system according to this application.

[0037] like Figure 2 As shown, the transformer area energy storage control system 200 includes a first construction module 210, a first output module 220, a solution module 230, an update module 240, a second construction module 250, a training module 260, and a second output module 270.

[0038] The system comprises the following modules: a first construction module 210, configured to collect the current operating status information of the distribution area and construct the reinforcement learning state space for the current moment based on the operating status information; a first output module 220, configured to treat the energy storage system as an intelligent agent, define energy storage control actions, and input the current reinforcement learning state space and the energy storage control actions into a preset state transition model, which updates the energy storage state of charge and photovoltaic accessibility margin for the next moment and outputs the reinforcement learning state space for the next moment; a solution module 230, configured to construct a multi-objective coordinated optimization function using the current reinforcement learning state space and the next reinforcement learning state space as input, the multi-objective coordinated optimization function containing multiple optimization objective values; and an update module 240, configured to input the multiple optimization objective values ​​into a preset dynamic weight adjustment mechanism to calculate the operating risk index and objective bias corresponding to each optimization objective. The system dynamically updates the weights of each optimization objective based on the operational risk indicators and the degree of deviation from the target. A second construction module 250 is configured to input the multiple optimization objective values ​​and the updated weights of each optimization objective into a preset hierarchical reinforcement learning reward function, constructing immediate operational rewards, photovoltaic carrying capacity rewards, and energy storage lifetime rewards respectively. The rewards at each level are weighted and summed to obtain a comprehensive reward function. A training module 260 is configured to use the current reinforcement learning state space, the energy storage control action, the next reinforcement learning state space, and the comprehensive reward function as reinforcement learning training samples. A reinforcement learning algorithm is used to train the energy storage control strategy, iteratively updating the strategy network to obtain the optimal control strategy. A second output module 270 is configured to deploy the optimal control strategy to the energy storage management system, collecting the real-time state space input of the optimal control strategy in each control cycle, outputting the energy storage control action, and executing it.

[0039] It should be understood that Figure 2 The modules and references described in the document Figure 1 The steps described in the text correspond to those in the method described above. Therefore, the operations, features, and corresponding technical effects described above also apply to the method described in the text. Figure 2 The various modules in the document will not be described in detail here.

[0040] In other embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor performs the multi-objective coordinated optimization distribution area energy storage control method in any of the above method embodiments. In one embodiment, the computer-readable storage medium of the present invention stores computer-executable instructions, which are configured as follows: Collect the current operating status information of the transformer area, and construct the reinforcement learning state space for the current moment based on the operating status information; The energy storage system is treated as an intelligent agent. Energy storage control actions are defined and input into a preset state transition model based on the current reinforcement learning state space and the energy storage control actions. The state transition model updates the energy storage state of charge and photovoltaic access capacity margin at the next moment and outputs the reinforcement learning state space at the next moment. Using the reinforcement learning state space at the current moment and the reinforcement learning state space at the next moment as input, a multi-objective coordinated optimization function is constructed, which contains multiple optimization objective values; The multiple optimization target values ​​are input into a preset dynamic weight adjustment mechanism to calculate the operational risk index and target deviation degree corresponding to each optimization target, and the weight of each optimization target is dynamically updated according to the operational risk index and target deviation degree. The multiple optimization target values ​​and the updated weights of each optimization target are input into a preset hierarchical reinforcement learning reward function to construct immediate operation reward, photovoltaic carrying capacity reward and energy storage life reward respectively. The rewards of each layer are weighted and summed to obtain a comprehensive reward function. The current reinforcement learning state space, the energy storage control action, the next reinforcement learning state space, and the comprehensive reward function are used as reinforcement learning training samples. The energy storage control strategy is trained using a reinforcement learning algorithm, and the policy network is iteratively updated to obtain the optimal control strategy. The optimal control strategy is deployed to the energy storage management system. In each control cycle, the real-time state space is collected to input the optimal control strategy, and the energy storage control action is output and executed.

[0041] Computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the multi-objective coordinated optimization substation energy storage control system, etc. Furthermore, the computer-readable storage medium may include high-speed random access memory, and may also include memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include memory remotely configured relative to a processor, which can be connected to the multi-objective coordinated optimization substation energy storage control system via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0042] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 3As shown, the device includes a processor 310 and a memory 320. The electronic device may also include an input device 330 and an output device 340. The processor 310, memory 320, input device 330, and output device 340 can be connected via a bus or other means. Figure 3 Taking a bus connection as an example, the memory 320 is the computer-readable storage medium described above. The processor 310 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 320, thereby realizing the multi-objective coordinated optimization distribution area energy storage control method described in the above embodiment. The input device 330 can receive input digital or character information and generate key signal inputs related to user settings and function control of the multi-objective coordinated optimization distribution area energy storage control system. The output device 340 may include a display screen or other display device.

[0043] The aforementioned electronic device can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0044] In one implementation, the aforementioned electronic device is applied in a multi-objective coordinated optimization distribution area energy storage control system for a client, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Collect the current operating status information of the transformer area, and construct the reinforcement learning state space for the current moment based on the operating status information; The energy storage system is treated as an intelligent agent. Energy storage control actions are defined and input into a preset state transition model based on the current reinforcement learning state space and the energy storage control actions. The state transition model updates the energy storage state of charge and photovoltaic access capacity margin at the next moment and outputs the reinforcement learning state space at the next moment. Using the reinforcement learning state space at the current moment and the reinforcement learning state space at the next moment as input, a multi-objective coordinated optimization function is constructed, which contains multiple optimization objective values; The multiple optimization target values ​​are input into a preset dynamic weight adjustment mechanism to calculate the operational risk index and target deviation degree corresponding to each optimization target, and the weight of each optimization target is dynamically updated according to the operational risk index and target deviation degree. The multiple optimization target values ​​and the updated weights of each optimization target are input into a preset hierarchical reinforcement learning reward function to construct immediate operation reward, photovoltaic carrying capacity reward and energy storage life reward respectively. The rewards of each layer are weighted and summed to obtain a comprehensive reward function. The current reinforcement learning state space, the energy storage control action, the next reinforcement learning state space, and the comprehensive reward function are used as reinforcement learning training samples. The energy storage control strategy is trained using a reinforcement learning algorithm, and the policy network is iteratively updated to obtain the optimal control strategy. The optimal control strategy is deployed to the energy storage management system. In each control cycle, the real-time state space is collected to input the optimal control strategy, and the energy storage control action is output and executed.

[0045] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-objective coordinated optimization method for transformer substation energy storage control, characterized in that, include: Collect the current operating status information of the transformer area, and construct the reinforcement learning state space for the current moment based on the operating status information; The energy storage system is treated as an intelligent agent. Energy storage control actions are defined and input into a preset state transition model based on the current reinforcement learning state space and the energy storage control actions. The state transition model updates the energy storage state of charge and photovoltaic access capacity margin at the next moment and outputs the reinforcement learning state space at the next moment. Using the reinforcement learning state space at the current moment and the reinforcement learning state space at the next moment as input, a multi-objective coordinated optimization function is constructed, which contains multiple optimization objective values; The multiple optimization target values ​​are input into a preset dynamic weight adjustment mechanism to calculate the operational risk index and target deviation degree corresponding to each optimization target, and the weight of each optimization target is dynamically updated according to the operational risk index and target deviation degree. The multiple optimization target values ​​and the updated weights of each optimization target are input into a preset hierarchical reinforcement learning reward function to construct immediate operation reward, photovoltaic carrying capacity reward and energy storage life reward respectively. The rewards of each layer are weighted and summed to obtain a comprehensive reward function. The current reinforcement learning state space, the energy storage control action, the next reinforcement learning state space, and the comprehensive reward function are used as reinforcement learning training samples. The energy storage control strategy is trained using a reinforcement learning algorithm, and the policy network is iteratively updated to obtain the optimal control strategy. The optimal control strategy is deployed to the energy storage management system. In each control cycle, the real-time state space is collected to input the optimal control strategy, and the energy storage control action is output and executed.

2. The multi-objective coordinated optimization method for transformer area energy storage control according to claim 1, characterized in that, The step of constructing the reinforcement learning state space at the current moment based on the running state information includes: The system acquires the current state of charge of energy storage, output power of distributed photovoltaic (PV) power, load power of the distribution area, node voltage status, reverse power flow, three-phase imbalance rate, and PV access capacity margin. The PV access capacity margin is the remaining capacity of the distribution area to further connect distributed PV under the current operating conditions, while meeting voltage constraints, power flow constraints, and energy storage regulation capacity constraints. The state of charge of energy storage, the output power of distributed photovoltaics, the load power of the transformer area, the node voltage state, the reverse power flow, the three-phase imbalance rate, and the available photovoltaic capacity margin are combined to form the reinforcement learning state space at the current moment, which is expressed as: ， In the formula, Let be the reinforcement learning state space at time t. Let t be the state of charge of the stored energy. Let be the output power of the distributed photovoltaic system at time t. Let t be the load power of the transformer area. Let t represent the voltage state of the transformer substation nodes. Let be the reverse power flow power at time t. Let be the three-phase imbalance rate at time t. Let t be the available photovoltaic capacity margin at time t.

3. The multi-objective coordinated optimization method for transformer area energy storage control according to claim 2, characterized in that, The process involves treating the energy storage system as an intelligent agent, defining energy storage control actions, and inputting the current reinforcement learning state space and the energy storage control actions into a preset state transition model. The state transition model updates the energy storage state of charge and the photovoltaic access capacity margin for the next time step, and outputs the reinforcement learning state space for the next time step, including: Define energy storage control actions ,in, Let be the active power regulation of energy storage at time t, and satisfy . , To maximize the active power regulation capability of the energy storage system, Let the energy storage reactive power regulation power at time t be, and satisfy the following conditions: , This represents the maximum reactive power regulation capacity of the energy storage system. Update the energy storage state of charge based on the energy storage charging and discharging process: ， In the formula, The energy storage state of charge at time t+1, Let t be the state of charge of the stored energy. To control the time interval, This refers to the rated capacity of the energy storage. Updated solar grid connection capacity margin: ， ， ， In the formula, This represents the available photovoltaic capacity margin at time t+1. This represents the maximum allowable photovoltaic (PV) capacity margin for the distribution network under current operating conditions. This is the capacity margin adjustment factor. This refers to the change in the safety margin of the node voltage before and after energy storage regulation. This serves as the baseline value for the node voltage safety margin. For energy storage active power regulation, For energy storage reactive power regulation, This is the maximum reverse current allowed in the area. This is the proportional coefficient for active and reactive power regulation. For the coordinated gain of active and reactive power regulation, The preset adjustment coefficient, Let be the output power of the distributed photovoltaic system at time t. , , , , All are preset weighting coefficients; The updated energy storage state of charge at time t+1 and the photovoltaic access capacity margin at time t+1 are combined with the distributed photovoltaic output power at time t+1, the transformer load power at time t+1, the transformer node voltage state at time t+1, the reverse power flow power at time t+1, and the three-phase imbalance rate at time t+1 to form the reinforcement learning state space at time t+1.

4. The multi-objective coordinated optimization method for transformer substation energy storage control according to claim 1, characterized in that, The multi-objective coordinated optimization function includes voltage deviation optimization objective, voltage limit overrun optimization objective, reverse power flow optimization objective, three-phase imbalance optimization objective, energy storage lifetime loss optimization objective, and photovoltaic carrying capacity improvement objective; The expression for the voltage deviation optimization objective is as follows: ， ， In the formula, Let t be the target for optimizing the voltage deviation. This represents the number of nodes in the distribution area. Let be the actual voltage of the k-th node at time t. Rated voltage, The voltage sensitivity weight of the k-th node, Let be the photovoltaic output power of the k-th node. Let k be the load power of the k-th node. The photovoltaic impact coefficient, To prevent the division into zero small constants; The expression for the voltage over-limit optimization objective is: ， In the formula, The optimization objective for voltage exceeding the limit at time t is... These are the maximum and minimum allowable voltages, respectively. This is the margin influence coefficient. Let t be the available photovoltaic capacity margin. This represents the maximum allowable photovoltaic (PV) capacity margin for the distribution network under current operating conditions. The expression for the reverse power flow optimization objective is: ， In the formula, The objective for optimizing the reverse power flow at time t is... Let be the reverse power flow power at time t. The photovoltaic load matching factor is... The current adjustable capacity of energy storage at time t. To maximize the adjustable capacity of energy storage, Let be the output power of the distributed photovoltaic system at time t. Let t be the load power of the transformer area. The expression for the three-phase imbalance optimization objective is: ， In the formula, The objective of the three-phase imbalance optimization at time t is... , , These are the currents of phase A, phase B, and phase C, respectively. , , These are the photovoltaic power connected to phase A, phase B, and phase C, respectively. This is the interphase power influence coefficient; The expression for the energy storage lifetime loss optimization objective is: ， In the formula, The optimization objective is the energy storage lifetime loss at time t. This represents the increased photovoltaic utilization after energy storage control. As a weighting factor for photovoltaic power consumption revenue, The energy storage state of charge at time t-1, Let t be the state of charge of the stored energy. To control the time interval; The expression for the photovoltaic carrying capacity improvement target is: ， In the formula, The target for improving photovoltaic carrying capacity at time t is... This represents the available photovoltaic capacity margin at time t+1. Let t be the available photovoltaic capacity margin at time t.

5. The multi-objective coordinated optimization method for transformer area energy storage control according to claim 1, characterized in that, The step of inputting the multiple optimization target values ​​into a preset dynamic weight adjustment mechanism, calculating the operational risk index and target deviation degree corresponding to each optimization target, and dynamically updating the weight of each optimization target based on the operational risk index and target deviation degree includes: The operational risk indicator is calculated using the following expression: ， In the formula, Let be the operational risk indicator for the i-th optimization objective at time t. Let be the i-th optimization objective at time t. Let be the expected value of the i-th optimization objective. To prevent the division into zero small constants; The degree of deviation from the target is calculated using the following expression: ， In the formula, Let be the degree of deviation of the i-th optimization objective at time t. Let be the allowable deviation threshold for the i-th optimization objective. Let be the deviation sensitivity index for the i-th optimization objective; A comprehensive evaluation index is determined based on the operational risk indicators and the degree of deviation from the target, expressed as follows: ， In the formula, The comprehensive evaluation index at time t, To determine the weighting of operational risk indicators, Weights are assigned to the degree of deviation from the target.

6. The multi-objective coordinated optimization method for transformer area energy storage control according to claim 1, characterized in that, The process involves inputting the multiple optimization target values ​​and the updated weights of each optimization target into a preset hierarchical reinforcement learning reward function to construct immediate operation rewards, photovoltaic carrying capacity rewards, and energy storage lifetime rewards, respectively. The weighted summation of the rewards at each level yields a comprehensive reward function, including: Build instant running rewards The expression is: ， In the formula, Let be the dynamic weights of the i-th optimization objective at time t. Let i be the optimization objective at time t; Building a photovoltaic carrying capacity incentive system The expression is: ， In the formula, Let be the dynamic weight of the photovoltaic carrying capacity improvement target at time t. The target for improving photovoltaic carrying capacity at time t; Building Energy Storage Lifetime Incentives The expression is: ， In the formula, The dynamic weights are used to optimize the energy storage lifetime loss target at time t. The target value for optimizing the energy storage lifetime loss at time t; The comprehensive reward function is calculated as follows: ， In the formula, This is the comprehensive reward function.

7. The multi-objective coordinated optimization method for transformer substation energy storage control according to claim 1, characterized in that, The step of using the current reinforcement learning state space, the energy storage control action, the next reinforcement learning state space, and the comprehensive reward function as reinforcement learning training samples, training the energy storage control policy using a reinforcement learning algorithm, iteratively updating the policy network, and obtaining the optimal control policy includes: The reinforcement learning state space at the current moment As input to the policy network and value network, energy storage control actions are used. As the output of the policy network, the comprehensive reward function As a strategy optimization target; Initialize the policy network parameters and value network parameters; In each training step, energy storage control actions are generated based on the current policy network. The environment transitions to the next state according to the state transition model. And calculate the comprehensive reward. , to use empirical samples Store in the experience replay pool; Randomly sample small batches of samples from the experience replay pool, update the value network parameters by minimizing the temporal difference error, and update the policy network parameters by maximizing the expected cumulative reward; Repeat the training steps until the policy network converges and outputs the optimal policy.

8. A multi-objective coordinated optimization system for transformer substation energy storage, characterized in that, include: The first construction module is configured to collect the current operating status information of the transformer area and construct the reinforcement learning state space at the current moment based on the operating status information. The first output module is configured to treat the energy storage system as an intelligent agent, define energy storage control actions, and input the current reinforcement learning state space and the energy storage control actions into a preset state transition model. The state transition model updates the energy storage state of charge and photovoltaic access capacity margin at the next moment, and outputs the reinforcement learning state space at the next moment. The solution module is configured to construct a multi-objective coordinated optimization function with the reinforcement learning state space at the current time and the reinforcement learning state space at the next time as input. The multi-objective coordinated optimization function contains multiple optimization objective values. The update module is configured to input the multiple optimization target values ​​into a preset dynamic weight adjustment mechanism, calculate the operational risk indicators and target deviation degree corresponding to each optimization target, and dynamically update the weight of each optimization target according to the operational risk indicators and target deviation degree; The second construction module is configured to input the multiple optimization target values ​​and the updated weights of each optimization target into a preset hierarchical reinforcement learning reward function to construct an instant operation reward, a photovoltaic carrying capacity reward, and an energy storage life reward, respectively, and to sum the rewards of each layer by weight to obtain a comprehensive reward function; The training module is configured to use the current reinforcement learning state space, the energy storage control action, the next reinforcement learning state space, and the comprehensive reward function as reinforcement learning training samples, train the energy storage control strategy using a reinforcement learning algorithm, iteratively update the policy network, and obtain the optimal control strategy. The second output module is configured to deploy the optimal control strategy to the energy storage management system, and to collect the real-time state space input of the optimal control strategy in each control cycle, output the energy storage control action and execute it.

9. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1 to 7.

Images