Self-adaptive net rack optimization method for improving large-scale water and electricity delivery capacity

Abstract

The invention provides a self-adaptive net rack optimization method for improving large-scale hydropower delivery capacity. The method comprises the steps that a double-layer power transmission network net rack optimization objective function is constructed by considering hydropower station delivery capacity; wherein the upper-layer optimization objective function is used for evaluating the economy of a power transmission network extension planning scheme, and the lower-layer optimization objective function is used for evaluating the renewable energy consumption level and the system reliability of the power transmission network; a multi-agent reinforcement learning method is adopted to configure an independent agent for each layer of optimization target for scheme search, and a scheme meeting the planning model is obtained through interlayer constraint conditions; wherein corresponding reward functions are set according to the upper and lower layer optimization objective function structures respectively. According to the method, two independent agents are constructed, the upper and lower model optimization objective functions are used as rewards for scheme search, the neural networks of the agents are independently trained according to the respective optimization process data, finally, the power transmission network extension planning scheme meeting the double-layer model is obtained, and the large-scale hydropower delivery capacity is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power system planning, and in particular to an adaptive grid optimization method for improving large-scale hydropower transmission capacity. Background Art

[0002] The existing grid structure in the power system usually adopts alternating current (AC) and direct current (DC) transmission. AC transmission is widely used due to its mature technology and relatively low construction cost, and is suitable for medium and short distance power transmission. However, for ultra-long distance power transmission, DC transmission has shown its superiority, especially in reducing energy loss during transmission and improving the stability of cross-regional power transmission. The DC transmission system is also convenient for connection and management with remote power grids, providing better control capabilities to cope with load changes.

[0003] In addition, modern grid designs also attempt to integrate smart grid technologies, including advanced monitoring systems and automation equipment, all of which are intended to improve the operational efficiency and reliability of the grid. By using smart grid technologies, the state of the grid can be monitored in real time, and the flow of electricity can be automatically adjusted to adapt to different power supply and demand conditions, thereby optimizing the performance of the entire power system. However, although these technologies offer significant advantages, their implementation and operation still face challenges of technical complexity and high costs. These systems require a high degree of technical expertise to design, build, and maintain, which limits their rapid promotion and application. In addition, the integration with emerging renewable energy technologies (such as wind and solar) also presents new technical requirements and challenges, requiring further technical innovation and system upgrades.

[0004] Deep reinforcement learning (DRL) provides a new approach to solving new power system expansion planning models. The traditional deep reinforcement learning-based transmission network expansion planning method needs to combine multi-objective indicator functions into a single reward function to assist algorithm training, which makes it difficult for the intelligent agent to identify the performance improvement of each indicator by line construction, and thus makes it difficult to obtain the optimal transmission network expansion planning scheme. Therefore, studying the deep reinforcement learning structure for solving the transmission network expansion planning model that is suitable for solving multi-objective models to improve the performance of the grid planning structure is an urgent problem to promote the application of this method in engineering.

[0005] For grid structure adaptive optimization methods to improve large-scale hydropower transmission capacity, there is always a need to pursue higher accuracy, stability and generalization. In terms of accuracy, previous grid structure models are usually more accurate, but the complex structure and large number of parameters of previous methods may cause performance fluctuations when facing different data sets or application scenarios. The stability of the method is particularly important. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0007] To this end, the present invention provides an adaptive grid optimization method for improving large-scale hydropower transmission capacity.

[0008] The present invention provides an adaptive grid optimization method for improving large-scale hydropower transmission capacity, comprising:

[0009] Considering the transmission capacity of hydropower stations, a two-layer transmission network optimization objective function is constructed; the upper optimization objective function is used to evaluate the economic feasibility of the transmission network expansion planning scheme, and the lower optimization objective function is used to evaluate the renewable energy consumption level and system reliability of the transmission network;

[0010] A multi-agent reinforcement learning method is used to configure independent agents for each layer of optimization objectives to search for solutions and obtain solutions that meet the planning model through inter-layer constraints; among them, the corresponding reward functions are set according to the optimization objective function structures of the upper and lower layers.

[0011] The adaptive grid optimization method for improving the large-scale hydropower transmission capacity according to the above technical solution of the present invention may also have the following additional technical features:

[0012] In the above technical solution, the optimization objective function of the double-layer transmission network is:

[0013]

[0014] Among them, F(X) represents the objective function of the double-layer transmission network optimization, f upper represents the upper optimization objective function, f lower represents the lower optimization objective function, C1(X) represents the transmission network construction cost, C2(X) represents the network loss cost, C3(X) represents the maintenance cost, P curt represents the reduction in hydroelectric power generation, P shed Indicates the load shedding amount, W J It indicates the electrical betweenness of the lines adjacent to the hydropower station.

[0015] In the above technical scheme, the constraints of the upper-level optimization objective function include power flow constraints and equipment operation constraints; the power flow constraints include balance constraints constructed according to the AC power flow; the equipment operation constraints include voltage amplitude and phase angle constraints, generator output constraints, line transmission capacity constraints and system load shedding constraints.

[0016] In the above technical solution, the constraint conditions of the upper-level optimization objective function are constructed according to the N-1 constraint conditions, and the constraint conditions of the upper-level optimization objective function under the N-1 constraint conditions include:

[0017]

[0018] in, represents the rated active power of the generator set at node j; represents the wind power active power discarded by node j; V j represents the voltage at node j; B jk represents the susceptance between node j and node k; G jk represents the conductivity between node j and node k; θ jk represents the phase angle between node j and node k; The load at node j absorbs active power; represents the active power of the removed load at node j; It represents the abandoned wind volume calculated according to the lower model; Indicates the minimum power output ratio; represents the active power output by the wind farm at node j;

[0019] represents the reactive power output by the generator at node j; represents the reactive power input of node j; represents the reactive power of wind power discarded by node j; represents the reactive power absorbed by the load at node j; represents the reactive power of the reactive compensation device at node j;

[0020] represents the minimum value of the voltage at node j; represents the maximum value of the voltage at node j;

[0021] Represents the minimum value of the phase angle of node j; Indicates the maximum value of the phase angle of node j;

[0022] Indicates the minimum value of the generator reactive power output; Indicates the maximum value of the reactive power output of the generator;

[0023] F l represents the power flow of line l; represents the maximum transmission apparent power of line l;

[0024] Indicates the minimum load rate; represents the amount of load abandonment calculated according to the lower-level optimization objective function;

[0025] The superscript N-1 indicates the corresponding operating state quantity of the system under the N-1 constraint condition.

[0026] In the above technical solution, the reward function R of the upper optimization objective function structure is set according to the comprehensive economic indicators of the transmission network. upper :

[0027] R upper =V base,upper -f upper ([C T (s t )] T )

[0028] Among them, V base,upper The reward function baseline represents the structure of the upper-level optimization objective function; C T (s t ) represents the comprehensive economic cost of the transmission network; s t represents the environmental state at time t, s t ∈S t , S t is the set of environmental states;

[0029] The reward function R of the lower optimization objective function structure is set according to the renewable energy consumption level and reliability of the transmission network. lower :

[0030]

[0031] Among them, V base,lower A reward function baseline that represents the structure of the underlying optimization objective function.

[0032] In the above technical solution, the multi-agent reinforcement learning method is used to configure an independent agent for each layer of optimization objectives to search for solutions and obtain solutions that meet the planning model through inter-layer constraints, including:

[0033] The process of searching for solutions with each layer's optimization objective function as a reward is defined as a layer of agents, that is, the upper layer's optimization objective function corresponds to the upper layer's agent, and the lower layer's optimization objective function corresponds to the lower layer's agent;

[0034] It is stipulated that during the solution search process, the upper-level agent needs to store three actions with the highest potential value each time and form a Pareto candidate set;

[0035] The Pareto candidate set is used as the optimization range of the lower-level intelligent agent, and the lower-level optimization objective function is used to judge the action value, and the lower-level intelligent agent determines the selection action within the Pareto candidate set;

[0036] When each action that determines the selection is executed, Qeval,upper and Qeval,lower are updated, and the parameters are copied to Qtarget,upper and Qtarget,lower according to a fixed period;

[0037] Among them, Qeval and Qtarget are value functions with the same initial parameters, Qeval,upper represents the function used by the upper-level agent to select the optimal value action, Qeval,lower represents the function used by the lower-level agent to select the optimal value action, Qtarget,upper represents the function used by the upper-level agent to predict the value of the selected optimal action, and Qtarget,lower represents the function used by the lower-level agent to predict the value of the selected optimal action.

[0038] In the above technical solution, the upper-layer agent uses the ε-greedy strategy to search for three routes with the greatest potential construction value to form a Pareto solution set S pareto {A t,upper}, Pareto solution set S pareto {A t,upper The composition rules of} include:

[0039]

[0040] Among them, Prob represents the random probability of the ε-greedy strategy, ε represents the exploration rate, ω eval,upper represents the process parameter in Qeval,upper value function, A max represents the best action, S t Indicates the environmental state, A t represents the selected action obtained by random selection, A t,upper Represents the selected action determined in the Pareto solution set.

[0041] In the above technical solution, the lower-level agent uses the ε-greedy strategy to search in the Pareto candidate set to determine the selected action A. t,lower , select action A t,lower The composition rules include:

[0042]

[0043] A t,lower ∈S pareto {A t,upper}

[0044] Among them, ω eval,lower Represents the process parameter in the Qeval,lower value function.

[0045] In the above technical solution, the multi-agent reinforcement learning method is trained based on a convolutional neural network.

[0046] In the above technical solution, the multi-agent reinforcement learning method is trained based on a deep convolutional network, including:

[0047] Initialize the transmission network expansion planning environment and initialize training parameters

[0048] Multi-agent reinforcement learning is performed based on the current environment state and agent parameters. The upper-layer agent searches to form a Pareto candidate set, and the lower-layer agent determines the selected action within the Pareto candidate set.

[0049] Execute actions and update the state of the reinforcement learning environment, calculate the reward functions of the upper and lower layer agents, form trajectory information and save it;

[0050] Determine whether the parameter update interval has been reached. If so, update the convolutional neural network parameters. If not, proceed to the next step.

[0051] Determine whether the grid value after updating according to the selected action is greater than the grid value before updating, if so, proceed to the next step, otherwise, search again;

[0052] Determine whether the maximum number of training scenes has been reached. If so, proceed to the next step. If not, initialize the environment state and search again.

[0053] Output the final transmission network expansion planning scheme.

[0054] In summary, due to the adoption of the above technical features, the beneficial effects of the present invention are:

[0055] The present invention proposes an adaptive grid optimization method for improving the large-scale hydropower transmission capacity based on a dual-agent deep reinforcement learning structure. The present invention constructs two independent DDQN agents, which respectively use the upper and lower layer model optimization objective functions as rewards to search for solutions and independently train their own neural networks based on their own optimization process data. Finally, the upper and lower layer constraints are used to obtain a transmission network expansion planning solution that meets the double-layer model, thereby improving the large-scale hydropower transmission capacity.

[0056] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0058] Figure 1 It is a flow chart of an adaptive grid optimization method for improving large-scale hydropower transmission capacity according to an embodiment of the present invention;

[0059] Figure 2 It is a schematic diagram of comprehensive economic cost in the PX power grid optimization and transformation scheme obtained by the method of the present invention;

[0060] Figure 3 It is a schematic diagram of the comprehensive economic cost of the PX power grid optimization and transformation scheme obtained by using the DQN method;

[0061] Figure 4 It is a schematic diagram comparing the line flow distribution of the PX power grid optimization transformation scheme obtained by the method of the present invention and the original grid;

[0062] Figure 5 It is a schematic diagram comparing the line flow distribution of the PX power grid optimization and transformation scheme obtained by the DQN method and the PX power grid optimization and transformation scheme obtained by the method of the present invention;

[0063] Figure 6 is a schematic diagram of an action feedback signal of an upper-level intelligent agent in one embodiment of the present invention;

[0064] Figure 7 It is a schematic diagram of the action feedback signal of the lower intelligent agent regarding the total amount of abandoned wind and abandoned load in one embodiment of the present invention;

[0065] Figure 8 It is a schematic diagram of the action feedback signal of the lower intelligent agent regarding the mean value of the K-order electrical betweenness entropy in one embodiment of the present invention;

[0066] Fig. 9 It is a schematic diagram comparing the prediction and changes of the line construction value during the training process of the method of the present invention and the DQN method. DETAILED DESCRIPTION

[0067] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0068] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0069] Refer to the following Figures 1 to 9 The following describes an adaptive grid optimization method for improving large-scale hydropower transmission capacity provided according to some embodiments of the present invention.

[0070] Some embodiments of the present application provide an adaptive grid optimization method for improving large-scale hydropower transmission capacity.

[0071] The first embodiment of the present invention proposes an adaptive grid optimization method for improving large-scale hydropower transmission capacity, which mainly includes two parts: constructing an optimization objective function and applying a multi-agent reinforcement learning method.

[0072] In constructing the optimization objective function, a two-layer transmission network framework optimization objective function is constructed by considering the transmission capacity of the hydropower station; among them, the upper optimization objective function is used to evaluate the economic feasibility of the transmission network expansion planning scheme, and the lower optimization objective function is used to evaluate the renewable energy consumption level and system reliability of the transmission network.

[0073] Specifically, the objective function of the double-layer transmission network optimization is:

[0074]

[0075] Among them, F(X) represents the objective function of the double-layer transmission network optimization, f upper represents the upper optimization objective function, f lower represents the lower optimization objective function, C1(X) represents the transmission network construction cost, C2(X) represents the network loss cost, C3(X) represents the maintenance cost, P curt represents the reduction in hydroelectric power generation, P shed Indicates the load shedding amount, W J It indicates the electrical betweenness of the lines adjacent to the hydropower station.

[0076] A two-layer optimization model was established with reference to different objectives in the double-layer transmission network optimization objective function.

[0077] In some embodiments, the constraints of the upper-level optimization objective function include power flow constraints and equipment operation constraints; the power flow constraints include balance constraints constructed based on AC power flow; the equipment operation constraints include voltage amplitude and phase angle constraints, generator output constraints, line transmission capacity constraints, and system load shedding constraints.

[0078] In this disclosure, the power flow constraint is a balance constraint based on the AC power flow, and its constraints are:

[0079]

[0080] in, represents the rated active power of the generator set at node j; represents the wind power active power discarded by node j; V j represents the voltage at node j; B jk represents the susceptance between node j and node k; G jk represents the conductivity between node j and node k; θ jk represents the phase angle between node j and node k; The load at node j absorbs active power; represents the active power of the removed load at node j.

[0081] represents the reactive power output by the generator at node j; represents the reactive power input of node j; represents the reactive power of wind power discarded by node j; represents the reactive power absorbed by the load at node j; Represents the reactive power of the reactive compensation device at node j.

[0082] The voltage amplitude and phase angle constraints are:

[0083]

[0084] in, represents the minimum value of the voltage at node j; represents the maximum value of the voltage at node j; Represents the minimum value of the phase angle of node j; Indicates the maximum value of the phase angle of node j.

[0085] Since hydropower plants have strong reactive power regulation capabilities, this paper does not impose special constraints. The output constraints of hydropower plants and general generators are:

[0086]

[0087] in, Indicates the minimum value of the generator's active power output; Indicates the maximum value of the generator's active power output; Indicates the minimum value of the generator reactive power output; Indicates the maximum value of the reactive power output of the generator; Indicates the active power output of the wind farm; Indicates the minimum value of the active power output of the wind farm; Indicates the maximum value of the active power output of the wind farm.

[0088] The line transmission capacity constraint is:

[0089]

[0090] Among them, F l represents the power flow of line l; Represents the maximum apparent transmission power of line l.

[0091] The system load shedding constraint is:

[0092]

[0093] in, Indicates the minimum load rate; Represents the amount of load abandonment calculated according to the underlying optimization objective function.

[0094] In the transmission network planning, the N-1 verification method is the most basic and effective verification method to ensure the reliability of the transmission network. The verification idea is to check whether the transmission network can continue to meet the stable operation conditions in the new state after any component in the line fails. If it meets the conditions, it passes the N-1 verification, otherwise it fails. In a specific embodiment, the constraints of the upper-level optimization objective function are constructed according to the N-1 constraints, and the constraints of the upper-level optimization objective function under the N-1 constraints include:

[0095]

[0096]

[0097] Among them, the superscript N-1 represents the operating state quantity of the corresponding parameters of the system under the N-1 constraint condition.

[0098] It represents the abandoned wind volume calculated according to the lower model; Indicates the minimum power output ratio; Represents the active power output by the wind farm at node j.

[0099] In multi-agent reinforcement learning, a multi-agent reinforcement learning method is used to configure independent agents for each layer of optimization objectives to search for solutions and obtain solutions that meet the planning model through inter-layer constraints; among them, the corresponding reward functions are set according to the optimization objective function structures of the upper and lower layers.

[0100] Specifically, multi-agent reinforcement learning is an algorithm structure improvement for optimizing each layer of a two-layer or multi-layer model. It configures independent agents for each layer to search for solutions and obtain solutions that meet the planning model through inter-layer constraints. Setting up multiple agents for the two-layer model to split the search tasks of different attribute objective functions can improve the accuracy of the judgment of the construction value of each line, thereby further improving the performance of the transmission network expansion planning scheme. Due to the different search goals of the dual agents, the corresponding reward function needs to be set according to the optimization objective function structure of the upper and lower layers.

[0101] In some embodiments, the reward function R of the upper optimization objective function structure is set according to the comprehensive economic indicators of the transmission network. upper :

[0102] R upper =V base,upper -f upper ([C T (s t )] T )

[0103] Among them, V base,upperThe reward function baseline represents the structure of the upper-level optimization objective function; C T (s t ) represents the comprehensive economic cost of the transmission network; s t represents the environmental state at time t, s t ∈S t , S t is the set of environmental states;

[0104] The reward function R of the lower optimization objective function structure is set according to the renewable energy consumption level and reliability of the transmission network. lower :

[0105]

[0106] Among them, V base,lower A reward function baseline that represents the structure of the underlying optimization objective function.

[0107] In some embodiments, the multi-agent reinforcement learning method is used to configure independent agents for each layer of optimization objectives to search for solutions and obtain solutions that meet the planning model through inter-layer constraints, including the following steps S21-S24.

[0108] S21. The process of searching for solutions with the optimization objective function of each layer as a reward is defined as a layer of intelligent agents, that is, the upper-layer optimization objective function corresponds to the upper-layer intelligent agent, and the lower-layer optimization objective function corresponds to the lower-layer intelligent agent.

[0109] S22. It is stipulated that in the process of searching for a solution, the upper-level agent needs to store three actions with the highest potential value each time and form a Pareto candidate set. It can be understood that "action" refers to an operation performed in the power grid expansion planning solution, such as adding a new transmission line, rebuilding a transmission line, etc.

[0110] In a specific embodiment, the upper-layer agent uses an ε-greedy strategy to search for three routes with the greatest potential construction value to form a Pareto solution set S pareto {A t,upper Specifically, the selection of each route follows the ε-greedy strategy, which introduces an ε threshold on the original greedy strategy, and can select actions randomly instead of along the direction of maximum benefit with a certain probability. Pareto solution set S pareto {A t,upper The composition rules of} include:

[0111]

[0112] Among them, Prob represents the random probability of the ε-greedy strategy, ε represents the exploration rate, ω eval,upperrepresents the process parameter in Qeval,upper value function, A max represents the best action, S t Indicates the environmental state, A t represents the selected action obtained by random selection, A t,upper Represents the selected action determined in the Pareto solution set.

[0113] S23, using the Pareto candidate set as the optimization range of the lower-level intelligent agent, and using the lower-level optimization objective function to judge the action value, and determining the selection action within the Pareto candidate set by the lower-level intelligent agent.

[0114] Specifically, similar to the upper layer, the lower layer agent still follows the ε-greedy strategy to search, and uses the lower layer optimization objective function to judge the action value to determine the selected action. The lower layer agent uses the Pareto candidate set as the optimization range. Select action A tlower The composition rules include:

[0115]

[0116] A t,lower ∈S pareto {A t,upper}

[0117] Among them, ω eval,lower Represents the process parameter in the Qeval,lower value function.

[0118] S24. When each determined selected action is executed, Qeval,upper and Qeval,lower are updated, and the parameters are copied to Qtarget,upper and Qtarget,lower according to a fixed period.

[0119] Among them, Qeval and Qtarget are value functions with the same initial parameters, Qeval,upper represents the function used by the upper-level agent to select the optimal value action, Qeval,lower represents the function used by the lower-level agent to select the optimal value action, Qtarget,upper represents the function used by the upper-level agent to predict the value of the selected optimal action, and Qtarget,lower represents the function used by the lower-level agent to predict the value of the selected optimal action.

[0120] In some embodiments, a multi-agent reinforcement learning method is trained based on a convolutional neural network.

[0121] In a specific embodiment, Figure 1 As shown, the multi-agent reinforcement learning method is trained based on a deep convolutional network, including the following steps S101-S107.

[0122] S101, initializing the transmission network expansion planning environment and initializing the training parameters, wherein initializing the transmission network expansion planning environment includes setting the initial state S0 corresponding to the initial grid structure to the current environment state S t , the reward corresponding to the initial grid structure is R; initializing the training parameters includes setting the maximum number of training episodes T epis and the parameter replication cycle is T step .

[0123] S102, based on the current environment state S t Multi-agent reinforcement learning is performed with agent parameters, wherein two groups of DDQN agents with the same parameters are generated, namely, the upper agent and the lower agent; the upper agent searches to form a Pareto candidate set, and the lower agent determines the selected action A in the Pareto candidate set t,lower .

[0124] The upper-layer agent performs neural network training according to the first data set, which includes obtaining t Next, confirm the selected action A t,lower The next environmental state S formed after t+1 The corresponding upper layer reward function result R t+1,upper During the process, the parameter S generated t , A t,upper , S t+1 and R t+1,upper The lower-level agent performs neural network training based on the second data set, which includes obtaining t Next, confirm the selected action A t,lower The next environmental state S formed after t+1 The corresponding lower layer reward function result R t+1,lower During the process, the parameter S generated t , A t,lower , S t+1 and R t+1,lower .

[0125] S103, execute action A t And update the state of the reinforcement learning environment, that is, update the grid structure according to the selected action determined in step S102 to form S t+1 , respectively calculate the reward function R of the upper and lower layer agents t+1,upper and R t+1,lower , forming trajectory information and saving it; the trajectory information records the data required for neural network training in S102; specifically, the trajectory information is saved in a RAM memory, and when training is performed in step S102, the data in the RAM can be directly read as a training data set.

[0126] S104: Determine whether the parameter update interval has been reached, that is, determine whether t step |T step =0, parameter copy cycle T step Refers to whether the model parameters need to be copied or updated to maintain consistency during the training process; if t step |T step =0, the convolutional neural network parameters are updated, otherwise, step S105 is performed.

[0127] S105, determine the grid value V (S) updated according to the selected action t+1 ) is greater than the grid value V before update base If it is greater than, proceed to the next step, otherwise the updated environment state S t+1 Set to the current environment state S t , and return to step S102 to search again.

[0128] S106, determine whether the maximum number of training scenes has been reached, that is, determine the current number of training scenes t epis Is it greater than the maximum number of training scenes T? epis If it is reached, proceed to step S107; if it is not reached, initialize the environment state and search again.

[0129] S107. Output the final transmission network expansion plan.

[0130] In a specific embodiment, the convolutional neural network can adopt a deep residual network, and a deep residual network is adopted to improve the performance of the multi-agent DDQN algorithm agents.

[0131] In order to better demonstrate the performance of the adaptive grid optimization method proposed in the present invention, a specific embodiment is described by taking the PX area power grid planning scheme as an example. It can be understood that the power grid planning scheme is not an actual power grid planning, but a virtual power grid architecture provided to assist technical personnel in the field to understand.

[0132] With the large-scale development and transmission of hydropower in the PX region, the PX power grid will add 4 ±800kV UHV DC lines, and the scale of UHV DC transmission in the PX region will increase to about 34.2 million kilowatts by 2025. At the same time, in order to ensure the safe and stable operation of multiple UHV DC lines, the PX AC power grid will build a new BT~SF double-circuit 500kV line and YCII switch station in 2023, and gradually implement the PX optimization and transformation project, and finally achieve the purpose of grouping four UHV DC lines, YZ DC, YC DC, BZ DC, and BS DC. DQN and PSO are used as comparison algorithms, and the parameter comparison of multi-agent DDQN and DQN is shown in Table 1. The three power grid expansion planning schemes are shown in Tables 2-4. Among them, DQN and PSO are both common deep learning methods, DQN is a deep Q network, PSO is a particle swarm optimization algorithm, and in this embodiment, DDQN is used to represent the multi-agent reinforcement learning method proposed in this disclosure.

[0133] Table 1 Multi-agent DDQN and DQN algorithm parameters

[0134]

[0135] This disclosure further expands the application scenario to a more complex PX power grid system to further evaluate the performance of the proposed method. We set the loads of the nodes 3, 4, 8, 16, 20, 24, 27, and 29 of the power grid system as variable loads, and set the lines 30, 32, 33, 34, and 38 as wind farm nodes. The load capacity is increased to 1.10 times, the capacity of conventional generator sets is increased to 1.15 times, and the capacity of wind farms is increased to 1.25 times. The planning results of the multi-agent DDQN, DQN, and PSO methods are shown in Tables 2 to 4.

[0136] Table 2 PX power grid structure multi-agent DDQN transmission network expansion planning scheme

[0137]

[0138] Table 3 PX power grid structure multi-agent DDQN transmission network expansion planning scheme

[0139]

[0140] Table 4 PX power grid structure PSO transmission network expansion planning scheme

[0141]

[0142]

[0143] Tables 2 to 4 show that all three methods optimize the stability and economy of the transmission network structure by constructing lines. The scheme based on the deep reinforcement learning method takes into account the difference in benefits brought by the construction of lines in order in the benefit structure, so the transmission network expansion planning scheme based on this method contains the order information of line construction, but the PSO method only optimizes the structure of the complete transmission network structure. The comprehensive costs of the three methods are 23.10 million, 23.47 million, and 21.64 million respectively. Among the three schemes, the scheme obtained by the PSO method has the lowest construction cost. The scheme obtained by DQN has the highest construction cost, and the scheme cost obtained by the multi-agent DDQN is between the other two schemes. In addition, the three schemes all affect the system structure and power flow distribution through the construction of new lines, and the system network loss is reduced to a certain extent.

[0144] The K-order electrical betweenness entropy average can calculate the uniformity of the power flow distribution of the adjacent lines of the wind farm to evaluate the system reliability. The solution obtained by the multi-agent DDQN increases the improved K-order electrical betweenness entropy average from 110.25MW to 191.50MW, which has the highest optimization range compared with the other three solutions, reflecting that it significantly reduces the probability of cascading failures. In addition, the new power system needs to ensure as little wind abandonment and load reduction as possible under high uncertainty to improve the renewable energy absorption capacity and power supply reliability. The transmission network expansion planning task can improve the power support capability by building a tighter transmission network structure for new line construction. For the modified IEEERTS24-BUS system (a benchmark system for power market and power system operation research) proposed in this disclosure, its notable feature is that the load shedding is relatively serious. The multi-agent DDQN solution and the DQN method both reduce the load shedding of the transmission network as much as possible, but the PSO solution does not achieve good performance in this direction. At the same time, the multi-agent DDQN solution performs best in reducing wind abandonment. Therefore, the performance of the three schemes is comprehensively analyzed. Although the PSO scheme requires the lowest cost, it ignores the optimization of the K-order electrical betweenness entropy average and load shedding. This is because the multi-objective transmission network expansion is a non-deterministic polynomial (NP) problem with a polynomial complexity. The method based on heuristic learning is prone to fall into the local optimal problem in the process of solving the transmission network expansion planning task, resulting in the optimization only in the direction of improving certain indicators. The method based on deep reinforcement learning can jump out of the optimal line selection determined by the neural network with a certain probability due to the ε-greedy strategy, thereby avoiding the influence of the local optimal problem to a certain extent, thereby coordinating the optimization of each model indicator. Although the multi-agent DDQN only performs slightly better than the DQN scheme in terms of wind abandonment and load abandonment, the multi-agent DDQN has a more significant improvement in the K-order electrical betweenness entropy average. This confirms that the dual DDQN agent structure improves the optimization capability of the transmission network structure at each layer, and its judgment on the value of each line construction is more accurate, which helps to obtain a better transmission network expansion planning scheme in the end.

[0145] Figure 2 and Figure 3 The comparison of the comprehensive economic costs of the multi-agent DDQN scheme and the DQN scheme shows the changes in the comprehensive costs of the schemes obtained by the multi-agent DDQN and DQN. The construction cost growth of the first three routes in the DQN scheme is relatively faster than that of the first three routes in the multi-agent DDQN scheme, which requires higher start-up capital at the beginning of the project construction. At the same time, according to market price fluctuations and technological progress, the actual investment required for the later stage of the plan may save more, so that the capital investment required for the implementation of the multi-agent DDQN scheme is more saved than that of the DQN scheme.

[0146] Figure 4 and Figure 5 The line flow distribution of the multi-agent DDQN scheme and the DQN scheme is shown. The initial power grid flow contains 2 lines above 400MW and 3 lines above 300MW, which indicates that the initial flow uniformity is poor. Both the multi-agent DDQN scheme and the DQN scheme transfer part of the flow to the underloaded line, which improves the utilization rate of the underloaded line and reduces the probability of cascading failure caused by the overloaded line. Further comparison of the flow distribution of the two shows that for lines with a flow greater than 200MW, the multi-agent DDQN scheme controls the lines greater than 250MW to 3, while the DQN scheme has 7 lines greater than 250MW. For lines with power less than 200MW, the multi-agent DDQN and DQN methods have almost the same performance in optimizing the line flow distribution in the range of 150 to 200MW. For lines below 100MW, the multi-agent DDQN scheme better increases the flow of the underloaded line. Therefore, the multi-agent DDQN scheme has higher system stability and higher line utilization.

[0147] Compared with the DQN scheme, the multi-agent DDQN scheme achieves better system flow distribution and lower wind abandonment and load shedding under uncertainty conditions at a lower economic comprehensive cost, and the system state transition during the construction process is smoother. This verifies that the dual-agent structure of the multi-agent DDQN scheme can achieve hierarchical prediction of line construction value. One group is used to search for lines with higher economic efficiency, and the other group is used to search for lines that can improve renewable energy consumption capacity and system reliability. This structure improves the accuracy of line construction value prediction and helps to form a better transmission network expansion planning scheme.

[0148] Figure 6 , Figure 7 and Figure 8The indicators of about 8000 network structures constructed by the upper and lower agents in the multi-agent DDQN in 200 trainings (comprehensive economic cost, total wind curtailment and wind load curtailment, and K-order electrical betweenness entropy balance) are shown. Before 1000 steps, the comprehensive economic cost indicator is concentrated in the poor performance area, and the lower indicators are distributed more evenly without obvious aggregation. This is because the multi-agent DDQN currently does not have enough data for neural network training, and there is a large error between its value prediction and the actual environmental benefits. Between 1000 and 2500 steps, the indicators of the planning scheme gradually improve. The neural network has achieved sufficient training, and the prediction error of the line construction value has gradually decreased. After 2500 steps, the indicators are evenly distributed between the best and the worst. This is because the multi-agent DDQN adopts the ε-greedy strategy, which allows the agent to randomly select line construction without selecting the prediction result of the neural network with a certain probability. This training mechanism can prevent the local optimal problem and obtain a more accurate value function.

[0149] Fig. 9 The figure shows the change in the sum of line value predictions during the training process of multi-agent DDQN and DQN. The line construction value represents a nonlinear estimate of the improvement of system indicators after the construction of each line by the agent using a neural network. The figure shows that the value prediction of the multi-agent DDQN is significantly lower than that of the DQN method. This is because the multi-agent DDQN method uses two neural network structures to form Qeval and Qtarget, which makes the selection of the optimal line independent of the value prediction of the optimal line. This structure avoids the impact of unexpected overestimation to a certain extent and improves the accuracy of line value prediction, which helps to solve the task of transmission network expansion planning.

[0150] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0151] Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive grid optimization method for improving large-scale hydropower transmission capacity, characterized in that: include: Considering the transmission capacity of hydropower stations, a two-layer transmission network optimization objective function is constructed; the upper optimization objective function is used to evaluate the economic feasibility of the transmission network expansion planning scheme, and the lower optimization objective function is used to evaluate the renewable energy consumption level and system reliability of the transmission network; A multi-agent reinforcement learning method is used to configure independent agents for each layer of optimization objectives to search for solutions and obtain solutions that meet the planning model through inter-layer constraints; among them, the corresponding reward functions are set according to the optimization objective function structures of the upper and lower layers.

2. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 1 is characterized in that: The objective function of the double-layer transmission network optimization is: Among them, F(X) represents the objective function of the double-layer transmission network optimization, f upper represents the upper optimization objective function, f lower represents the lower optimization objective function, C1(X) represents the transmission network construction cost, C2(X) represents the network loss cost, C3(X) represents the maintenance cost, P curt represents the reduction in hydroelectric power generation, P shed Indicates the load shedding amount, W J It indicates the electrical betweenness of the lines adjacent to the hydropower station.

3. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 2 is characterized in that: The constraints of the upper optimization objective function include power flow constraints and equipment operation constraints; the power flow constraints include balance constraints constructed according to the AC power flow; the equipment operation constraints include voltage amplitude and phase angle constraints, generator output constraints, line transmission capacity constraints and system load shedding constraints.

4. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 3 is characterized in that: The constraint conditions of the upper optimization objective function are constructed according to the N-1 constraint conditions. Then, the constraint conditions of the upper optimization objective function under the N-1 constraint conditions include: Among them, P j g represents the rated active power of the generator set at node j; represents the wind power active power discarded by node j; V j represents the voltage at node j; B jk represents the susceptance between node j and node k; G jk represents the conductivity between node j and node k; θ jk represents the phase angle between node j and node k; The load at node j absorbs active power; represents the active power of the removed load at node j; It represents the abandoned wind volume calculated according to the lower model; Indicates the minimum power output ratio; represents the active power output by the wind farm at node j; represents the reactive power output by the generator at node j; represents the reactive power input of node j; represents the reactive power of wind power discarded by node j; represents the reactive power absorbed by the load at node j; represents the reactive power of the reactive compensation device at node j; represents the minimum value of the voltage at node j; represents the maximum value of the voltage at node j; Represents the minimum value of the phase angle of node j; Indicates the maximum value of the phase angle of node j; Indicates the minimum value of the generator reactive power output; Indicates the maximum value of the reactive power output of the generator; F l represents the power flow of line l; represents the maximum transmission apparent power of line l; Indicates the minimum load rate; represents the amount of load abandonment calculated according to the lower optimization objective function; The superscript N-1 indicates the corresponding operating state quantity of the system under the N-1 constraint condition.

5. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 4 is characterized in that: The reward function R of the upper optimization objective function structure is set according to the comprehensive economic indicators of the transmission network. upper : R upper =V base,upper -f upper ([C T (s t )] T ) Among them, V base,upper The reward function baseline represents the structure of the upper-level optimization objective function; C T (s t ) represents the comprehensive economic cost of the transmission network; s t represents the environmental state at time t, s t ∈S t , S t is the set of environmental states; The reward function R of the lower optimization objective function structure is set according to the renewable energy consumption level and reliability of the transmission network. lower : Among them, V base,lower A reward function baseline that represents the structure of the underlying optimization objective function.

6. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 1, characterized in that: The multi-agent reinforcement learning method is used to configure an independent agent for each layer of optimization objectives to search for solutions and obtain solutions that meet the planning model through inter-layer constraints, including: The process of searching for solutions with each layer's optimization objective function as a reward is defined as a layer of agents, that is, the upper layer's optimization objective function corresponds to the upper layer's agent, and the lower layer's optimization objective function corresponds to the lower layer's agent; It is stipulated that during the solution search process, the upper-level agent needs to store three actions with the highest potential value each time and form a Pareto candidate set; The Pareto candidate set is used as the optimization range of the lower-level intelligent agent, and the lower-level optimization objective function is used to judge the action value, and the lower-level intelligent agent determines the selection action within the Pareto candidate set; When each action that determines the selection is executed, Qeval,upper and Qeval,lower are updated, and the parameters are copied to Qtarget,upper and Qtarget,lower according to a fixed period; Among them, Qeval and Qtarget are value functions with the same initial parameters, Qeval,upper represents the function used by the upper-level agent to select the optimal value action, Qeval,lower represents the function used by the lower-level agent to select the optimal value action, Qtarget,upper represents the function used by the upper-level agent to predict the value of the selected optimal action, and Qtarget,lower represents the function used by the lower-level agent to predict the value of the selected optimal action.

7. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 6, characterized in that: The upper-level agent uses the ε-greedy strategy to search for three routes with the greatest potential construction value to form a Pareto solution set S pareto {A t,upper }, Pareto solution set S pareto {A t,upper The composition rules of} include: Among them, Prob represents the random probability of the ε-greedy strategy, ε represents the exploration rate, ω eval,upper represents the process parameter in Qeval,upper value function, A max represents the best action, S t Indicates the environmental state, A t represents the selected action obtained by random selection, A t,upper Represents the selected action determined in the Pareto solution set.

8. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 7, characterized in that: The lower-level agent uses an ε-greedy strategy to search in the Pareto candidate set to determine the selected action A t,lower , select action A t,lower The composition rules include: A t,lower ∈S pareto {A t,upper } Among them, ω eval,lower Represents the process parameter in the Qeval,lower value function.

9. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 6, characterized in that: Training multi-agent reinforcement learning methods based on convolutional neural networks.

10. The adaptive grid optimization method for improving large-scale hydropower transmission capacity according to claim 9, characterized in that: The method for training a multi-agent reinforcement learning method based on a deep convolutional network includes: Initialize the transmission network expansion planning environment and initialize training parameters Multi-agent reinforcement learning is performed based on the current environment state and agent parameters. The upper-layer agent searches to form a Pareto candidate set, and the lower-layer agent determines the selected action within the Pareto candidate set. Execute actions and update the state of the reinforcement learning environment, calculate the reward functions of the upper and lower layer agents, form trajectory information and save it; Determine whether the parameter update interval has been reached. If so, update the convolutional neural network parameters. If not, proceed to the next step. Determine whether the grid value after updating according to the selected action is greater than the grid value before updating, if so, proceed to the next step, otherwise, search again; Determine whether the maximum number of training scenes has been reached. If so, proceed to the next step. If not, initialize the environment state and search again. Output the final transmission network expansion planning scheme.

Images