Power grid optimization and reconstruction method based on double-agent cooperation

Abstract

The invention discloses a power grid optimization and reconstruction method based on double-agent cooperation, and the method optimizes the control of discrete agents and continuous agents in a power grid, and achieves the cooperation of the discrete agents and the continuous agents through a separation switch state and SOP power output optimization task. The power grid optimization reconstruction method comprises the following steps: constructing a power grid dynamic reconstruction optimization model; state initialization, discrete agent decision making, continuous agent decision making, reward calculation, priority experience playback, target network update iteration and convergence judgment; the optimization target priority is dynamically adjusted, the system state change can be rapidly adapted, a more effective global optimization scheme is provided in a dynamic and multi-period environment, and reliable technical support is provided for operation safety and stability of a new energy power system.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent dispatching decision analysis of electric power systems, and in particular to a power grid optimization reconstruction method based on dual-intelligent-agent collaboration. Background Art

[0002] As the global energy structure transforms towards low-carbon and sustainable development, the high proportion of clean energy (such as wind power and photovoltaic power) access has greatly increased the complexity and dynamics of distribution network operation. The randomness of clean energy output and the volatility of load demand make the system urgently need an optimization method that can dynamically adapt to system state changes while ensuring operational stability, so as to effectively reduce energy loss in complex environments and give priority to the consumption of clean energy.

[0003] The application of smart soft switches (Soft Open Point, SOP) provides a new technical means for distribution network optimization. SOP can independently adjust active and reactive power, thereby enhancing the ability to absorb clean energy while meeting voltage stability, bringing possibilities for distribution network optimization in a high-penetration clean energy environment. However, the deployment cost of SOP is high, and it is difficult to replace traditional switches in the short term. Therefore, the synergy of SOP and traditional switches needs to be comprehensively considered when designing the optimization model. In this context, the distribution network reconstruction problem is transformed into a multi-period, multi-objective mixed integer nonlinear programming problem (MINLP), which requires coordinated control of the switch state (discrete variable), i.e., discrete agent, and SOP power output (continuous variable), i.e., continuous agent, in different time periods to achieve improved system global performance.

[0004] When existing distribution networks attempt to introduce reinforcement learning to solve dynamic optimization problems, they usually adopt a single-agent architecture. Single agents perform well in optimizing a single objective, but it is difficult to achieve efficient global coordination when dealing with complex distribution network reconstruction tasks with multiple objectives and multi-period requirements. To this end, multi-agent architectures are gradually applied to distribution network optimization and reconstruction, and multiple agents collaborate to complete global optimization through task decomposition and information sharing. However, this type of multi-agent method still has the following defects in practical applications: lack of coordination mechanism between agents, agents often only focus on their own tasks, and it is difficult to achieve overall optimal control in a global scope; insufficient priority for clean energy consumption, existing methods have limited design of priority consumption of clean energy in optimization strategies, and fail to fully tap its potential for carbon emission reduction; distribution network optimization efficiency and convergence speed need to be improved. With the increasing complexity of the power grid environment, existing multi-agent methods are insufficient in convergence speed and strategy optimization, and it is difficult to quickly respond to changes caused by load fluctuations and clean energy uncertainties. Summary of the invention

[0005] The purpose of the present invention is to provide a power grid optimization and reconstruction method based on dual-agent collaboration, so as to solve the problems proposed in the above background technology that even if the existing power grid optimization and reconstruction adopts a multi-agent architecture, there is still a lack of coordination mechanism between agents, insufficient priority of clean energy consumption, and the optimization efficiency and convergence speed of the distribution network need to be improved. In view of the limitations of the existing distribution network optimization and reconstruction methods, the present invention proposes a power grid optimization and reconstruction strategy based on dual-agent collaborative intelligent optimization to solve the problems that multi-objective optimization in the operation of distribution networks in dynamic environments is difficult to meet the real-time response requirements under dynamic load conditions, lacks an effective agent coordination mechanism and an efficient learning strategy, making it difficult for the overall system to obtain the optimal solution.

[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a power grid optimization and reconstruction method based on dual-agent collaboration, which optimizes the control of discrete agents and continuous agents in the power grid, realizes the collaboration of discrete agents and continuous agents by separating the switch state and SOP power output optimization tasks, and designs the discrete agent and continuous agent collaborative reward function in the power grid. The optimization goal of the discrete agent is to minimize the system energy loss and switch operation frequency of the power grid; the optimization goal of the continuous agent is to maintain voltage stability and maximize clean energy consumption in the power grid; the power grid optimization and reconstruction method includes the following steps:

[0007] S1. Construct a power grid dynamic reconstruction optimization model including an objective function and constraint conditions; construct a dual-agent architecture based on the constraints of the power grid dynamic reconstruction optimization model, wherein the dual-agent architecture includes a discrete agent and a continuous agent;

[0008] S2, initializing the state of the power grid dynamic reconstruction optimization model and the parameters of the intelligent agent;

[0009] S3, giving discrete agent decision and continuous agent decision through dual agent architecture in power grid dynamic reconstruction optimization model;

[0010] S4, calculate the instant rewards and collaborative rewards of discrete agents and continuous agents, and dynamically adjust the weights based on the current state;

[0011] S5. Optimizing the power grid dynamic reconstruction optimization model by adopting the priority experience playback strategy;

[0012] S6. Update the target network: regularly copy the parameters of the Q network to the target network to ensure the stability of the algorithm;

[0013] S7. Repeat steps S2-S6 until the strategy converges, and output the optimized power grid dynamic reconstruction optimization model topology structure and SOP power regulation strategy to achieve the optimized reconstruction of the power grid.

[0014] Preferably, the objective function of the dynamic reconstruction optimization model of the power grid in S1 includes the energy loss minimization target, the clean energy consumption maximization target, the switch operation number minimization target and the voltage deviation minimization target; energy loss minimization target: reduce the system's power loss by the cumulative active power loss value of the system in each time period, thereby improving the system's operating economy; set the clean energy consumption target: by giving priority to distributed clean energy such as wind power and photovoltaics, indirectly support carbon emission reduction strategies, reduce dependence on fossil fuels, and improve the environmental benefits of the system; switch operation number minimization target: constrain the operation frequency of switching equipment to reduce equipment wear; voltage deviation minimization target: in multi-time period optimization, suppress voltage fluctuations at each node to improve system stability.

[0015] Preferably, the constraints of the power grid dynamic reconstruction optimization model in S1 include radial constraints, node voltage stability constraints, power balance constraints, SOP operation constraints, distributed energy output constraints, branch flow constraints and switch operation frequency constraints.

[0016] Preferably, the constructed dual-agent architecture includes designing the state space, action space, and immediate reward function of the discrete agent and the continuous agent respectively; designing a collaborative reward function of the discrete agent and the continuous agent based on the coupling optimization of the two agents; and designing a comprehensive reward function using a dynamic weight mechanism and the collaborative reward function of the discrete agent and the continuous agent.

[0017] Preferably, the synergy reward function includes energy loss synergy reward, clean energy consumption synergy reward and comprehensive synergy reward.

[0018] Preferably, in S4, the immediate rewards of the discrete agent and the continuous agent are calculated respectively by the immediate reward function, and the collaborative rewards of the discrete agent and the continuous agent are calculated respectively by the collaborative reward function.

[0019] Preferably, the dynamic weight mechanism adopted in S4 for dynamically adjusting weights based on the current state is used to adjust the weights of different optimization objectives in the system in real time, thereby adapting to changes in the power grid state and ensuring reasonable priority allocation of various optimization objectives at different stages.

[0020] Preferably, the priority experience replay strategy in S5 adopts the priority experience replay mechanism introduced on the traditional D3QN framework to improve the learning efficiency of the model in complex power grids.

[0021] Preferably, in S5, the priority experience playback mechanism sorts the samples based on the time difference error, and assigns the sample sampling priority according to the error value of the time difference error.

[0022] Preferably, in S1, initializing the state of the power grid dynamic reconstruction optimization model includes initializing the distribution system voltage, load, distributed energy output, and SOP initial power; initializing the intelligent agent parameters includes the Q network, the target network, and the experience pool.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention adopts a data-driven reinforcement learning framework, and combines a dynamic weight mechanism and a priority experience playback strategy to construct an optimization reconstruction method that adapts to dynamic loads and clean energy output fluctuations. This method separates the switch state and SOP power output optimization tasks to achieve the collaborative work of discrete agents and continuous agents, thereby overcoming the bottleneck problem of lack of collaboration of agents in existing multi-agent methods. In addition, the present invention can quickly adapt to system state changes by dynamically adjusting the optimization target priority, providing a more effective global optimization solution in a dynamic, multi-period environment, and providing reliable technical support for the operational safety and stability of new energy power systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flow chart of a power grid optimization and reconstruction method based on dual-agent collaboration provided by the present invention;

[0026] Figure 2 This is a training framework diagram of the collaborative intelligent optimization reconstruction strategy based on reinforcement learning in the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] like Figure 1 As shown, the present invention provides a power grid optimization reconstruction method based on dual-agent collaboration, and the power grid optimization reconstruction method includes the following steps:

[0029] S1. Construct a power grid dynamic reconstruction optimization model including an objective function and constraint conditions; construct a dual-agent architecture based on the constraints of the power grid dynamic reconstruction optimization model, wherein the dual-agent architecture includes a discrete agent and a continuous agent;

[0030] (1) Objective function

[0031] The objective functions of the dynamic reconstruction optimization model of the power grid in S1 include the energy loss minimization target, the clean energy consumption maximization target, the switch operation number minimization target and the voltage deviation minimization target; the energy loss minimization target: reduce the system's power loss by the cumulative active power loss value of the system in each time period, thereby improving the system's operating economy; set the clean energy consumption target: by giving priority to distributed clean energy such as wind power and photovoltaics, indirectly support carbon emission reduction strategies, reduce dependence on fossil fuels, and improve the environmental benefits of the system; the switch operation number minimization target: constrain the operation frequency of the switch equipment to reduce equipment wear; the voltage deviation minimization target: in multi-time period optimization, suppress the voltage fluctuation of each node and improve the stability of the system.

[0032] 1) Minimize energy loss:

[0033] The goal of minimizing energy loss is to reduce the power loss of the system by calculating the accumulated active power loss of the system in each period, thereby improving the operating economy of the system. It is defined as:

[0034]

[0035] In the formula, R loss is the total energy consumption of the system operation; T is the total number of time periods in the optimization cycle; n is the number of nodes, P i (t) is the active power loss of node i during period t.

[0036] 2) Maximizing clean energy consumption:

[0037] Set clean energy consumption targets, and by giving priority to distributed clean energy such as wind power and photovoltaics, indirectly support carbon emission reduction strategies, reduce dependence on fossil fuels, and improve the environmental benefits of the system. The total amount of clean energy consumption is defined as:

[0038]

[0039] In the formula, R renewable is the clean energy consumption; N DER is the set of distributed energy nodes; P DER,i (t) is the active power of the distributed generation at node i in period t.

[0040] 3) Minimize the number of switching operations: limit the operation frequency of the switch equipment to reduce equipment wear.

[0041]

[0042] In the formula, R switch is the number of switch operations; N sw is a switchgear assembly; C i(t) is the on / off state of switch i in period t (0 means closed, 1 means open).

[0043] 4) Minimize voltage deviation: In multi-period optimization, suppress voltage fluctuations at each node and improve system stability.

[0044]

[0045] In the formula, R voltage is the node voltage deviation; N is the node set; V i (t) is the voltage amplitude of node i in time period t; V ref is the reference value of the node voltage.

[0046] (2) Constraints

[0047] The constraints of the dynamic reconstruction optimization model of the power grid in S1 include radial constraints, node voltage stability constraints, power balance constraints, SOP operation constraints, distributed energy output constraints, branch flow constraints and switch operation frequency constraints.

[0048] 1) Radial constraint: The distribution network must maintain a radial structure after reconstruction to avoid loops and ensure stable operation of the system. This constraint ensures the physical feasibility of the system topology.

[0049]

[0050] Where T(t) represents the network topology used in period t, G rad represents the set of network topologies that satisfy radial constraints, It represents the set of topologies that meet the radial requirements when connected to SOP. This set includes structures that allow branches containing SOPs to form local rings within the power regulation range.

[0051] 2) Node voltage stability constraint: The voltage at each node of the distribution network must fluctuate within a specified range to ensure voltage stability.

[0052] V imin ≤V i (t)≤V imax

[0053] Where V imax and V imin are the upper and lower limits of the voltage amplitude at node i.

[0054] 3) Power balance constraint: The sum of active and reactive power of each node must satisfy the power flow equation to maintain the power balance of the system:

[0055] Active power constraint:

[0056]

[0057] Reactive power constraint:

[0058]

[0059] Where P i gen (t), are respectively the active power and reactive power of the traditional power supply of node i in period t; P i DER (t), are the active power and reactive power output of the distributed power source at node i in period t; P i sop (t), are the active power and reactive power of SOP in period t respectively; P i load (t), are respectively the active power and reactive power consumed by the load of node i in period t; V i (t), V j (t) are the voltage amplitudes of nodes i and j in time period t; θ ij (t) is the phase angle difference between nodes i and j in period t; G ij , B ij are the conductance and susceptance between nodes i and j respectively.

[0060] 4) Operation constraints of SOP: To prevent SOP from operating under overload conditions, its active and reactive outputs must be constrained. Assume that where represents the active and reactive outputs of SOP respectively.

[0061] Active power constraints of SOP:

[0062]

[0063] Reactive power constraints of SOP:

[0064]

[0065] Capacity constraints of SOP:

[0066]

[0067] In the formula, are the active power and reactive power of the two converters in the kth SOP of period t, respectively; are the upper and lower limits of reactive power of the two converters of the kth SOP respectively; are the access capacities of the two converters of the kth SOP respectively.

[0068] 5) Distributed energy output constraints:

[0069] The minimum output of wind power and photovoltaic power is set to 0, and the maximum value is limited by the temporal changes in the intensity of wind energy and solar energy at different times, and both operate at a constant power factor.

[0070]

[0071] In the formula, are the minimum and maximum active powers of the distributed generation connected to node i, respectively; are the minimum and maximum reactive power of the distributed generation connected to node i, respectively.

[0072] 6) Branch flow constraints:

[0073] Limit the current on the branch to ensure the safety of the distribution network and the physical carrying capacity of the equipment.

[0074]

[0075] In the formula, I ij (t) is the current amplitude between branches i and j at time t; G ij , B ij are the conductance and susceptance between branches i and j respectively; θ ij (t) is the phase angle difference between nodes i and j at time t; V i (t) and V j (t) is the voltage amplitude of nodes i and j at time t; I ij,max are the maximum current amplitudes allowed between branches i and j.

[0076] 7) Switch operation frequency constraint, which limits the switch operation frequency within a certain period of time to reduce switch wear and extend equipment life. This constraint can be expressed as:

[0077]

[0078] Where O is the total number of switch operations involved in the reconstruction during the optimization period, C n (t) is the open state of switch n during period t, O total It is the upper limit of the total number of all switch operations.

[0079] Limitation of the number of operations for a single switch:

[0080]

[0081] In the formula, O n represents the total number of operations of switch n during the optimization cycle, O n,max This is the upper limit of the number of times a single switch can be operated. Avoid frequent operation of a specific switch.

[0082] Collaborative intelligent optimization and reconstruction strategy based on reinforcement learning Figure 2 As shown, a dual-agent architecture based on the constraints of the dynamic reconstruction optimization model of the power grid is constructed, and the dual-agent architecture includes a discrete agent and a continuous agent; the dual-agent architecture includes a discrete agent and a continuous agent; the constructed dual-agent architecture includes designing the state space, action space, and immediate reward function of the discrete agent and the continuous agent respectively; designing the collaborative reward function of the discrete agent and the continuous agent based on the coupling optimization of the two agents; and designing the comprehensive reward function using the dynamic weight mechanism and the collaborative reward function of the discrete agent and the continuous agent.

[0083] 1. State space design:

[0084] (1) Discrete Agents:

[0085] S sw = {P L,i (t),P DG,i (t),G(t)}

[0086] Where P L,i (t) is the load power during period t, P DG,i (t) is the predicted output value of renewable distributed generation in period t.

[0087] (2) Continuous Agents:

[0088] S sop = {V i ,P ij (t),S SOP,i (t),P DG,i (t)}

[0089] Where V i is the voltage amplitude of node i; P ij (t) is the active power of branch (i, j); S SOP,i is the power output value of SOP i.

[0090] 2. Action space design:

[0091] (1) Discrete Agent: The action space of a discrete agent is defined as a set of switch operation combinations used to control the network topology. Each action represents a state adjustment of a set of switches, defined as follows:

[0092] A sw ={a sw ,i|a sw ,i∈{0,1},i=1,2,...n}

[0093] In the formula, A swis the action set of the discrete agent; a sw ,i is the action of switch i, 0 means keeping the original state, 1 means changing the current state; n is the total number of switches in the system.

[0094] (2) Continuous Agent: The action space of the continuous agent is the active power output adjustment range of the SOP

[0095] A sop = {P sop,i ,Q sop,i |i=1,2,...,n sop}

[0096] In the formula, A sop is the action set of the continuous agent, P sop,i is the active output of SOP i, Q sop,i is the reactive power output of SOP i, n sop is the total number of SOPs in the system.

[0097] 3. Design of immediate reward function: The immediate reward function is used to evaluate the effectiveness of the actions taken by the agent in the current state. Reward functions are designed for discrete and continuous agents respectively.

[0098] (1) Discrete Agents

[0099] The reward of the discrete agent is designed to minimize the system energy loss and switching operation frequency, which is specifically defined as:

[0100]

[0101] In the formula, R loss Energy loss reward; R sw is the switching penalty; ΔP loss is the difference between the current energy loss and the energy loss at the previous moment; k is the penalty coefficient; N sw is the current number of switch operations; R d is the immediate reward of the discrete agent.

[0102] (2) Continuous Agent

[0103] The reward function of the continuous agent aims to maintain voltage stability and maximize clean energy consumption. The specific definition is:

[0104]

[0105] In the formula, R v Reward for voltage stability; R qj Rewards for clean energy consumption; V sj is the actual voltage; V mb is the target voltage; ΔR qjis the clean energy consumption; R c is the immediate reward of the continuous agent.

[0106] 4. Collaborative reward function design.

[0107] A collaborative reward function is designed for discrete agents and continuous agents based on the coupling optimization of the two agents; the collaborative reward function includes energy loss collaborative reward, clean energy consumption collaborative reward and comprehensive collaborative reward.

[0108] In order to achieve the collaborative optimization effect of dual agents, CIORS introduces a collaborative reward mechanism, which subdivides the global goal into sub-goals for each agent and realizes the coupled optimization of the two agents through information sharing. The main components of the reward mechanism are as follows:

[0109] (1) Energy loss synergy reward:

[0110]

[0111] Where γ is the weight coefficient of energy loss synergy reward; and are the total energy loss at the current and previous moments respectively.

[0112] (2) Clean Energy Consumption Collaborative Incentives:

[0113]

[0114] Where δ is the weight coefficient of energy loss synergy reward; and They are the total amount of clean energy consumed at the current and previous moments respectively.

[0115] (3) Comprehensive synergy reward:

[0116] R co =R co_loss +R co_qj

[0117] In the formula, R co It is the total reward of dual agents under multi-objective collaborative optimization.

[0118] 5. Comprehensive reward function design:

[0119] The comprehensive reward function is designed by using a dynamic weight mechanism and a collaborative reward function between discrete agents and continuous agents:

[0120] R t =w d R d +w c R c +w c0 Rco

[0121] (1) Dynamic Weight Mechanism

[0122] In CIORS (Collaborative Intelligent Optimization Reconfiguration Strategy), the dynamic weight mechanism is used to adjust the weights of different optimization objectives in the system in real time, so as to adapt to the changes in the power grid state and ensure the reasonable allocation of priorities of various optimization objectives at different stages. This mechanism is based on the sigmoid nonlinear function and dynamically adapts to the fluctuations of the system state by adjusting the weights of key parameters such as load, voltage, and power loss.

[0123] The weight value is calculated using the sigmoid nonlinear function:

[0124]

[0125] In the formula, w d is the weight of the discrete agent's immediate reward, which affects the switch's operation choice; c is the weight of the continuous agent's immediate reward, which adjusts the output of SOP; co is the weight of the synergy reward;

[0126] S2. Initialize the state of the power grid dynamic reconstruction optimization model and the intelligent agent parameters; the initialization state of the power grid dynamic reconstruction optimization model includes the voltage, load, distributed energy output and SOP initial power of the power grid, and the intelligent agent parameters include the Q network, target network and experience pool.

[0127] S3, giving discrete agent decision and continuous agent decision through dual agent architecture in power grid dynamic reconstruction optimization model;

[0128] Discrete agent decision-making: Based on the current state and ε-greedy strategy, select the switch operation combination, execute and update the system's network topology and power flow, and generate a new state;

[0129] Continuous agent decision making: Based on the updated state, select the active and reactive power output of the SOP, execute and update the voltage and power flow of the system, and generate immediate rewards;

[0130] S4. Calculate the instant rewards and collaborative rewards of discrete agents and continuous agents, and dynamically adjust the weights based on the current state; calculate the instant rewards of discrete and continuous agents including energy loss, voltage stability, and clean energy consumption.

[0131] In S4, the immediate rewards of discrete agents and continuous agents are calculated through immediate reward functions, and the collaborative rewards of discrete agents and continuous agents are calculated through collaborative reward functions.

[0132] The dynamic weight mechanism adopted in S4 to dynamically adjust the weight based on the current state is used to adjust the weights of different optimization objectives in the system in real time, so as to adapt to the changes in the power grid state and ensure the reasonable allocation of priorities of various optimization objectives at different stages.

[0133] S5. Optimizing the power grid dynamic reconstruction optimization model by adopting the priority experience playback strategy;

[0134] The priority experience replay strategy introduces a priority experience replay mechanism on the traditional D3QN framework to improve the learning efficiency of the model in complex power grids. The priority experience replay mechanism sorts samples based on the time difference error and assigns sample sampling priority according to the error value of the time difference error.

[0135] When implementing the optimal strategy in a dynamic environment, the huge action space will slow down the convergence of the algorithm. In order to improve the learning efficiency of the model in complex distribution networks, this paper introduces the Prioritized Experience Replay (PER) mechanism on the traditional D3QN framework. PER-D3QN sorts samples based on the temporal-difference (TD) error and assigns sample sampling priority according to the size of the TD error value. D3QN enables the agent to more accurately evaluate the relative value of each action by introducing a dual valuation structure and an advantage function.

[0136] In PER-D3QN, the TD error is defined as: δ i =|Q target,i -Q current,i Among them, samples with larger TD errors have higher priority and are more likely to be sampled during training, which improves the utilization rate of key samples during training and avoids suboptimal convergence problems that may be caused by random sampling. This mechanism ensures that PER-D3QN can more efficiently utilize important data samples in the experience pool and accelerate the convergence speed of the strategy. The algorithm flow is as follows:

[0137] (1) Action value function update: based on the state value function V(s; θ t ,β) and advantage function A(s,a;θ t ,α), the action value function Q(s,a;θ t ,α,β) is expressed as:

[0138]

[0139] (2) TD error calculation: Using the target Y t Update the action-value function:

[0140] Y t =r t+1 +γQ T (s t+1 ,argmax a Q E (s t+1 ,a;θ t ,α,β);θ t - ,α',β')

[0141] (3) Loss function calculation: Based on the difference between the current Q value and the target value:

[0142] L(θ t ,α,β)=Ε[(Y t -Q E (s,a;θ t ,α,β)) 2 ]

[0143] (4) Parameter update:

[0144]

[0145] In the formula, Q(s,a;θ t ,α,β) is the action value function, which is divided into the state value function V(s;θ t ,β) and advantage function A(s,a;θ t ,α); θ is the parameter of the neural network, which is used to update the Q value; α is the learning rate, which is used to update the Q network parameters; β is the priority weight of the priority experience replay.

[0146] In the present invention, samples are prioritized and stored according to TD error, the Q network is updated according to the priority sampling, high TD error samples are trained emphatically, and network parameters are adjusted through back propagation to improve training efficiency;

[0147] S6. Update the target network; regularly copy the parameters of the Q network to the target network to ensure the stability of the algorithm;

[0148] S7. Continuously initialize the state of the power grid dynamic reconstruction optimization model and the parameters of the intelligent agent, repeat steps S2-S5 until the strategy converges, output the optimized topology structure and SOP power regulation strategy, and realize the optimized reconstruction of the power grid.

[0149] The collaborative intelligent optimization reconstruction strategy in the power grid dynamic reconstruction optimization model provided by the present invention includes the following algorithm steps:

[0150] CIORS is based on the D3QN reinforcement learning algorithm. The specific steps include state initialization, alternating decision-making between discrete agents and continuous agents, collaborative reward calculation, and experience replay. The detailed process is as follows:

[0151] State initialization: Initialize system state (voltage, load, distributed energy output, SOP initial power) and agent parameters (Q network, target network, experience pool)

[0152] Discrete agent decision-making: Based on the current state and ε-greedy strategy, select a combination of switch operations, execute and update the system's network topology and power flow, and generate a new state

[0153] Continuous agent decision making: Based on the updated state, select the active and reactive power output of the SOP, execute and update the voltage and power flow of the system, and generate instant rewards

[0154] Reward calculation: Calculate the instant rewards of discrete and continuous agents (energy loss, voltage stability, clean energy consumption), calculate collaborative rewards, and dynamically adjust weights based on the current state

[0155] Priority experience replay: prioritize and store samples according to TD error, update the Q network according to priority sampling, focus on training samples with high TD error, adjust network parameters through back propagation, and improve training efficiency

[0156] Target network update: Regularly copy the parameters of the Q network to the target network to ensure the stability of the algorithm

[0157] Iteration and convergence judgment: Repeat steps S2-S6 until the strategy converges, output the optimized topology structure and SOP power regulation strategy, and realize dynamic optimization of the distribution network;

[0158] The invention proposes a dual-agent collaborative power grid optimization and reconstruction method, which adopts a dual-agent collaborative intelligent optimization and reconstruction strategy (Collaborative Intelligent Optimization Reconfiguration Strategy, CIORS) and has significant advantages over the prior art in the multi-objective optimization of dynamic distribution networks. The specific effects are as follows:

[0159] (1) Significant improvement in overall system optimization efficiency

[0160] CIORS effectively achieves global optimal control of the distribution network under multi-objective optimization through the collaborative optimization of discrete and continuous agents. The discrete agent is responsible for the reconstruction of the switch state to optimize the topology of the distribution network; the continuous agent ensures the voltage stability of the system and the priority consumption of clean energy by adjusting the active and reactive power of the smart soft switch (SOP). The division of labor and interaction of the dual-agent collaborative architecture enables the system to achieve optimization goals faster in a dynamic environment, significantly improving the overall efficiency.

[0161] (2) Enhanced adaptability and flexibility

[0162] CIORS introduces a dynamic weight mechanism to adjust the priority of each optimization objective in real time based on an S-shaped nonlinear function. This mechanism ensures the dynamic adaptability of the system under load fluctuations and uncertainty in clean energy output, allowing the optimization strategy to be flexibly adjusted as the system state changes, thereby avoiding the rigid response of traditional methods under multi-objective requirements.

[0163] (3) Improved sample utilization efficiency and faster convergence speed

[0164] The present invention selects key samples for priority learning during the training process through the Prioritized Experience Replay (PER) mechanism, which significantly improves the sample utilization efficiency and accelerates the convergence speed of the strategy. Compared with the traditional random sampling method, PER can ensure that the dual agents learn the optimal strategy more quickly in a complex distribution network environment, reduce training time and improve model performance.

[0165] (4) System energy loss is reduced and clean energy utilization rate is improved

[0166] In an environment with high penetration of clean energy, the present invention achieves maximum consumption of clean energy by giving priority to distributed energy such as wind power and photovoltaics. This not only reduces dependence on fossil fuels and improves the environmental benefits of the system, but also effectively reduces the energy loss of the system, providing strong support for energy conservation and consumption reduction and carbon emission reduction goals.

[0167] (5) Significant improvement in grid operation safety and economy

[0168] The optimization strategy of the present invention extends the service life of the equipment and reduces the operation and maintenance costs by limiting the frequency of switch operation. In addition, the coordinated control of SOP and traditional switches ensures the voltage stability of the system under different load conditions, effectively avoiding instability caused by voltage fluctuations or system overload, and providing higher safety and economic protection for the distribution network.

[0169] Therefore, this application proposes a method for dynamic reconstruction of power grids based on reinforcement learning, which considers the collaborative optimization of SOP and traditional switches. Through the innovative strategy of dual-agent collaborative optimization, it has significant advantages over traditional methods in terms of system energy efficiency, operational adaptability and environmental benefits, and provides an efficient and reliable multi-objective optimization solution for distribution networks with high penetration of clean energy.

[0170] The collaborative intelligent optimization reconstruction strategy proposed in this paper is based on the optimization framework of dual-agent reinforcement learning and is designed for multi-period dynamic distribution network reconstruction. The innovation of CIORS lies in that it effectively overcomes the limitations of existing research in global optimization and dynamic adaptation through collaborative mechanism and dynamic weight mechanism:

[0171] (1) Dual-agent collaborative architecture: The reconstruction task is divided into the optimization of switch state and SOP power output, which are completed by discrete and continuous agents respectively. The discrete agent is responsible for optimizing the switch operation and realizing the topological reconstruction of the distribution network; the continuous agent is responsible for adjusting the active and reactive power output of the SOP, improving the system voltage stability and giving priority to the consumption of clean energy.

[0172] (2) Collaborative reward mechanism: A collaborative reward mechanism is designed to strengthen the collaboration of the two agents on the global goal, so that the requirements of minimizing energy loss, maximizing clean energy consumption, and voltage stability can be balanced in multi-objective optimization. Collaborative rewards achieve better global results by guiding the joint actions of the agents.

[0173] (3) Dynamic weight mechanism: The dynamic weight mechanism is embedded in the comprehensive reward function to adjust the priority of immediate rewards and collaborative rewards in real time according to load fluctuations and clean energy output, thereby enhancing the agent's ability to respond to different optimization needs.

[0174] (4) Prioritized Experience Replay (PER): The Prioritized Experience Replay mechanism is introduced into the agent training to select key samples for priority learning, which improves the sample utilization efficiency and the convergence speed of the model, making it possible to quickly obtain stable optimization strategies in complex and changing environments.

Claims

1. A power grid optimization and reconstruction method based on dual-agent collaboration, characterized in that: The power grid optimization and reconstruction method optimizes the control of discrete intelligent agents and continuous intelligent agents in the power grid, and realizes the coordination of discrete intelligent agents and continuous intelligent agents by separating the switch state and SOP power output optimization tasks; the power grid optimization and reconstruction method includes the following steps: S1. Construct a power grid dynamic reconstruction optimization model including an objective function and constraint conditions; construct a dual-agent architecture based on the constraints of the power grid dynamic reconstruction optimization model, wherein the dual-agent architecture includes a discrete agent and a continuous agent; S2, initializing the state of the power grid dynamic reconstruction optimization model and the parameters of the intelligent agent; S3, giving discrete agent decision and continuous agent decision through dual agent architecture in power grid dynamic reconstruction optimization model; S4, calculate the instant rewards and collaborative rewards of discrete agents and continuous agents, and dynamically adjust the weights based on the current state; S5. Optimizing the power grid dynamic reconstruction optimization model by adopting the priority experience playback strategy; S6. Update the target network: regularly copy the parameters of the Q network to the target network to ensure the stability of the algorithm; S7, iteration and convergence judgment: repeat steps S2-S6 until the strategy converges, output the optimized power grid dynamic reconstruction optimization model topology structure and SOP power regulation strategy, and realize the optimized reconstruction of the power grid.

2. A power grid optimization and reconstruction method based on dual-agent collaboration according to claim 1, characterized in that: The objective functions of the dynamic reconstruction optimization model of the power grid in S1 include the energy loss minimization target, the clean energy consumption maximization target, the switch operation number minimization target and the voltage deviation minimization target; the energy loss minimization target: reduce the power loss of the system through the accumulated active power loss value of the system in each time period, thereby improving the operation economy of the system; Set clean energy consumption targets: By giving priority to distributed clean energy such as wind power and photovoltaics, indirectly support carbon emission reduction strategies, reduce dependence on fossil fuels, and improve the environmental benefits of the system; minimize the number of switching operations: constrain the operation frequency of switching equipment to reduce equipment wear; minimize voltage deviation: in multi-period optimization, suppress voltage fluctuations at each node and improve system stability.

3. A method for optimizing and reconfiguring a power grid based on dual-agent collaboration according to claim 1, characterized in that: The constraints of the dynamic reconstruction optimization model of the power grid in S1 include radial constraints, node voltage stability constraints, power balance constraints, SOP operation constraints, distributed energy output constraints, branch flow constraints and switch operation frequency constraints.

4. The method for optimizing and reconfiguring a power grid based on dual-agent collaboration according to claim 1 is characterized in that: The constructed dual-agent architecture includes designing the state space, action space, and immediate reward function of the discrete agent and the continuous agent respectively; designing the collaborative reward function of the discrete agent and the continuous agent based on the coupling optimization of the two agents; and designing the comprehensive reward function using the dynamic weight mechanism and the collaborative reward function of the discrete agent and the continuous agent.

5. A method for optimizing and reconfiguring a power grid based on dual-agent collaboration according to claim 4, characterized in that: The synergy reward function includes energy loss synergy reward, clean energy consumption synergy reward and comprehensive synergy reward.

6. A method for optimizing and reconfiguring a power grid based on dual-agent collaboration according to claim 4, characterized in that: In S4, the immediate rewards of discrete agents and continuous agents are calculated through immediate reward functions, and the collaborative rewards of discrete agents and continuous agents are calculated through collaborative reward functions.

7. A method for optimizing and reconfiguring a power grid based on dual-agent collaboration according to claim 6, characterized in that: The dynamic weight mechanism adopted in S4 to dynamically adjust the weight based on the current state is used to adjust the weights of different optimization objectives in the system in real time, so as to adapt to the changes in the power grid state and ensure the reasonable allocation of priorities of various optimization objectives at different stages.

8. The method for optimizing and reconfiguring a power grid based on dual-agent collaboration according to claim 1, characterized in that: The priority experience replay strategy in S5 introduces a priority experience replay mechanism on the traditional D3QN framework to improve the learning efficiency of the model in complex power grids.

9. A method for optimizing and reconfiguring a power grid based on dual-agent collaboration according to claim 8, characterized in that: In S5, the priority experience playback mechanism sorts the samples based on the time difference error and assigns the sample sampling priority according to the error value of the time difference error.

10. The method for optimizing and reconfiguring a power grid based on dual-agent collaboration according to claim 1, characterized in that: In S1, the state of the dynamic reconstruction optimization model of the power grid is initialized, including the initialization of the distribution system voltage, load, distributed energy output, and SOP initial power; the parameters of the intelligent agent are initialized, including the Q network, target network, and experience pool.

Images