Multi-agent voltage cooperative control method for multiple types of voltage regulating devices

Abstract

The invention discloses a multi-agent voltage cooperative control method for multi-type voltage regulating devices, and relates to the technical field of electrical engineering and the technical field of computer science, and the method comprises the steps: 10), aiming at the operation characteristics of a static var compensator, a photovoltaic inverter, an energy storage device and a line voltage regulator in a power distribution network, carrying out the cooperative control of the voltage of a plurality of types of voltage regulating devices; constructing a voltage control model capable of coordinating different types of voltage regulation equipment; (20) based on the voltage control model, control variables are submitted to different agents for independent learning and decision making, and a voltage control problem is converted into an MDP problem which can be solved by MADRL; (30) an MADRL algorithm based on a multi-agent soft actor-commentator algorithm is adopted to solve the MDP problem, and guide-softmax is introduced for a discrete voltage regulation device to improve the algorithm; and step 40) obtaining a multi-device cooperative voltage control scheme through parallel training and online execution of a plurality of agents. Compared with the prior art, the method has the advantages that a unified voltage control model capable of coordinating different types of voltage regulation equipment is constructed from the perspective of power distribution network voltage optimization regulation and control, strategy optimization is carried out on control decisions of various types of voltage regulation equipment through a multi-agent reinforcement learning algorithm, and coordinated regulation of the power distribution network voltage is achieved.

Description

Technical Field

[0001] This invention relates to the fields of electrical engineering and computer technology, specifically a multi-agent voltage collaborative control method for various types of voltage regulating devices. Background Technology

[0002] As the scale of distributed photovoltaic (PV) power generation in distribution networks continues to expand, the power supply structure of distribution systems is gradually shifting from the traditional unidirectional radial operation mode to a multi-source, mutually coupled operation mode. This change significantly improves the operational flexibility of distribution networks, but also makes their topology and electrical characteristics more complex, placing higher demands on the accuracy, response speed, and coordination capabilities of voltage regulation. Meanwhile, the widespread application of static var compensators (SVCs), energy storage systems, and distributed power sources with regulation capabilities in distribution networks provides diverse regulation methods for voltage optimization, but also further enhances the nonlinear characteristics and operational uncertainties of the system.

[0003] Current voltage control methods for distribution networks are mostly based on physical models of the system, achieving optimization by establishing mathematical models and combining linearization or parameter simplification. However, with the continuous expansion of distribution network scale and frequent changes in operating conditions, it is difficult to simultaneously guarantee the accuracy of the model and the real-time performance of the control. Especially in the case of a high proportion of distributed power sources, the distribution network structure is complex and the randomness of power fluctuations is enhanced. Traditional model-based control methods often fail to maintain stable and effective voltage regulation performance, and may even experience control performance degradation. In addition, such methods are usually computationally complex and time-consuming, making it difficult to meet the real-time regulation requirements under conditions of rapid voltage fluctuations in the distribution network. To address this, a multi-agent voltage collaborative control method for multiple types of voltage regulating devices is proposed. This method collaboratively models continuous and discrete voltage regulation variables under a unified voltage control model and optimizes the control strategies of each voltage regulating device within a multi-agent reinforcement learning framework, thereby achieving dynamic and coordinated regulation of the distribution network voltage. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-agent voltage collaborative control method for multiple types of voltage regulating devices, which realizes effective collaborative control of different types of voltage regulating devices in power distribution network voltage control, and effectively improves the voltage stability and safe operation level of power distribution network.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-agent voltage collaborative control method for multiple types of voltage regulating devices, comprising the following steps:

[0006] Based on the operating characteristics of various voltage regulation devices in the distribution network, such as static var compensators, photovoltaic inverters, energy storage devices, and line voltage regulators, a voltage control model capable of coordinating different types of voltage regulation equipment is constructed.

[0007] Based on the voltage control model, the control variables are independently learned and decided by different agents, transforming the voltage control problem into an MDP problem that can be solved by MADRL.

[0008] The MADRL algorithm based on the multi-agent soft actor-commentator algorithm is used to solve the MDP problem, and gumble-softmax is introduced to improve the algorithm for discrete voltage regulators.

[0009] By training and executing multiple agents in parallel and online, a multi-device collaborative voltage control scheme is obtained.

[0010] Furthermore, at the beginning of each time step, the line voltage regulator is configured, and the outputs of the inverter, static var compensator (SVC), and energy storage device charging and discharging are controlled. The corresponding control equipment model is represented as follows:

[0011]

[0012] In the formula For the control variables of the static var compensator; For line voltage regulator control variables; For control variables of photovoltaic inverters; For control variables of energy storage devices; The reactive power nameplate value of the static var compensator; The rated capacity of the inverter. U′(T) represents the reactive power output of the capacitor, and U′(T) is the reference value of the bus voltage. For the reactive power output of the photovoltaic inverter, P i PV (t) represents the active power output of the photovoltaic inverter. This represents the maximum available reactive power capacity of the photovoltaic inverter; V max and V min These are the maximum and minimum allowable voltage values, respectively, for LVR. max and LVR min These represent the maximum and minimum values ​​for the gear, P. i ESS ,max For maximum charge / discharge power, P i ESS (t) The charging and discharging power of the energy storage device.

[0013] Furthermore, based on the control characteristics of various voltage regulating devices, the corresponding control variables are assigned to different intelligent agents for processing; the voltage regulation problem is abstracted into an MDP problem, and the State space, Action space, and Reward function are defined for each intelligent agent:

[0014] ai (t)=[a1(t),a2(t),...,a n (t)] T

[0015] Where i∈{SVC,LVR,ESS,PV};

[0016]

[0017] In the above formula, a is the Action space, s is the State space, and r is the reward function. This refers to the action of the static reactive power compensator at the previous moment. This refers to the action of the line voltage regulator at the previous moment. U(t) represents the average active power of all buses in time T, U(t) is the voltage amplitude of all buses in time t, and r is the Reward function.

[0018] At time step t, the agent selects control action a according to policy π. t And receive cumulative discount rewards R t :

[0019]

[0020] Where r t+n It is the reward at time step t+n, γ∈[0,1] is the discount factor, and n is the index of the future time step, representing the nth time step from the current time t;

[0021] To describe the expected return of a policy under different state-action combinations, an action value function Q is introduced. π (s t ,a t ):

[0022] Q π (s t ,a t ) = E π [R t |s t ,a t ]

[0023] Where E π [] represents the expectation of π under a given policy, where the control policy π is used to describe the system state s. t With corresponding control action a t The mapping relationship between them, s t For the agent's observation information at time t, a t To control actions;

[0024] According to the definition of the Bellman equation, the action-value function can be recursively expressed as:

[0025] Q π (s t ,a t ) = E π [r t +γQ π (s t+1 ,a t+1 )).

[0026] Furthermore, to measure the overall performance of the strategy, a performance function J(π) is defined:

[0027]

[0028] Where, ρ π Let π represent the discount state distribution under policy π, a~π represent the probability distribution of a certain job, and the performance function J(π) represents the expected cumulative discount reward that can be obtained by choosing an action according to policy π at each time step.

[0029] Optimal control strategy π * This is obtained by maximizing the performance function J(π), i.e.:

[0030]

[0031] Furthermore, for the constructed MDP process involving multiple agents, a multi-agent deep reinforcement learning algorithm based on MASAC is employed; the MASAC includes a policy network and an action-value network, whereby the policy network represents the random policy distribution π(a|s; θ). π Value networks are used to approximate the soft Q-function Q(s,a;θ). Q To reduce Q-value overestimation, a dual-commenter structure is introduced, where each agent is configured with two sets of critic networks Q1, Q2 and their corresponding target networks Q1′, Q2′.

[0032] During the centralized training phase, each agent's critic network input includes not only its own state and action information, but also the state and action information of other agents, in order to form a centralized value assessment. During the execution phase, each agent outputs actions through the policy network based solely on local observations, thereby achieving distributed control.

[0033] For continuous variable devices such as static var compensators, energy storage devices, and photovoltaic inverters, after sampling the minimum pool from the replay buffer, the critic for each agent i is updated by minimizing the following loss:

[0034] L(θ i,1 )=E[(Q i,1 (s,a)-y i ) 2 ]

[0035] Among them, soft target y i Defined as:

[0036] y i =r i +γ(1-d)(min(Q′ i,1 (s′,a′),Q′ i,2 (s′,a′))-αlogπ i (a′ i |o′ i ))

[0037] The policy network of each agent i is updated according to the maximum entropy objective, and its loss can be written as:

[0038]

[0039] The target network uses a soft update method:

[0040] θ′←τθ+(1-τ)θ′

[0041] Where τ is the soft update coefficient, 0 < τ1.

[0042] 6. A multi-agent voltage collaborative control method for multiple types of voltage regulating devices according to claim 1, characterized in that step 3 includes the following steps: For the discrete control device line voltage regulator, a probabilistic strategy-based action selection method is adopted, and the output vector of the actor network is used as the probability distribution corresponding to each voltage regulation level of the line voltage regulator, wherein each element of the vector represents the probability of selecting the corresponding level; in this way, the discrete decision problem of the voltage regulation level of the line voltage regulator is transformed into a continuous optimization problem of its corresponding probability distribution parameters, and solved by the MASAC algorithm under the maximum entropy reinforcement learning framework.

[0043] For continuously moving devices, the actor outputs Gaussian distribution parameters and samples the motion:

[0044] a i ~π i (·|o i )

[0045] The motion is limited using the tanh(·) equation, and the corresponding logπ is calculated simultaneously. i (a i |o i ).

[0046] For OLTC discrete action devices, the actor outputs an action probability vector, and actions are sampled according to probability during training:

[0047] a t =SampleByProb(Pt )

[0048] The action with the highest probability can be used during execution:

[0049] a t =argmax(P t )

[0050] This invention has at least the following beneficial effects:

[0051] This invention provides a multi-agent voltage collaborative control method for various types of voltage regulating devices. Based on the voltage optimization control requirements of power distribution networks, this method constructs a multi-device voltage collaborative control model under the background of large-scale distributed and controllable equipment access, and introduces a reinforcement learning framework combined with the MASAC algorithm to achieve hierarchical decision-making and collaborative control of discrete and continuous voltage regulating devices.

[0052] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the method described in this invention.

[0054] Figure 2 This is a diagram showing the distribution structure and voltage regulation equipment of the IEEE-123 node distribution network in this embodiment of the invention.

[0055] Figure 3 This is a cumulative reward curve during the reinforcement learning training phase in this embodiment of the invention.

[0056] Figure 4 This is a voltage distribution diagram at 0:00 on a certain day under the original load and different control methods in an embodiment of the present invention.

[0057] Figure 5 This is a voltage distribution diagram at 12:00 on a certain day under the original load and different control methods in an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0059] Example 1:

[0060] Please see Figures 1-5The present invention provides a technical solution: a multi-agent voltage collaborative control method for multiple types of voltage regulating devices, comprising the following steps:

[0061] S1. Based on the operating characteristics of various voltage regulation units such as static var compensators, photovoltaic inverters, energy storage devices and line voltage regulators in the distribution network, a voltage control model that can coordinate different types of voltage regulation equipment is constructed.

[0062] S11. At the beginning of each time step, the line voltage regulator is configured, and the outputs of the inverter, static var compensator, and energy storage device are controlled. The corresponding control equipment model is represented as follows:

[0063]

[0064] In the formula For the control variables of the static var compensator; For line voltage regulator control variables; For control variables of photovoltaic inverters; For control variables of energy storage devices; The reactive power nameplate value of the static var compensator; The rated capacity of the inverter. U′(T) represents the reactive power output of the capacitor, and U′(T) is the reference value of the bus voltage. For the reactive power output of the photovoltaic inverter, P i PV (t) represents the active power output of the photovoltaic inverter. This represents the maximum available reactive power capacity of the photovoltaic inverter; V max and V min These are the maximum and minimum allowable voltage values, respectively, for LVR. max and LVR min These represent the maximum and minimum values ​​for the gear, P. i ESS ,max For maximum charge / discharge power, P i ESS (t) The charging and discharging power of the energy storage device.

[0065] S2. Based on the aforementioned voltage control model, the control variables are independently learned and decided by different agents, transforming the voltage control problem into an MDP problem solvable by MADRL.

[0066] S21. Based on the control characteristics of various voltage regulators, the corresponding control variables are assigned to different agents for processing; the voltage regulation problem is abstracted into an MDP problem, and the State space, Action space, and Reward function are defined for each agent:

[0067] ai (t)=[a1(t),a2(t),…,a n (t)] T

[0068] Where i∈{SVC,LVR,ESS,PV};

[0069]

[0070] In the above formula, a is the Action space, s is the State space, and r is the reward function. This refers to the action of the static reactive power compensator at the previous moment. This refers to the action of the line voltage regulator at the previous moment. U(t) represents the average active power of all buses in time T, U(t) is the voltage amplitude of all buses in time t, and r is the Reward function.

[0071] S22. At time step t, the agent selects control action a according to policy π. t And receive cumulative discount rewards R t :

[0072]

[0073] Where r t+n It is the reward at time step t+n, γ∈[0,1] is the discount factor, and n is the index of the future time step, representing the nth time step from the current time t;

[0074] S23. To describe the expected return of a strategy under different state-action combinations, an action value function Q is introduced. π (s t ,a t ):

[0075] Q π (s t ,a t ) = E π [R t |s t ,a t ]

[0076] Where E π [] represents the expectation of π under a given policy, where the control policy π is used to describe the system state s. t With corresponding control action a t The mapping relationship between them, s t For the agent's observation information at time t, a t To control actions;

[0077] S24. According to the definition of the Bellman equation, the action-value function can be recursively expressed as:

[0078] Q π (s t ,a t ) = E π [r t +γQ π (s t+1 ,a t+1 )).

[0079] S25. Further, to measure the overall performance of the strategy, a performance function J(π) is defined:

[0080]

[0081] Where, ρ π Let π represent the discount state distribution under policy π, a~π represent the probability distribution of a certain job, and the performance function J(π) represents the expected cumulative discount reward that can be obtained by choosing an action according to policy π at each time step.

[0082] S26. Optimal Control Strategy π * This is obtained by maximizing the performance function J(π), i.e.:

[0083]

[0084] S3. The MADRL algorithm based on the multi-agent soft actor-commentator algorithm is used to solve the MDP problem, and gumble-softmax is introduced to improve the algorithm for discrete voltage regulation devices;

[0085] S31. For the MDP process involving multiple agents constructed in S2, a multi-agent deep reinforcement learning algorithm based on MASAC is adopted; the MASAC includes a policy network and an action-value network, and the policy network is used to represent the random policy distribution π(a|s; θ). π Value networks are used to approximate the soft Q-function Q(s,a;θ). Q To reduce Q-value overestimation, a dual-commenter structure is introduced, where each agent is configured with two sets of critic networks Q1, Q2 and their corresponding target networks Q1′, Q2′.

[0086] During the centralized training phase, each agent's critic network input includes not only its own state and action information, but also the state and action information of other agents, in order to form a centralized value assessment. During the execution phase, each agent outputs actions through the policy network based solely on local observations, thereby achieving distributed control.

[0087] S311. For continuous variable devices such as static var compensators, energy storage devices, and photovoltaic inverters, after sampling the minimum pool from the replay buffer, the critic for each agent i is updated by minimizing the following loss:

[0088] L(θ i,1 )=E[(Q i,1 (s,a)-y i ) 2 ]

[0089] Among them, soft target y i Defined as:

[0090] y i =r i +γ(1-d)(min(Q′ i,1 (s′,a′),Q′ i,2 (s′,a′))-αlogπ i (a′ i |o′ i ))

[0091] The policy network of each agent i is updated according to the maximum entropy objective, and its loss can be written as:

[0092]

[0093] The target network uses a soft update method:

[0094] θ′←τθ+(1-τ)θ′

[0095] Where τ is the soft update coefficient, 0 < τ1.

[0096] S312. For the line voltage regulator of the discrete control device, a probabilistic strategy-based action selection method is adopted. The output vector of the actor network is used as the probability distribution corresponding to each voltage regulation position of the line voltage regulator, where each element of the vector represents the probability of selecting the corresponding position. In this way, the discrete decision problem of the voltage regulation position of the line voltage regulator is transformed into a continuous optimization problem of its corresponding probability distribution parameters, and the problem is solved by the MASAC algorithm under the maximum entropy reinforcement learning framework.

[0097] For continuously moving devices, the actor outputs Gaussian distribution parameters and samples the motion:

[0098] a i ~π i (·|o i )

[0099] The motion is limited using the tanh(·) equation, and the corresponding logπ is calculated simultaneously. i (a i |oi ).

[0100] For OLTC discrete action devices, the actor outputs an action probability vector, and actions are sampled according to probability during training:

[0101] a t =SampleByProb(P t )

[0102] The action with the highest probability can be used during execution:

[0103] a t =argmax(P t S4. Construct agent models based on the deep reinforcement learning algorithm proposed in step S3. Each agent establishes its corresponding actor network, critic network, and replay buffer, and trains using the training data according to the algorithm in S3. During the execution phase, each agent autonomously decides control actions based on the current operating state of the distribution network, achieving coordinated voltage control of energy storage, inverters, line voltage regulators, and static var compensators.

[0104] The technical solution of the present invention will be further described below with reference to a specific embodiment:

[0105] A comparative experiment was conducted based on the following two machine learning algorithms:

[0106] Algorithm 1: MADDPG

[0107] Algorithm 2: MASAC

[0108] Figure 1 This is a flowchart illustrating the method of this embodiment, which introduces the basic steps of the method of the present invention:

[0109] Step 1: Based on the operating characteristics of various voltage regulation devices such as static var compensators, photovoltaic inverters, energy storage devices, and line voltage regulators in the distribution network, construct a voltage control model that can coordinate different types of voltage regulation equipment.

[0110] Step 2: Based on the voltage control model, the control variables are independently learned and decided by different agents, transforming the voltage control problem into an MDP problem that can be solved by MADRL;

[0111] Step 3: The MADRL algorithm based on the multi-agent soft actor-commentator algorithm is used to solve the MDP problem. Gumble-softmax is introduced to improve the algorithm for discrete voltage regulating devices.

[0112] Step 4: Obtain a multi-device collaborative voltage control scheme by training and executing multiple agents in parallel and online.

[0113] 1) To verify the feasibility of the method proposed in this invention, an improved IEEE-123 node distribution network was selected as a case study for analysis. The rated voltage of the system was set to 4.16 kV. Its network topology is as follows: Figure 2 As shown in the diagram, in this power distribution network, 12 distributed photovoltaic (PV) units with a capacity of 0.4 MVA are configured at buses 24, 31, 39, 50, 63, 70, 79, 87, 92, 100, 106, and 113, respectively, and grid-connected inverters with reactive power regulation capabilities are configured to support voltage regulation requirements. Four sets of static var compensators (SVCs) are connected in parallel on buses 33, 59, 64, and 115, each with a reactive power compensation capacity of 0.04 Mvar. Four energy storage devices are installed at buses 54, 85, 94, and 116, with a maximum capacity of 0.5 MWh and a rated charging / discharging power of 0.08 MW. A line voltage regulator is installed between node 122 and node 1, with an adjustable voltage range of 0.9–1.1 pu, a total of 17 levels, and a voltage regulation step of 1.25% for each level.

[0114] 2) During the simulation, it is assumed that each distributed photovoltaic unit has consistent power generation characteristics. Its active power output curve is normalized according to the illumination variation characteristics under typical sunny conditions and used as the time series input of photovoltaic power output. The load-side power data is scaled with reference to the typical daily load variation pattern of the actual distribution area. Different load factors are used for different nodes to reflect the differences in load distribution in the distribution network.

[0115] 3) This embodiment is implemented based on the PyTorch framework, and the training process is executed on GPU and CPU. The multi-agent algorithm adopts the MASAC structure. Each agent contains one actor network and two sets of critic networks, and corresponding target critic networks are configured. Soft updates are used to achieve target network parameter following. The actor network includes an input layer, two fully connected layers, and an output layer. The input layer is the state space of the Markov decision process, the two fully connected layers are used for feature extraction, and the output layer is used to generate action distribution parameters. For continuous action agents, the actor outputs Gaussian distribution parameters and samples them. For discrete action agents, the actor uses Gumbel-Softmax to achieve differentiable probability sampling.

[0116] Table 1 Other hyperparameter settings for DRL

[0117]

[0118]

[0119] 4) The multi-agent voltage collaborative control method for various types of voltage regulators proposed in this patent is used to learn the MASAC-based multi-agent control strategy in parallel according to a predetermined training process. The training process consists of 200 rounds, each containing 288 control steps, with a corresponding time resolution of t = 15 minutes. During the training process, the cumulative reward changes for each agent are as follows: Figure 3 As shown, the cumulative reward gradually stabilizes after about the 30th round as the number of training rounds increases, indicating that the learned control strategy has reached a stable state and is capable of being applied to online voltage control of the system. Further comparison of the training results of different algorithms reveals that, in the stable phase, the cumulative reward level obtained by the MASAC algorithm is generally better than that of MADDPG, indicating that the MASAC method has higher learning efficiency and better training stability in the collaborative control scenario of continuous and discrete voltage regulation devices.

[0120] 5) Based on the trained MASAC multi-agent control strategy, voltage regulation control is implemented for different types of voltage regulators in the power distribution system. Relevant simulation results are as follows: Figure 4 and Figure 5 As shown, compared with the operating state without any voltage control measures, the proposed method can effectively improve the voltage distribution at each node of the distribution network, significantly reduce the node voltage deviation, and keep it stably within the allowable safe operating range, indicating that the method has good voltage regulation capability.

[0121] 6) To verify the effectiveness of the voltage control method proposed in this invention, it was compared and analyzed with different control strategies. The relevant results are as follows: Figure 4 and Figure 5 As shown in the figure. The results indicate that without voltage control, some node voltages exceed the safe operating range during operation; however, with the control method proposed in this invention, the voltages of all nodes can be effectively adjusted to the safe operating range. In contrast, although the control method MADDPG can achieve convergence during the training phase, its control effect in the test scenario is poor, making it difficult to maintain the stability of the node voltages, and the voltage deviation even increases further under some operating conditions.

[0122] In summary, this invention effectively improves the voltage regulation capability of the power distribution system by uniformly modeling the operating characteristics of different types of voltage regulating devices and combining multi-agent reinforcement learning methods to achieve collaborative control. This method enables coordinated decision-making for multiple types of voltage regulating devices, significantly improving node voltage distribution characteristics, thereby enhancing the safety, stability, and overall control level of the power distribution network.

Claims

1. A multi-agent voltage collaborative control method for multiple types of voltage regulating devices, characterized in that, Includes the following steps: Based on the operating characteristics of various voltage regulation devices in the distribution network, such as static var compensators, photovoltaic inverters, energy storage devices, and line voltage regulators, a voltage control model capable of coordinating different types of voltage regulation equipment is constructed. Based on the voltage control model, the control variables are independently learned and decided by different agents, transforming the voltage control problem into an MDP problem that can be solved by MADRL. The MADRL algorithm based on the multi-agent soft actor-commentator algorithm is used to solve the MDP problem, and gumble-softmax is introduced to improve the algorithm for discrete voltage regulators. By training and executing multiple agents in parallel and online, a multi-device collaborative voltage control scheme is obtained.

2. The power distribution network voltage control method according to claim 1, characterized in that: Step 1 includes: at the beginning of each time step, configuring the line voltage regulator and controlling the output of the inverter, static var compensator, and the charging and discharging of the energy storage device. The corresponding control equipment model is represented as follows: In the formula For the control variables of the static var compensator; For line voltage regulator control variables; For control variables of photovoltaic inverters; For control variables of energy storage devices; The reactive power nameplate value of the static var compensator; The rated capacity of the inverter. U′(T) represents the reactive power output of the capacitor, and U′(T) is the reference value of the bus voltage. For the reactive power output of the photovoltaic inverter, P i PV (t) represents the active power output of the photovoltaic inverter, Q. i PV,max (t) represents the current maximum available reactive power capacity of the photovoltaic inverter; V max and V min These are the maximum and minimum allowable voltage values, respectively, for LVR. max and LVR min These represent the maximum and minimum values ​​for the gear, P. i ESS ,max For maximum charge / discharge power, P i ESS (t) The charging and discharging power of the energy storage device.

3. The multi-agent voltage collaborative control method for multiple types of voltage regulating devices according to claim 1, characterized in that, Step 2 includes: assigning the corresponding control variables to different agents for processing based on the control characteristics of various voltage regulating devices; abstracting the voltage regulation problem into an MDP problem, and defining the State space, Action space, and Reward function for each agent. a i (t)=[a1(t),a2(t),...,a n (t)] T Where i∈{SVC,LVR,ESS,PV}; In the above formula, a is the Action space, s is the State space, and r is the reward function. This refers to the action of the static reactive power compensator at the previous moment. This refers to the action of the line voltage regulator at the previous moment. U(t) represents the average active power of all buses in time T, U(t) is the voltage amplitude of all buses in time t, and r is the Reward function. At time step t, the agent selects control action a according to policy π. t and receive cumulative discount rewards R t : Where r t+n It is the reward at time step t+n, γ∈[0,1] is the discount factor, and n is the index of the future time step, representing the nth time step from the current time t; To describe the expected return of a policy under different state-action combinations, an action value function Q is introduced. π (s t ,a t ): Q π (s t ,a t )=E π [R t ∣s t ,a t ] Where E π [] represents the expectation of π under a given policy, where the control policy π is used to describe the system state s. t With corresponding control action a t The mapping relationship between them, s t For the agent's observation information at time t, a t To control actions; According to the definition of the Bellman equation, the action-value function can be recursively expressed as: Q π (s t ,a t )=E π [r t +γQ π (s t+1 ,a t+1 )]。 4. The multi-agent voltage collaborative control method for multiple types of voltage regulating devices according to claim 1, characterized in that, To measure the overall performance of the strategy, a performance function J(π) is defined: Where, ρ π Let π represent the discount state distribution under policy π, a ~ π represent the probability distribution of a certain job, and the performance function J(π) represents the expected cumulative discount reward that can be obtained by choosing an action according to policy π at each time step. Optimal control strategy π * This is obtained by maximizing the performance function J(π), i.e.:

5. The multi-agent voltage collaborative control method for multiple types of voltage regulating devices according to claim 1, characterized in that, Step 3 includes the following steps: For the MDP process involving multiple agents constructed in Step 2, a multi-agent deep reinforcement learning algorithm based on MASAC is adopted; the MASAC includes a policy network and an action-value network, the policy network being used to represent the random policy distribution π(a|s; θ). π Value networks are used to approximate the soft Q-function Q(s,a;θ). Q To reduce Q-value overestimation, a dual-commenter structure is introduced, where each agent is configured with two sets of critic networks Q1, Q2 and their corresponding target networks Q′1, Q′2. During the centralized training phase, each agent's critic network input includes not only its own state and action information, but also the state and action information of other agents, in order to form a centralized value assessment. During the execution phase, each agent outputs actions through the policy network based solely on local observations, thereby achieving distributed control. For continuous variable devices such as static var compensators, energy storage devices, and photovoltaic inverters, after sampling the minimum pool from the replay buffer, the critic for each agent i is updated by minimizing the following loss: L(θ i,1 )=E[(Q i,1 (s,a)-y i ) 2 ] Among them, soft target y i Defined as: y i =r i +γ(1-d)(min(Q′ i,1 (s′,a′),Q′ i,2 (s′,a′))-αlogπ i (a i ′|o i ′)) The policy network of each agent i is updated according to the maximum entropy objective, and its loss can be written as: The target network uses a soft update method: θ′←τθ+(1-τ)θ′ Where τ is the soft update coefficient, 0 < τ1.

6. The multi-agent voltage collaborative control method for multiple types of voltage regulating devices according to claim 1, characterized in that, Step 3 includes the following steps: For the discrete control device line voltage regulator, a probabilistic strategy-based action selection method is adopted, and the output vector of the actor network is used as the probability distribution corresponding to each voltage regulation position of the line voltage regulator, where each element of the vector represents the probability of selecting the corresponding position; in this way, the discrete decision problem of the voltage regulation position of the line voltage regulator is transformed into a continuous optimization problem of its corresponding probability distribution parameters, and solved by the MASAC algorithm under the maximum entropy reinforcement learning framework. For continuously moving devices, the actor outputs Gaussian distribution parameters and samples the motion: to i ~π i (·|o i ) The motion is limited using the tanh(·) equation, and the corresponding logπ is calculated simultaneously. i (a i |o i ). For OLTC discrete action devices, the actor outputs an action probability vector, and actions are sampled according to probability during training: a t =SampleByProb(P t ) The action with the highest probability can be used during execution: a t =argmax(P t )。

Images