A reinforcement learning power distribution network voltage regulation method and system considering photovoltaic heterogeneous characteristics

Abstract

The application discloses a kind of reinforcement learning power distribution network voltage regulation method and system considering photovoltaic heterogeneity characteristics, it is related to power system operation optimization and artificial intelligence technical field, including based on voltage active and voltage reactive sensitivity Construction electrical distance index, and the power grid is divided into multiple sub-regions, constructs multi-agent distributed control architecture;According to the control difference of photovoltaic equipment in power distribution network, three kinds of heterogeneous photovoltaic inverter models including uncontrollable, only reactive power controllable, active and reactive power collaborative control are constructed;The distributed voltage regulation problem of power distribution network is modeled as Markov game, and a training framework based on multi-agent flexible actor-critic algorithm is constructed, and the agent is centrally trained using historical operation data;The trained agent is deployed to each sub-region, and each agent generates a control strategy in real time according to local observation information, and is mapped to the actual power setting value of the heterogeneous photovoltaic inverter.The method of the application accurately cooperates and regulates heterogeneous resources.

Description

Technical Field

[0001] This invention relates to the fields of power system operation optimization and artificial intelligence technology, specifically to a reinforcement learning-based distribution network voltage regulation method and system that considers the heterogeneous characteristics of photovoltaics. Background Technology

[0002] With increasing global energy demand, distributed energy sources, represented by photovoltaics (PV), are increasingly penetrating power distribution networks. However, due to the significant intermittency and uncertainty of PV power generation, the high proportion of PV grid integration greatly increases the operational complexity of active distribution networks, especially leading to frequent voltage fluctuations and even voltage limit violations. Current voltage regulation methods mainly utilize traditional physical devices such as on-load tap changers and static var compensators (SVCs). However, these devices typically suffer from high costs, slow response times, and shortened lifespans due to frequent operation, making them ill-suited to handle rapid voltage fluctuations caused by PV. Therefore, utilizing the reactive power of PV inverters for voltage regulation has become an ideal alternative. In existing PV inverter-based voltage regulation research, stochastic programming or robust optimization methods are mainly used to address the uncertainty of PV power generation. However, stochastic programming methods require extensive scenario sampling and reduction, resulting in a heavy computational burden and difficulty meeting real-time requirements; while robust optimization methods are often too conservative, leading to uneconomical and inefficient control strategies. In recent years, deep reinforcement learning has been widely applied to voltage control due to its powerful decision-making capabilities. However, existing single-agent reinforcement learning methods struggle to handle high-dimensional state spaces in large-scale systems, while independent multi-agent methods lacking cooperative mechanisms are prone to control strategy conflicts. More critically, most existing research on photovoltaic (PV) control models neglects the heterogeneous nature of PV systems. In actual distribution network operation, due to differences in construction cycles, equipment manufacturers, and communication coverage, distributed PV systems are not ideally homogeneous, meaning they are not assumed to be identical in capacity, communication infrastructure, and control strategies. In reality, distributed PV exhibits significant heterogeneity: some systems only have maximum power point tracking (MPPT) and are uncontrollable, some can only regulate reactive power, while others possess both active power reduction and reactive power regulation capabilities. Ignoring this heterogeneity leads to control models that fail to accurately reflect the real power grid operating environment, thus affecting the accuracy and effectiveness of voltage regulation. Therefore, a distribution network voltage regulation method that fully considers the heterogeneous characteristics of PV systems and possesses high computational efficiency and global cooperative capabilities is urgently needed. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by this invention is that existing power distribution network voltage regulation methods suffer from slow equipment operation, overly conservative robust optimization, difficulty in processing high-dimensional data with single-agent algorithms, and the problem of how to ignore control inaccuracies caused by equipment heterogeneity.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a reinforcement learning-based distribution network voltage regulation method considering the heterogeneous characteristics of photovoltaics, comprising constructing an electrical distance index based on voltage-active and voltage-reactive power sensitivity, dividing the power grid into multiple sub-regions, and constructing a multi-agent distributed control architecture; constructing three types of heterogeneous photovoltaic inverter models, including uncontrollable, reactive power-only controllable, and active and reactive power coordinated control, based on the control differences of photovoltaic equipment in the distribution network; using extracted real-time voltage, photovoltaic, and load data as state features, and inverter power regulation commands as the action space, transforming the voltage smoothing problem into a Markov game process, and constructing a training framework based on a multi-agent flexible actor-commentator algorithm, using historical operating data for centralized training of the agents; deploying the trained agents to each sub-region, with each agent generating control strategies in real time based on local observation information, and mapping normalized control commands to the actual power setpoints of the heterogeneous photovoltaic inverters.

[0006] As a preferred embodiment of the reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity described in this invention, the power grid is divided into multiple sub-regions, including voltage-active power sensitivity and voltage-reactive power sensitivity between output nodes, and a comprehensive electrical distance index is obtained by weighting the sub-regions using weight coefficients; a similarity matrix is ​​constructed based on the comprehensive electrical distance index, and a nearest neighbor propagation clustering algorithm is used to determine the cluster center by iteratively updating the attraction and membership information, thereby dividing the sub-regions.

[0007] As a preferred embodiment of the reinforcement learning-based distribution network voltage regulation method considering photovoltaic heterogeneity as described in this invention, the comprehensive electrical distance index is expressed as: , in, To comprehensively consider electrical distance indicators, For the node of voltage-reactive sensitivity and nodes Electrical distance between them This is the electrical distance index of the voltage-active power sensitivity matrix.

[0008] As a preferred embodiment of the reinforcement learning distribution network voltage regulation method considering the heterogeneous characteristics of photovoltaics described in this invention, the three types of heterogeneous photovoltaic inverter models include: an uncontrollable photovoltaic inverter model, where the active power output follows the maximum power point tracking strategy and the reactive power output remains zero; a photovoltaic inverter model where only reactive power is controllable, where the active power is uncontrollable and the reactive power is adjustable within the range of the inverter's rated apparent power and power factor constraints; and a photovoltaic inverter with coordinated control of active and reactive power, which has the ability to reduce active power and uses the capacity released after reducing active power to enhance reactive power support capability.

[0009] As a preferred embodiment of the reinforcement learning-based distribution network voltage regulation method considering the heterogeneous characteristics of photovoltaics described in this invention, the distributed voltage regulation problem of the distribution network is modeled as a Markov game, which includes defining the three elements of a Markov game: state space, action space, and reward function. The state space contains information on the active power output, reactive power output, active load demand, and reactive load demand of each node in each sub-region. The action space consists of normalized control commands, including reactive power control commands for photovoltaics with only reactive power controllable, and active power reduction commands and reactive power control commands for photovoltaics with coordinated active and reactive power control.

[0010] As a preferred embodiment of the reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity described in this invention, the reward function includes a function designed to guide the agent to reduce the distribution network voltage fluctuation level at each time step, wherein the instantaneous reward obtained by the agent at each time step is expressed as: , in, For instant rewards, For intelligent agents, This is the scaling factor. This represents the actual voltage at the node. This is the reference voltage.

[0011] As a preferred embodiment of the reinforcement learning distribution network voltage regulation method considering the heterogeneous characteristics of photovoltaics described in this invention, the construction of a training framework based on a multi-agent flexible actor-commentator algorithm includes solving Markov games using the MASAC algorithm. In the MASAC algorithm, each agent contains an actor network and a commentator network; both the actor network and the commentator network are composed of three fully connected neural networks.

[0012] As a preferred embodiment of the reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity described in this invention, the executor network and the critic network include: the executor network, which takes into account the local observation state of the current sub-region, updates it by maximizing the accumulated expected reward and the entropy regularization term, and outputs a local control strategy; and the critic network, which takes into account the global state and joint actions of the agent, updates it by minimizing the Bellman error, and evaluates the value of the joint actions.

[0013] As a preferred embodiment of the reinforcement learning distribution network voltage regulation method considering the heterogeneous characteristics of photovoltaics described in this invention, the mapping to the actual power setpoint of the heterogeneous photovoltaic inverter includes: for photovoltaics with only reactive power controllable, the normalized control command is directly mapped to the reactive power setpoint within the constraint range; for photovoltaics with coordinated active and reactive power controllable, the active power reduction command is first mapped to the active power setpoint, then the remaining reactive power capacity is calculated based on the active power setpoint, and finally the reactive power command is mapped to the reactive power setpoint within the capacity.

[0014] Another objective of this invention is to provide a reinforcement learning distribution network voltage regulation system that considers the heterogeneous characteristics of photovoltaics. This system can model the distributed voltage regulation problem of the distribution network as a Markov game and construct a training framework based on a multi-agent flexible actor-commentator algorithm. It can also use historical running data to perform centralized training of the agents, thus solving the problem that current distribution network voltage regulation methods have difficulties in handling high-dimensional data due to single-agent algorithms.

[0015] As a preferred embodiment of the reinforcement learning-based distribution network voltage regulation system considering the heterogeneous characteristics of photovoltaics described in this invention, the system comprises: a partitioning module, a model building module, an agent training module, and an execution module; the partitioning module is used to construct a similarity matrix based on a comprehensive electrical distance index and divide the distribution network into multiple sub-regions using a nearest neighbor propagation clustering algorithm; the model building module is used to construct three types of heterogeneous photovoltaic inverter models, including uncontrollable, reactive power-only controllable, and active and reactive power coordinated control, based on the active and reactive power regulation characteristics of photovoltaics; the agent training module is used to model the distribution network voltage regulation problem as a Markov game, define the state space, action space, and reward function, and perform centralized training based on the MASAC algorithm using historical operating data; the execution module is used to deploy the trained agents to each sub-region, generate control strategies in real time based on local observation information, and map them to the actual power setpoints of the heterogeneous photovoltaic inverters.

[0016] The beneficial effects of this invention are: This invention provides a reinforcement learning-based distribution network voltage regulation method considering the heterogeneous characteristics of photovoltaics. Based on voltage-active and voltage-reactive sensitivity, an electrical distance index is constructed. The power grid is divided into multiple sub-regions, and a multi-agent distributed control architecture is built. This method accurately quantifies the electrical coupling between nodes, ensuring strong coupling within each sub-region and weak coupling between sub-regions, thereby reducing the complexity of multi-agent collaborative control. Based on the control differences of photovoltaic equipment in the distribution network, three types of heterogeneous photovoltaic inverter models are constructed: uncontrollable, reactive power-only controllable, and active and reactive power coordinated control. This accurately characterizes the operating characteristics of different photovoltaic nodes, avoiding control strategy failure or exceeding limits due to model homogenization, and improving the adaptability of the voltage regulation model to real-world scenarios. Using extracted real-time voltage, photovoltaic, and load data as state features and inverter power adjustment commands as the action space, the voltage smoothing problem is transformed into a Markov game process, and a... Based on a training framework using a multi-agent flexible actor-critic algorithm, this invention employs centralized training of agents using historical operational data. It leverages global information to evaluate the value of joint actions to address voltage coupling issues, relying solely on local observations during execution. This ensures both global coordination and rapid online decision-making. Furthermore, entropy regularization enhances the policy's exploration capabilities and robustness. The trained agents are deployed to various sub-regions, each generating control strategies in real-time based on local observations. Normalized control commands are mapped to the actual power setpoints of heterogeneous photovoltaic inverters, significantly reducing reliance on communication systems and response latency. Pre-defined mapping rules convert normalized control commands into actual power setpoints that satisfy physical constraints, ensuring the executability of control commands. This invention achieves superior results in distributed control architecture, heterogeneous photovoltaic modeling accuracy, and multi-agent system control capabilities. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an overall flowchart of a reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity characteristics provided in Embodiment 1 of the present invention.

[0019] Figure 2 This is a control framework diagram of a reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity characteristics provided in Embodiment 1 of the present invention.

[0020] Figure 3The distribution network partitioning results and photovoltaic distribution maps of various types are provided for a reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity characteristics, as provided in Embodiment 2 of the present invention.

[0021] Figure 4 This is a voltage distribution diagram of a distribution network under the condition of not performing voltage regulation, provided in Embodiment 2 of the present invention, which is a reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity.

[0022] Figure 5 The voltage distribution diagram of the distribution network under the control of this invention is provided in Embodiment 2 of the present invention, which is a reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity. Detailed Implementation

[0023] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0024] Example 1, referring to Figures 1-2 As an embodiment of the present invention, a reinforcement learning-based distribution network voltage regulation method considering photovoltaic heterogeneity is provided, comprising: S1: Based on voltage-active and voltage-reactive sensitivity, construct the electrical distance index I, divide the power grid into multiple sub-regions R, and construct a multi-agent distributed control architecture.

[0025] Specifically, the electrical distance index I is used to characterize the coupling relationship between distribution network nodes, and the nearest neighbor propagation clustering algorithm is used to partition the distribution network based on this index, dividing the distribution network into multiple sub-regions R. Then, each sub-region R is modeled as an independent intelligent agent to implement distributed control and reduce communication costs.

[0026] Furthermore, the power grid is divided into multiple sub-regions R, including voltage-active power sensitivity and voltage-reactive power sensitivity between output nodes. A comprehensive electrical distance index I is obtained by weighting the sub-regions R using weighted coefficients. A similarity matrix is ​​constructed based on the comprehensive electrical distance index I. The nearest neighbor propagation clustering algorithm is used to determine the cluster center by iteratively updating the attraction and affiliation information, and then the sub-regions R are divided.

[0027] The electrical distance index I is used to characterize the coupling relationship between nodes in a distribution network. This invention considers the coordinated regulation of active and reactive power to reduce voltage fluctuations in the distribution network, and therefore constructs an electrical distance index I based on voltage-active and voltage-reactive power sensitivity matrices.

[0028] The electrical distance index I of the voltage-active power sensitivity matrix can be expressed as: , in, The electrical distance index is the voltage-active power sensitivity matrix. For nodes and nodes The active-voltage sensitivity coefficient between the nodes is used to characterize the nodes. Active power injection caused by node Voltage amplitude deviation, For nodes voltage amplitude, For nodes Injected active power.

[0029] Similarly, nodes are defined based on voltage-reactive sensitivity. and nodes The electrical distance between them is expressed as: , in, For the node of voltage-reactive sensitivity and nodes Electrical distance between them For nodes and nodes The reactive-voltage sensitivity coefficient between nodes is used to characterize the nodes. The node caused by reactive power injection Voltage amplitude deviation, For nodes voltage amplitude, For nodes Injected reactive power.

[0030] It should be noted that, considering that both active and reactive power injections affect voltage, a weighting coefficient is introduced to construct a comprehensive electrical distance index I in order to fully reflect the coupling degree between nodes, as expressed as: , in, To comprehensively consider electrical distance indicators, For the node of voltage-reactive sensitivity and nodes Electrical distance between them This is the electrical distance index of the voltage-active power sensitivity matrix.

[0031] Furthermore, based on the calculated comprehensive electrical distance index I, the similarity matrix required for the nearest neighbor propagation algorithm is constructed. The elements of the matrix Represents a node and nodes The similarity between them is specifically defined as the negative value of electrical distance, expressed as: , in, Let be the elements of the matrix.

[0032] diagonal elements Known as the preference value, it is a parameter influenced by the number of subregions R and used to guide the clustering process. The preference value is typically set to the median of all off-diagonal elements in the similarity matrix. During the clustering process, nodes determine cluster centers through bidirectional information exchange.

[0033] First, calculate the attraction information. ,in Represents a node Suitable as a node The degree of cluster centers is updated as follows: , in, For nodes Suitable as a node The degree of cluster centers, For preference values, For attribution information, To remove Other candidate cluster center nodes besides those mentioned above.

[0034] Next, we need to calculate the affiliation information matrix. ,in Represents a node Select node The suitability of its cluster centers is updated as follows: , in, For nodes Select node The suitability of it as a cluster center For nodes Self-attraction For other nodes For nodes Its attractiveness.

[0035] It should also be noted that, in order to prevent numerical oscillations during the iteration process, a damping rule is used to update the attraction and belonging information, specifically expressed as follows: , , in, For iteration Other nodes For nodes The attractiveness This represents the current iteration number. The weighting coefficients for historical information. For iteration Secondary node Select node The suitability of it as a cluster center This is the updated attraction value calculated in this iteration. This is the updated affiliation value calculated in this iteration.

[0036] Through continuous updates and iterations, the optimal cluster center can eventually be determined, that is, the distribution network nodes are divided into different clusters, each cluster is a sub-region R, and each cluster is controlled by an agent.

[0037] S2: Based on the control differences of photovoltaic equipment in the distribution network, construct three types of heterogeneous photovoltaic inverter models 100, including uncontrollable, reactive power-only controllable, and active and reactive power coordinated control.

[0038] Specifically, taking into full account the heterogeneous characteristics of distributed photovoltaic (PV) systems in actual operating scenarios, three types of heterogeneous PV inverter models are constructed based on the active and reactive power regulation characteristics of PV systems: uncontrollable, reactive power-only controllable, and active and reactive power coordinated control. In actual distribution networks, PV inverters at different nodes exhibit significant differences in operating characteristics. To achieve precise voltage regulation, this step defines three PV operating modes and establishes mathematical models for each.

[0039] The three types of heterogeneous photovoltaic inverter models 100 include: an uncontrollable photovoltaic inverter model 101, in which the active power output follows the maximum power point tracking strategy and the reactive power output remains zero; a photovoltaic inverter model 102 in which only reactive power is controllable, the active power is uncontrollable, and the reactive power is adjustable within the range of the inverter's rated apparent power and power factor constraints; and a photovoltaic inverter 103 in which active and reactive power are co-controllable, which has the ability to reduce active power and uses the capacity released after reducing active power to enhance reactive power support capability.

[0040] Furthermore, the first type is the uncontrollable photovoltaic inverter model 101. This type of photovoltaic inverter is in offline mode and does not have the ability to participate in grid voltage regulation. Its active power output fully follows the maximum power point tracking strategy to achieve maximum energy utilization, and it does not provide reactive power support; the reactive power output remains zero. The mathematical model of this mode is expressed as: , , in, For this type of photovoltaic inverter, at any time The active power of the photovoltaic inverter This refers to the active power of the photovoltaic inverter under the maximum power point tracking strategy. For this type of photovoltaic inverter, at any time The reactive power output of the photovoltaic inverter This represents the apparent power of the photovoltaic inverter.

[0041] It should be noted that the second type is photovoltaic inverter model 102, where only reactive power is controllable. This type of photovoltaic inverter operates in a semi-offline mode, with uncontrollable active power, always maintaining operation at the maximum power point. However, it possesses reactive power regulation capability, allowing adjustment of reactive power output within a certain range to support node voltage. Its reactive power regulation range is limited by the inverter's rated apparent power and power factor constraints, specifically expressed in the mathematical model as follows: , , , in, For the second type of photovoltaic inverter at time The active power of the photovoltaic inverter For the second type of photovoltaic inverter at time The reactive power of the photovoltaic inverter It is the minimum power factor.

[0042] Therefore, it can be seen that under this operating mode, the reactive power regulation range of the photovoltaic inverter is affected by both the rated apparent power and the minimum power factor.

[0043] It should also be noted that the third type is the active and reactive power coordinated controllable photovoltaic inverter 103. This type of photovoltaic inverter operates in online mode, possessing the highest control flexibility, allowing for coordinated adjustment of active and reactive power. Its reactive power support capability dynamically and smoothly evolves with the real-time active power reduction. It can not only flexibly adjust active power output (i.e., perform active power reduction) between 0 and maximum available power, but also utilize the capacity released after active power reduction to enhance reactive power support capability. The mathematical model of this mode is expressed as: , , , in, For the third type of photovoltaic inverter at time The active power of the photovoltaic inverter For the third type of photovoltaic inverter at time The reactive power of the photovoltaic inverter.

[0044] S3: The distributed voltage regulation problem of the distribution network is modeled as a Markov game, and a training framework 200 based on the multi-agent flexible actor-commentator algorithm is constructed to centrally train the agents using historical running data.

[0045] Specifically, after dividing the distribution network into multiple sub-regions R and establishing a control model for photovoltaic inverters, the distributed voltage regulation problem of the distribution network needs to be modeled as a Markov game, and a solution framework based on the Multi-Agent Flexible Actor-Critic Algorithm (MASAC) needs to be designed. To solve the distributed voltage collaborative control problem of the distribution network, the three elements of a Markov game are first defined: state space, action space, and reward function. Then, a training process based on the MASAC algorithm is constructed.

[0046] Modeling the distributed voltage regulation problem in a distribution network as a Markov game involves defining the three elements of a Markov game: state space, action space, and reward function.

[0047] The state space includes information on photovoltaic active power output, photovoltaic reactive power output, active power load demand, and reactive power load demand for each node in each sub-region R.

[0048] The set of states of all agents at time t Composition. For the first The local state of the agent corresponding to each subregion R This includes information on photovoltaic power output and load demand for each node in the region, specifically represented as follows: , in, For nodes The photovoltaic system is contributing its power. For nodes The reactive power output of photovoltaic power For nodes Active load, For nodes The reactive load.

[0049] The action space consists of normalized control commands, including reactive power control commands for reactive power controllable photovoltaics, and active power reduction commands and reactive power control commands for active and reactive power co-controllable photovoltaics.

[0050] The set of actions of all agents Composition. Intelligent agent action It consists of normalized control commands for each controllable photovoltaic inverter within its region, and is represented as: , , , , in, This is the reactive power control command for photovoltaic system in Mode 2. This is a power reduction instruction for Mode 3 photovoltaic systems. This is the reactive power control command for Mode 3 photovoltaic systems. For nodes The active power output of photovoltaic power in the second mode, For nodes Photovoltaic power output in the third mode For nodes Photovoltaic reactive power output in the third mode.

[0051] To meet physical constraints, the normalized action values ​​output by the neural network need to be mapped to actual physical quantities.

[0052] The reward function includes features designed to guide the agent to reduce the voltage fluctuation level of the distribution network at each time point. Instant rewards received Defined as the negative of the sum of the absolute values ​​of the voltage deviations of all nodes in this region, it is expressed as: , in, For instant rewards, For intelligent agents, This is the scaling factor. This represents the actual voltage at the node. This is the reference voltage.

[0053] Furthermore, after constructing the Markov game, the MASAC algorithm is used to solve it. MASAC is a multi-agent reinforcement learning algorithm with a "centralized training, distributed execution" framework.

[0054] The construction of a training framework 200 based on a multi-agent flexible actor-critic algorithm includes using the MASAC algorithm to solve Markov games. In the MASAC algorithm, each agent contains an actor network and a critic network. Both the actor network and the critic network are composed of three fully connected neural networks. In the MASAC training framework, the reward function is designed as a negative feedback that is directly linked to the physical deviation of the distribution network voltage. By setting the sum of the absolute values ​​of the voltage deviation as a penalty term, the MASAC algorithm can perceive the risk of the physical operation of the power grid exceeding the limit during the exploration process.

[0055] The executor network and the critic network consist of: the critic network, which takes the global state and joint actions of the agent as input, updates it by minimizing the Bellman error, and evaluates the value of the joint actions; and the executor network, which takes the local observation state of the current sub-region R as input, updates it by maximizing the cumulative expected reward and the entropy regularization term, and outputs a local control policy. During the training phase, it acquires global physical state information and evaluates the physical coupling impact of actions in each region on the voltage distribution of the entire network. This ensures that although the executor network deployed on each hardware terminal relies only on local observation information for decision-making, its output control commands have global coordination at the physical level, achieving the unification of algorithm logic and the spatial coupling characteristics of the distribution network.

[0056] It should be noted that the number of neurons in each layer of the actor network is 256-256-128, and the number of neurons in each layer of the critic network is 256-256-256. The critic network updates its parameters by minimizing the loss function and uses global state information to evaluate the value of actions to solve the voltage coupling problem between sub-networks, as shown below: , , in, For critics' network damage, For the target value, For the first The action value function of an agent. As a discount factor, For the target network parameters, This is the temperature coefficient.

[0057] The executor network is based solely on local observations. Generate actions whose parameters are obtained by maximizing the objective function. To update, it is indicated as: , , in, This is the baseline term, used to reduce training variance. These are network parameters.

[0058] To ensure training stability, the target network parameters are updated using a soft update method, as follows: , , in, These are the parameters of the current network. For the parameters of the target network, To update the coefficients.

[0059] In summary, the model training process is as follows: After initializing the network parameters and the experience replay pool, in each training round, each agent first determines its local state. Select Action The agent then performs an action, and the environment transitions to the next state. And provide feedback and rewards Transfer samples The data is stored in the experience replay pool. When the experience replay pool meets the capacity requirement, a small batch of data is randomly sampled, and the parameters of the critic network, executor network, and target network are updated sequentially according to the above formula. Through repeated iterative training, the model converges, and the optimal collaborative regulation strategy is obtained.

[0060] S4: Deploy the trained agents to each sub-region R. Each agent generates a control strategy in real time based on local observation information and maps it to the actual power setpoint of the heterogeneous photovoltaic inverter.

[0061] Specifically, the mapping to the actual power setpoint of the heterogeneous photovoltaic inverter includes: for photovoltaics with only reactive power controllable, the normalized control command is directly mapped to the reactive power setpoint within the constraint range; for photovoltaics with coordinated active and reactive power controllable, the active power reduction command is first mapped to the active power setpoint, then the remaining reactive power capacity is calculated based on the active power setpoint, and finally the reactive power command is mapped to the reactive power setpoint within the capacity.

[0062] Furthermore, by utilizing the trained multi-agent deep reinforcement learning model, voltage regulation commands are generated in real time based on local observation information from each sub-region R, achieving rapid and precise control of the distribution network voltage. Specifically, after completing offline training of the model, the executor network of each agent is deployed to the corresponding sub-region R controller, entering the online operation phase. During real-time operation, each agent only needs to collect active / reactive load and photovoltaic output data from its own region as local observation data, without needing to communicate with other regions.

[0063] Based on local observations, the intelligent agent rapidly generates normalized control commands for each heterogeneous photovoltaic inverter within the region using the executor network. Subsequently, combining defined heterogeneous photovoltaic model constraints (such as capacity limits and power factor limits), the normalized control commands are mapped to actual active power reduction and reactive power setpoints and distributed to the underlying devices for execution. Through decentralized collaborative adjustment by the R agents in each sub-region, rapid smoothing of grid-wide voltage fluctuations is achieved, maintaining node voltages within a safe range.

[0064] Example 2, refer to Figures 3-5 As an embodiment of the present invention, a reinforcement learning-based distribution network voltage regulation method considering the heterogeneous characteristics of photovoltaics is provided. To verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculations and simulation experiments.

[0065] First, to further verify the effectiveness of the proposed method, simulation experiments were conducted on the IEEE 33-bus standard test system. The MASAC control algorithm was used to regulate the voltage of the IEEE 33-bus distribution network, and the results were compared with those obtained without any voltage regulation to demonstrate the effectiveness of the proposed control method.

[0066] The specific experiment is as follows: First, the electrical distance index I of the IEEE 33-node system is calculated. The distribution network is then partitioned using the nearest neighbor propagation clustering algorithm, with distributed photovoltaic (PV) devices in different modes existing within each sub-partition. Next, each sub-partition is modeled as an agent to regulate the active and reactive power output of PV within the sub-partition, thereby achieving voltage regulation. Following this, based on the MASAC architecture and incorporating historical load and PV output datasets, a multi-agent system is trained to obtain real-time voltage regulation strategies. Finally, the voltage distribution results of the distribution network after MASAC regulation are compared with those without considering voltage regulation strategies to verify the effectiveness of the regulation strategy.

[0067] like Figure 3 As shown, this illustrates the network partitioning results and the distribution of heterogeneous photovoltaic (PV) inverters based on electrical distance index I. It can be seen that the distribution network is divided into three sub-regions R, each containing heterogeneous PV inverters with different control modes (Mode 1, Mode 2, and Mode 3), which aligns with the complex operating environment of the actual power grid. Figure 4 The figure shows the voltage distribution of the distribution network without any voltage control measures. As can be seen from the figure, due to power fluctuations caused by the high proportion of photovoltaic access, the system voltage experienced severe oscillations and serious over-limit phenomena. The voltage amplitude reached a maximum of 1.0875 pu and dropped to a minimum of 0.9134 pu, far exceeding the safe operating range of [0.95, 1.05] pu, greatly threatening the safe and stable operation of the distribution network. Figure 5 The effect of applying the multi-agent reinforcement learning voltage regulation method considering photovoltaic heterogeneity proposed in this invention is shown. Figure 4 In comparison, system voltage fluctuations were significantly suppressed, and the voltage surface became flat and smooth. Throughout the entire test cycle, the node voltage amplitude was strictly controlled between 0.9680 pu and 1.0067 pu, which not only completely eliminated the voltage over-limit problem but also maintained the voltage at an ideal level close to 1.0 pu.

[0068] In summary, the simulation results strongly demonstrate that the method of the present invention can effectively coordinate heterogeneous photovoltaic resources in different sub-regions R, and achieve rapid smoothing and optimized control of the distribution network voltage through precise active and reactive power coordinated regulation.

[0069] Example 3, an embodiment of the present invention, provides a reinforcement learning distribution network voltage regulation system that considers the heterogeneous characteristics of photovoltaics, including a partitioning module, a model building module, an agent training module, and an execution module.

[0070] The partitioning module is used to construct a similarity matrix based on the comprehensive electrical distance index I, and divide the distribution network into multiple sub-regions R through the nearest neighbor propagation clustering algorithm.

[0071] The model building module is used to construct three types of heterogeneous photovoltaic inverter models 100 based on the active and reactive power regulation characteristics of photovoltaics, including uncontrollable, reactive power-only controllable, and active and reactive power coordinated control.

[0072] The agent training module is used to model the voltage regulation problem of the distribution network as a Markov game, define the state space, action space and reward function, and perform centralized training based on the MASAC algorithm using historical running data.

[0073] The execution module is used to deploy the trained agent to each sub-region R, generate control strategies in real time based on local observation information, and map them to the actual power setpoint of the heterogeneous photovoltaic inverter.

[0074] This embodiment also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as proposed in the above embodiments.

[0075] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as proposed in the above embodiments.

[0076] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0078] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0079] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0080] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A reinforcement learning-based distribution network voltage regulation method considering the heterogeneous characteristics of photovoltaics, characterized in that, include: Based on voltage-active and voltage-reactive sensitivity, an electrical distance index (I) is constructed, and the power grid is divided into multiple sub-regions (R) to construct a multi-agent distributed control architecture. Based on the control differences of photovoltaic equipment in the distribution network, three types of heterogeneous photovoltaic inverter models are constructed, including uncontrollable, reactive power controllable only, and active and reactive power coordinated control (100). Using the extracted real-time voltage, photovoltaic and load data as state features and the inverter power regulation command as the action space, the voltage stabilization problem is transformed into a Markov game process, and a training framework based on the multi-agent flexible actor-commentator algorithm is constructed (200). The agents are trained in a centralized manner using historical running data. The trained agents are deployed to each sub-region (R). Each agent generates a control strategy in real time based on local observation information and maps the normalized control commands to the actual power setpoint of the heterogeneous photovoltaic inverter.

2. The reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in claim 1, characterized in that: The division of the power grid into multiple sub-regions (R) includes, The voltage-active power sensitivity and voltage-reactive power sensitivity between output nodes are weighted by weighting coefficients to obtain the comprehensive electrical distance index (I). Based on the comprehensive electrical distance index (I), a similarity matrix is ​​constructed, and the nearest neighbor propagation clustering algorithm is used to determine the cluster center by iteratively updating the attraction and belonging information, and then the sub-regions (R) are divided.

3. The reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in claim 2, characterized in that: The comprehensive electrical distance index (I) is expressed as: ， in, To comprehensively consider electrical distance indicators, For the node of voltage-reactive sensitivity and nodes Electrical distance between them This is the electrical distance index of the voltage-active power sensitivity matrix.

4. The reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in any one of claims 1 to 3, characterized in that: The three types of heterogeneous photovoltaic inverter models (100) include, The uncontrollable photovoltaic inverter model (101) has active power output following the maximum power point tracking strategy and reactive power output remaining at zero. The photovoltaic inverter model (102) with only controllable reactive power has uncontrollable active power, while reactive power is adjustable within the range of the inverter's rated apparent power and power factor constraints. The active and reactive power coordinated controllable photovoltaic inverter (103) has the ability to reduce active power and uses the capacity released after reducing active power to enhance reactive power support capability.

5. The reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in claim 4, characterized in that: The process of transforming the voltage stabilization problem into a Markov game includes... Define the three elements of a Markov game: state space, action space, and reward function. The state space includes information on photovoltaic active power output, photovoltaic reactive power output, active power load demand, and reactive power load demand for each node in each sub-region (R). The action space consists of normalized control commands, including reactive power control commands for reactive power controllable photovoltaics, and active power reduction commands and reactive power control commands for active and reactive power co-controllable photovoltaics.

6. The reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in claim 5, characterized in that: The reward function includes, The aim is to guide the agent to reduce the voltage fluctuation level of the distribution network at each time point. The instantaneous reward obtained by the agent at each time point is represented as follows: ， in, For instant rewards, For intelligent agents, This is the scaling factor. This represents the actual voltage at the node. This is the reference voltage.

7. The reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in any one of claims 1-3 and 5-6, characterized in that: The construction of the training framework (200) based on the multi-agent flexible actor-critic algorithm includes, The MASAC algorithm is used to solve Markov games. In the MASAC algorithm, each agent contains an actor network and a critic network. Both the executor network and the critic network consist of three fully connected neural networks.

8. The reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in claim 7, characterized in that: The executor network and the critic network include, The executor network takes the local observation state of the current sub-region (R) as input, updates it by maximizing the cumulative expected reward and the entropy regularization term, and outputs the local control policy. The critic network takes the global state and joint actions of the agent as input, updates them by minimizing the Bellman error, and evaluates the value of the joint actions.

9. The reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in any one of claims 1-3, 5-6, and 8, characterized in that: The mapping to the actual power setpoint of the heterogeneous photovoltaic inverter includes: For photovoltaic systems with only controllable reactive power, the normalized control command is directly mapped to the reactive power setpoint within the constraint range; For active and reactive power co-controllable photovoltaic systems, the active power reduction command is first mapped to the active power setpoint, then the remaining reactive power capacity is calculated based on the active power setpoint, and finally the reactive power command is mapped to the reactive power setpoint within the capacity.

10. A reinforcement learning distribution network voltage regulation system considering photovoltaic heterogeneity, employing the reinforcement learning distribution network voltage regulation method considering photovoltaic heterogeneity as described in any one of claims 1 to 9, characterized in that: It includes a partitioning module, a model building module, an agent training module, and an execution module; The partitioning module is used to construct a similarity matrix based on the comprehensive electrical distance index (I) and divide the distribution network into multiple sub-regions (R) through the nearest neighbor propagation clustering algorithm. The model building module is used to construct three types of heterogeneous photovoltaic inverter models (100) based on the photovoltaic active and reactive power regulation characteristics, including uncontrollable, reactive power-only controllable, and active and reactive power coordinated control. The intelligent agent training module is used to model the voltage regulation problem of the distribution network as a Markov game, define the state space, action space and reward function, and perform centralized training based on the MASAC algorithm using historical running data. The execution module is used to deploy the trained agent to each sub-region (R), generate control strategies in real time based on local observation information, and map them to the actual power setpoint of the heterogeneous photovoltaic inverter.

Images