Harmonic suppression method and system for power distribution network

Abstract

The invention discloses a power distribution network harmonic suppression method and system, and the method comprises the steps: building an objective function with the minimum total harmonic distortion rate of nodes of a whole network, building a power flow constraint, a harmonic distortion rate constraint and an active filter device related constraint, and obtaining a power distribution network harmonic suppression model according to the objective function and constraint conditions. A distributed partially observable Markov decision process is established based on the MADDPG algorithm, and harmonic suppression is performed on the power distribution network based on the distributed partially observable Markov decision process by using the PRE-combined MADDPG algorithm, so that the problem of low sample efficiency of a reinforcement learning algorithm is effectively solved, the stability of a training process and the convergence speed of the algorithm are promoted, and the robustness of the power distribution network is improved. According to the method, the effectiveness of the finally obtained harmonic suppression strategy is ensured, based on a centralized training-distributed execution architecture, it can be ensured that a plurality of active filter devices can make a cooperative control strategy giving consideration to the harmonic problem of the whole power distribution network in real time only by means of distributed observation, and therefore efficient whole network harmonic cooperative governance is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of power distribution networks, and in particular to a method and system for harmonic control in power distribution networks. Background Art

[0002] With the widespread access of distributed photovoltaics and the rapid improvement of the level of power electronics of users, the source-load interaction mechanism of the distribution network has become more and more complex, and the dynamic change degree of grid harmonics has gradually increased. The superposition of a large number of micro-harmonic sources has made the harmonic pollution of modern distribution networks present the characteristics of high density, decentralization and full network. In some distribution networks with severe harmonic problems, the production process containing sensitive equipment cannot operate normally, and distributed photovoltaics are automatically disconnected from the grid, causing large-scale economic losses, which will have an adverse impact on the safe and economic operation of the distribution network.

[0003] The number of harmonic sources in traditional power grids is small and highly concentrated, generally concentrated in a few special users. Direct governance on the user side can control the harmonic level of the entire network. This traditional governance method is the point-to-point governance mode, which widely adopts the method of detecting the harmonic current of the harmonic source load feeder and controlling the active power filter (APF) to inject equal-amplitude and anti-phase harmonics to resist. However, the new power system shows a significant trend of power electronics, and the harmonic pollution of the power grid shows the characteristics of high density, decentralization, and full network. It is not feasible to implement point-to-point governance for a large number of micro-harmonic sources. Therefore, it is necessary to study a control strategy that can coordinate multiple APF devices in the network to achieve coordinated governance of harmonic problems in the entire network. Since the power grid harmonics are in real-time dynamic changes, to achieve accurate and effective governance of power grid harmonics, the coordinated control strategy of the APF device needs to meet the real-time control requirements. At present, the method based on centralized control relies on accurate mathematical models, and the solution speed is difficult to meet the actual engineering needs. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for harmonic control in a distribution network to achieve efficient coordinated control of harmonics in the entire network.

[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is:

[0006] A method for harmonic control in a distribution network comprises the following steps:

[0007] The objective function is established by minimizing the total harmonic distortion rate of all network nodes, and the power flow constraints, harmonic distortion rate constraints and active filter device related constraints are established;

[0008] Obtaining a distribution network harmonic control model according to the objective function, the power flow constraint, the harmonic distortion rate constraint and the active filter device related constraints;

[0009] Establish a decentralized partially observable Markov decision process based on the above-mentioned distribution network harmonic governance model;

[0010] Use the multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process.

[0011] To solve the above technical problems, another technical solution adopted by the present invention is:

[0012] A distribution network harmonic governance system includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0013] Establish an objective function with the minimum total harmonic distortion rate of all network nodes, and establish power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices;

[0014] Obtain a distribution network harmonic governance model according to the objective function, the power flow constraints, the harmonic distortion rate constraints, and the constraints related to the active filter devices;

[0015] Establish a decentralized partially observable Markov decision process based on the distribution network harmonic governance model;

[0016] Use the multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process.

[0017] The beneficial effects of the present invention are as follows: An objective function is established with the minimum total harmonic distortion rate of all network nodes, and power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices are established. A distribution network harmonic governance model is obtained according to the objective function and the constraint conditions. A decentralized partially observable Markov decision process is established based on the distribution network harmonic governance model. The multi-agent deep deterministic policy gradient (MADDPG) algorithm combined with prioritized experience replay (PRE) is used to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process. In this way, by introducing the PRE technology, the problem of low sample efficiency of the reinforcement learning algorithm is effectively solved, the stability of the training process and the algorithm convergence speed are promoted, and the effectiveness of the finally obtained harmonic governance strategy is guaranteed. The MADDPG algorithm is based on a centralized training-decentralized execution architecture, which can ensure that multiple active filter devices can make collaborative control strategies that take into account the harmonic problems of the entire distribution network only relying on decentralized observations, thus realizing efficient harmonic collaborative governance of the entire network. Description of the Drawings

[0018] Figure 1It is a flowchart of the steps of a method for harmonic governance of a distribution network according to an embodiment of the present invention;

[0019] Figure 2 It is a schematic structural diagram of a harmonic governance system of a distribution network according to an embodiment of the present invention;

[0020] Figure 3 It is a governance flowchart in the method for harmonic governance of a distribution network according to an embodiment of the present invention;

[0021] Figure 4 It is a schematic structural diagram of an action network and an evaluation network improved based on D2RL in the method for harmonic governance of a distribution network according to an embodiment of the present invention. Specific implementation manners

[0022] To describe in detail the technical content, achieved objectives and effects of the present invention, the following is described in conjunction with the implementation manners and with reference to the accompanying drawings.

[0023] Please refer to Figure 1 , a method for harmonic governance of a distribution network, including the steps:

[0024] Establish an objective function with the minimum total harmonic distortion rate of all network nodes, and establish power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices;

[0025] Obtain a harmonic governance model of the distribution network according to the objective function, the power flow constraints, the harmonic distortion rate constraints, and the constraints related to the active filter device;

[0026] Establish a decentralized partially observable Markov decision process based on the harmonic governance model of the distribution network;

[0027] Use a multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process.

[0028] As can be seen from the above description, the beneficial effects of the present invention are as follows: The objective function is established with the minimum total harmonic distortion rate of all network nodes, and power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices are established. According to the objective function and constraint conditions, a harmonic governance model for the distribution network is obtained. Based on the harmonic governance model of the distribution network, a decentralized partially observable Markov decision process is established. The multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay is used to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process. In this way, by introducing the PRE technology, the problem of low sample efficiency of the reinforcement learning algorithm is effectively solved, the stability of the training process and the algorithm convergence speed are promoted, and the effectiveness of the finally obtained harmonic governance strategy is guaranteed. The MADDPG algorithm is based on a centralized training-decentralized execution architecture, which can ensure that multiple active filter devices can make collaborative control strategies that take into account the harmonic problems of the entire distribution network only relying on decentralized observations, thus realizing efficient harmonic collaborative governance of the entire network.

[0029] Further, the establishment of the objective function with the minimum total harmonic distortion rate of all network nodes includes:

[0030]

[0031] In the formula, f represents the objective function, T represents the set of optimization periods, I represents the set of all nodes in the distribution network, A represents phase A, B represents phase B, C represents phase C, represents the total harmonic distortion rate of phase φ of node i at time t, H represents the maximum harmonic frequency considered, represents the h-th harmonic voltage of phase φ of node i at time t, represents the fundamental voltage of the h-th harmonic of phase φ of node i at time t.

[0032] As can be seen from the above description, establishing the objective function with the minimum total harmonic distortion rate of all network nodes can effectively coordinate the APF equipment to achieve collaborative governance of complex and dispersed harmonic problems in the distribution network.

[0033] Further, the power flow constraint is specifically:

[0034]

[0035] In the formula, represents the active power of phase φ of node i at time t, represents the reactive power of phase φ of node i at time t, represents the voltage of phase φ of node i at time t, represents the voltage of phase φ' of node j at time t, represents the line conductance between phase φ and phase φ' between node i and node j, Denote the voltage phase angle difference between phase φ and phase φ' between node i and node j at time t. Denote the line susceptance between phase φ and phase φ' between node i and node j. Denote the h - th order injected harmonic current of phase φ of node i at time t. Denote the h - th order harmonic coupling admittance between phase φ and phase φ' between node i and node j at time t. Denote the h - th order harmonic voltage of phase φ' of node i at time t;

[0036] The harmonic distortion rate constraint is specifically as follows:

[0037]

[0038] In the formula, Denote the maximum allowable value of the total harmonic distortion rate of phase φ of node i at time t;

[0039] The constraints related to the active power filter device are specifically as follows:

[0040]

[0041] In the formula, Denote the harmonic voltage value of phase φ of node i equipped with an active power filter device at time t, Denote the harmonic current value of phase φ of node i equipped with an active power filter device at time t, Denote the capacity of the active power filter device, Denote the lower limit of the output harmonic current value of the active power filter device, Denote the upper limit of the output harmonic current value of the active power filter device.

[0042] From the above description, it can be seen that establishing power flow constraints ensures the safe operation of the power system and optimizes the economic operation of the power system. Establishing harmonic distortion rate constraints can ensure that the total harmonic distortion rate of all nodes in the network is within a reasonable range during the regulation process. The APF device realizes the regulation of harmonic distortion problems by grid - connected output of harmonic current. The present invention believes that all the capacity of the APF device is used for harmonic governance, so the constraints related to the active power filter device are established.

[0043] Furthermore, the establishment of a decentralized partially observable Markov decision process based on the distribution network harmonic governance model includes:

[0044] Regarding each active power filter device as an agent, and regarding the distribution network as the environment for the joint actions of the agents;

[0045] Determine the number of the agents;

[0046] Determine the state space according to all possible states in the environment;

[0047] Determine the joint observation space according to the observation spaces of the respective agents;

[0048] Determine the joint action space according to the action spaces of the respective agents;

[0049] Determine the reward function according to the objective function, the harmonic distortion rate constraint, and the constraints related to the active power filter device;

[0050] Determine the state transition function;

[0051] Obtain a tuple according to the number of agents, the state space, the joint observation space, the joint action space, the reward function, and the state transition function, and use the tuple to represent the decentralized partially observable Markov decision process.

[0052] As can be seen from the above description, in order to effectively solve the problem of coordinated regulation of harmonics, the dynamic interaction process between the APF device and the distribution network is constructed as a decentralized partially observable Markov decision process. The agents embedded in each APF device can only observe the information of the local node, that is, the sampling information of the APF device itself or the relevant information processed by the internal module of the device, which meets the strict decentralized execution architecture and does not require any additional communication. The decentralized partially observable Markov decision process can be represented by a tuple composed of the number of agents, the state space, the joint observation space, the joint action space, the reward function, and the state transition function, which can better adapt to complex environments, realize the cooperative optimization of multiple agents, and improve the dynamic adaptability and decision-making efficiency of the system.

[0053] Furthermore, the use of the multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process includes:

[0054] Initialize the parameters of the evaluation network, the parameters of the action network, and the experience replay pool. The evaluation network and the action network are the evaluation network and the action network improved based on the deep dense reinforcement learning architecture;

[0055] Set the number of training rounds to 0;

[0056] Judge whether the number of training rounds reaches the first preset value. If so, output the parameters of the trained action network. If not, each agent obtains the initial state value of the environment, initializes the random noise, and sets the time step to 0;

[0057] Determine whether the time step reaches a second preset value. If so, after incrementing the number of training rounds by one, return to execute the step of determining whether the number of training rounds reaches a first preset value. If not, each agent selects a corresponding action according to the observation value and the action network, and the agent executes the action with the added random noise;

[0058] Calculate the reward value after the action and obtain the observation value of the next time step;

[0059] Store the state value, action value, reward value, and state value of the next time step of each agent at the current time step into the experience replay pool;

[0060] Use the prioritized experience replay technique to extract a batch of state transition samples from the experience replay pool for updating the network parameters;

[0061] Calculate the loss function of the evaluation network of each agent, and optimize and update the parameters of the evaluation estimation network by minimizing the loss function;

[0062] Calculate the loss function of the action network of each agent, and update the parameters of the action estimation network using gradient backpropagation;

[0063] Update the parameters of the evaluation target network and the parameters of the action target network;

[0064] After incrementing the time step by one, return to execute the step of determining whether the time step reaches the second preset value;

[0065] Load the parameters of the trained action network into the active power filter device, obtain the observation value of the local node of the current power system, and execute an action according to the observation value and the trained action network, and assign the optimal action value to the active power filter device to complete harmonic suppression.

[0066] As can be seen from the above description, the MADDPG algorithm adopts the Actor-Critic framework form in the network architecture, that is, each agent has two types of networks. One is the evaluation (Critic) network that requires global information, and the other is the action (Actor) network that only requires local observations. The Critic network and Actor network of the MADDPG algorithm are improved using D2RL (Deep Dense Reinforcement Learning), which effectively improves the parameterization level of the Critic network and Actor network, thereby significantly enhancing the performance of the MADDPG algorithm in complex environments, and further improving the effectiveness of harmonic suppression.

[0067] Please refer to Figure 2, A harmonic governance system for a distribution network, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0068] Establish an objective function with the minimum total harmonic distortion rate of all network nodes, and establish power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices;

[0069] Obtain a harmonic governance model for the distribution network according to the objective function, the power flow constraints, the harmonic distortion rate constraints, and the constraints related to the active filter devices;

[0070] Establish a decentralized partially observable Markov decision process based on the harmonic governance model of the distribution network;

[0071] Use the multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process.

[0072] From the above description, the beneficial effects of the present invention are as follows: An objective function is established with the minimum total harmonic distortion rate of all network nodes, and power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices are established. A harmonic governance model for the distribution network is obtained according to the objective function and the constraint conditions. A decentralized partially observable Markov decision process is established based on the harmonic governance model of the distribution network. The multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay is used to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process. In this way, by introducing the PRE technology, the problem of low sample efficiency of the reinforcement learning algorithm is effectively solved, the stability of the training process and the algorithm convergence speed are promoted, and the effectiveness of the finally obtained harmonic governance strategy is ensured. The MADDPG algorithm is based on a centralized training - decentralized execution architecture, which can ensure that multiple active filter devices can make a collaborative control strategy that takes into account the harmonic problems of the entire distribution network only relying on decentralized observations, thus realizing efficient harmonic collaborative governance of the entire network.

[0073] Further, the establishment of the objective function with the minimum total harmonic distortion rate of all network nodes includes:

[0074]

[0075] In the formula, f represents the objective function, T represents the set of optimization periods, I represents the set of all nodes in the distribution network, A represents phase A, B represents phase B, C represents phase C, represents the total harmonic distortion rate of phase φ of node i at time t, H represents the maximum harmonic frequency considered, represents the h-th harmonic voltage of phase φ of node i at time t, It represents the h-th fundamental voltage of phase φ at node i at time t.

[0076] As can be seen from the above description, by establishing the objective function with the minimum total harmonic distortion rate of all network nodes, it can effectively coordinate the APF devices to achieve the collaborative governance of complex and scattered harmonic problems in the distribution network.

[0077] Furthermore, the power flow constraint is specifically:

[0078]

[0079] In the formula, It represents the active power of phase φ at node i at time t, It represents the reactive power of phase φ at node i at time t, It represents the voltage of phase φ at node i at time t, It represents the voltage of phase φ' at node j at time t, It represents the line conductance between phase φ and phase φ' between node i and node j, It represents the voltage phase angle difference between phase φ and phase φ' between node i and node j at time t, It represents the line susceptance between phase φ and phase φ' between node i and node j, It represents the h-th injected harmonic current of phase φ at node i at time t, It represents the h-th harmonic coupling admittance between phase φ and phase φ' between node i and node j at time t, It represents the h-th harmonic voltage of phase φ' at node i at time t;

[0080] The harmonic distortion rate constraint is specifically:

[0081]

[0082] In the formula, It represents the maximum allowable value of the total harmonic distortion rate of phase φ at node i at time t;

[0083] The relevant constraints of the active filter device are specifically:

[0084]

[0085] In the formula, It represents the harmonic voltage value of phase φ at node i equipped with the active filter device at time t, It represents the harmonic current value of phase φ at node i equipped with the active filter device at time t, It represents the capacity of the active filter device, It represents the lower limit of the output harmonic current value of the active filter device, It represents the upper limit of the output harmonic current value of the active filter device.

[0086] From the above description, it can be seen that establishing power flow constraints ensures the safe operation of the power system and optimizes the economic operation of the power system. Establishing harmonic distortion rate constraints can ensure that the total harmonic distortion rate of all network nodes is within a reasonable range during the regulation process. The APF device achieves regulation of harmonic distortion problems by outputting harmonic current through the grid. The present invention believes that all the capacity of the APF device is used for harmonic control, and therefore establishes relevant constraints on the active filter device.

[0087] Furthermore, the decentralized partially observable Markov decision process based on the distribution network harmonic governance model includes:

[0088] Each active filter device is regarded as an intelligent agent, and the distribution network is regarded as the environment for the joint action of the intelligent agents;

[0089] determining the number of said agents;

[0090] determining a state space based on all possible states in the environment;

[0091] Determining a joint observation space based on the observation spaces of each of the agents;

[0092] Determining a joint action space based on the action spaces of each of the agents;

[0093] Determining a reward function according to the objective function, the harmonic distortion rate constraint and the active filter device related constraints;

[0094] Determine the state transition function;

[0095] A multi-tuple is obtained according to the number of the agents, the state space, the joint observation space, the joint action space, the reward function and the state transfer function, and the multi-tuple is used to represent a decentralized partially observable Markov decision process.

[0096] From the above description, it can be seen that in order to effectively solve the problem of coordinated regulation of harmonics, the dynamic interaction process between the APF equipment and the distribution network is constructed as a distributed partially observable Markov decision process. The intelligent agent embedded in each APF device can only observe the information of the local node, that is, the sampling information of the APF device itself or the related information processed by the internal module of the device, which meets the strict distributed execution architecture. Without any additional communication, the distributed partially observable Markov decision process can be represented by a multi-tuple consisting of the number of intelligent agents, state space, joint observation space, joint action space, reward function and state transfer function, which can better adapt to complex environments, realize the coordinated optimization of multiple agents, and improve the dynamic adaptability and decision-making efficiency of the system.

[0097] Further, the multi-agent deep deterministic policy gradient algorithm using prioritized experience replay for harmonic control of the distribution network based on the decentralized partially observable Markov decision process includes:

[0098] Initialize the parameters of the evaluation network, the parameters of the action network, and the experience replay pool. The evaluation network and the action network are the evaluation network and the action network improved based on the deep dense reinforcement learning architecture;

[0099] Set the number of training episodes to 0;

[0100] Determine whether the number of training episodes reaches a first preset value. If so, output the parameters of the trained action network. If not, each agent obtains the initial state value of the environment, initializes random noise, and sets the time step to 0;

[0101] Determine whether the time step reaches a second preset value. If so, increment the number of training episodes by one and return to execute the step of determining whether the number of training episodes reaches the first preset value. If not, each agent selects a corresponding action according to the observation value and the action network, and the agent executes the action with the added random noise;

[0102] Calculate the reward value after the action and obtain the observation value of the next time step;

[0103] Store the state value, action value, reward value, and state value of the next time step of each agent at the current time step in the experience replay pool;

[0104] Use the prioritized experience replay technique to sample a batch of state transition samples from the experience replay pool for updating the network parameters;

[0105] Calculate the loss function of the evaluation network of each agent, and optimize and update the parameters of the evaluation estimation network by minimizing the loss function;

[0106] Calculate the loss function of the action network of each agent, and update the parameters of the action estimation network using gradient backpropagation;

[0107] Update the parameters of the evaluation target network and the parameters of the action target network;

[0108] Increment the time step by one and return to execute the step of determining whether the time step reaches the second preset value;

[0109] Load the parameters of the trained action network into the active power filter device, obtain the observation value of the local node of the current power system, execute an action according to the observation value and the trained action network, and assign the optimal action value to the active power filter device to complete harmonic control.

[0110] As can be seen from the above description, the MADDPG algorithm adopts the Actor-Critic framework in the network architecture, that is, each agent has two types of networks. One is the evaluation (Critic) network that requires global information, and the other is the action (Actor) network that only requires local observations. The Critic network and Actor network of the MADDPG algorithm are improved using D2RL (Deep Dense Reinforcement Learning), which effectively improves the parameterization level of the Critic network and Actor network, thereby significantly enhancing the performance of the MADDPG algorithm in complex environments and further improving the effectiveness of harmonic governance.

[0111] The above method and system for harmonic governance of a distribution network according to the present invention can be applied to the scenario of harmonic governance of a distribution network, which will be described below through specific embodiments:

[0112] Please refer to Figure 1 、 Figure 3 and Figure 4 , the first embodiment of the present invention is:

[0113] A method for harmonic governance of a distribution network, comprising the steps of:

[0114] S1. Establish an objective function with the minimum total harmonic distortion rate of all network nodes, and establish power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices.

[0115] Specifically, collect historical load data of the distribution network, historical monitoring data of harmonic sources, and power quality monitoring data. Analyze the main harmonic problems existing in the distribution network according to the historical load data of the distribution network, the historical monitoring data of harmonic sources, and the power quality monitoring data to determine the governance objectives of the distribution network harmonics; obtain the topology and line parameters of the distribution network, and construct a distribution network model according to the topology and line parameters; establish an objective function with the minimum total harmonic distortion rate of all network nodes according to the governance objectives and the distribution network model, and establish power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices.

[0116] Among them, establishing the objective function with the minimum total harmonic distortion rate of all network nodes includes:

[0117]

[0118] In the formula, f represents the objective function, T represents the set of optimization periods, I represents the set of all nodes in the distribution network, A represents phase A, B represents phase B, C represents phase C, represents the total harmonic distortion rate of phase φ of node i at time t, H represents the maximum harmonic frequency considered, represents the h - th harmonic voltage of phase φ at node i at time t, represents the fundamental voltage of phase φ at node i at time t.

[0119] The power flow constraints include fundamental power flow constraints and harmonic power flow constraints, specifically:

[0120]

[0121] In the formula, represents the active power of phase φ at node i at time t, represents the reactive power of phase φ at node i at time t, represents the voltage of phase φ at node i at time t, represents the voltage of phase φ' at node j at time t, represents the line conductance between phase φ and phase φ' between node i and node j, represents the voltage phase angle difference between phase φ and phase φ' between node i and node j at time t, represents the line susceptance between phase φ and phase φ' between node i and node j, represents the h - th injected harmonic current of phase φ at node i at time t, represents the h - th harmonic coupling admittance between phase φ and phase φ' between node i and node j at time t, represents the h - th harmonic voltage of phase φ' at node i at time t;

[0122] The harmonic distortion rate constraint is specifically:

[0123]

[0124] In the formula, represents the maximum allowable value of the total harmonic distortion rate of phase φ at node i at time t;

[0125] The constraints related to the active power filter device are specifically:

[0126]

[0127] In the formula, represents the harmonic voltage value of phase φ at node i equipped with an active power filter device at time t, represents the harmonic current value of phase φ at node i equipped with an active power filter device at time t, represents the capacity of the active power filter device, represents the lower limit of the harmonic current value output by the active power filter device, represents the upper limit of the harmonic current value output by the active power filter device.

[0128] S2. Obtain a harmonic governance model for the distribution network according to the objective function, the power flow constraint, the harmonic distortion rate constraint, and the relevant constraints of the active filter device.

[0129] S3. Establish a decentralized partially observable Markov decision process based on the harmonic governance model for the distribution network, specifically including S3.1 - S3.8:

[0130] S3.1. Take each active filter device as an agent, and take the distribution network as the environment for the joint actions of the agents.

[0131] S3.2. Determine the number N of the agents.

[0132] S3.3. Determine the state space according to all possible states in the environment

[0133] Among them, for any state Satisfy the following formula:

[0134]

[0135] In the formula, V t,f Represents the voltage matrix composed of the three-phase fundamental voltage values of each node at time t, θ t,f Represents the phase angle matrix corresponding to the voltage matrix composed of the three-phase fundamental voltage values of each node at time t, V t,h Represents the voltage matrix composed of the three-phase harmonic voltage values of each node at time t, θ t,h Represents the phase angle matrix corresponding to the voltage matrix composed of the three-phase harmonic voltage values of each node at time t, I t,h Represents the three-phase injected harmonic current of each node at time t, Represents the phase angle matrix corresponding to the three-phase injected harmonic current of each node at time t.

[0136] S3.4. Determine the joint observation space according to the observation space of each agent

[0137] The joint observation space is jointly composed of the observation spaces of each agent, and is a partial description of the multi-agent environment. Under the decentralized control framework, the observation space of the agent located at node i is only composed of local information, and its observed value o at time t i,t Satisfies Specifically:

[0138]

[0139] In the formula, Represents the fundamental voltage of node i equipped with an active filter device at time t, denotes the phase of the fundamental voltage at node i equipped with an active filter device at time t, denotes the harmonic voltage at node i equipped with an active filter device at time t, denotes the phase of the harmonic voltage at node i equipped with an active filter device at time t, a i,t-1 denotes the action value of the active filter device at node i at time t - 1, which can increase the effectiveness of the decentralized control action. The number of observed harmonic voltages needs to be determined according to specific harmonic management requirements.

[0140] S3.5. Determine the joint action space according to the action spaces of the respective agents

[0141] The joint action space is jointly constituted by the action spaces of the respective agents. All agents executing the joint action can achieve the compensation of the harmonics in the distribution network. The action spaces of the agents located at node i are independent of each other, and its observed value a at time t i,t satisfies Specifically:

[0142]

[0143] In the formula, denotes the three-phase injected harmonic current of the active filter device at node i at time t, denotes the phase angle command of the three-phase injected harmonic current of the active filter device at node i at time t.

[0144] S3.6. Determine the reward function according to the objective function, the harmonic distortion rate constraint, and the constraints related to the active filter device

[0145] The problem of harmonic management in the distribution network proposed in the present invention is established as a fully cooperative multi-agent problem. Therefore, all agents share the same reward. The main objective of the present invention is to reduce the total harmonic distortion rate of all nodes in the network. The reward function constructed based on the objective function, the harmonic distortion rate constraint, and the constraints related to the active filter device is specifically:

[0146]

[0147] In the formula, r t denotes the immediate reward, F p denotes the penalty function. When the agent violates the harmonic distortion rate constraint and the constraints related to the active filter device after executing the joint action, a large penalty value is imposed on the agent.

[0148] S3.7. Determine the state transition function

[0149] Among them, the state transition function is specifically: for the state s at any time t t , under the action of the joint action of the agents, the environment transfers to the next state s t+1 with a probability, that is a t represents the joint action of the agents.

[0150] S3.8. Obtain a tuple according to the number of the agents, the state space, the joint observation space, the joint action space, the reward function, and the state transition function and use the tuple to represent the decentralized partially observable Markov decision process.

[0151] S4. Use the multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic control on the distribution network based on the decentralized partially observable Markov decision process, as Figure 3 shown, specifically including S4.1 - S4.12:

[0152] Among them, the MADDPG algorithm is a classic multi-agent reinforcement learning algorithm. It adopts the method of centralized training and decentralized execution, which can effectively improve the stability of the multi-agent system and can be applied to multi-agent environments of cooperation and hybrid cooperation-competition. In the MADDPG algorithm, the Actor-Critic framework form is adopted in the network architecture. Each agent has two types of networks, namely the evaluation network and the action network. Both the evaluation network and the action network include an estimation network and a target network. The parameters of the action estimation network and the evaluation estimation network of the i-th agent are respectively denoted as and The parameters of the action target network and the evaluation target network are respectively denoted as and The evaluation network evaluates the value of the joint state and action according to the global information during the centralized training process, fits the joint state-action value function, and simultaneously corrects the policy of the action network; the trained action network gives the corresponding optimal action according to the local observation during the decentralized execution process. Therefore, the essence of the MADDPG algorithm is the parameter update process of the neural network to obtain a neural network that can achieve the control goal.

[0153] The magnitude of the TD-error (the difference between the predicted value and the actual value of the current state) is used as the basis for measuring the importance of state transition samples, reflecting the potential learning value that the agent can obtain from the current state transition samples. Specifically, when the TD-error is positive and large in magnitude, it indicates that this state transition sample may contain information of significant value for policy optimization, and the agent should prioritize learning such state transition samples to accelerate policy improvement; when the TD-error is negative and large in magnitude, it means that the agent may have taken a suboptimal or incorrect action policy, and such state transition samples need to be focused on to correct the policy deviation. Since the TD-error is vulnerable to noise interference, resulting in estimation errors, directly relying on its original value may lead to unstable training. To improve the sampling efficiency of the experience replay mechanism and promote the convergence of reinforcement learning methods, the present invention uses the prioritized experience replay technique to improve the MADDPG algorithm.

[0154] S4.1. Initialize the parameters of the evaluation network, the parameters of the action network, and the experience replay pool. The evaluation network and the action network are the evaluation network and the action network improved based on the deep dense reinforcement learning architecture (D2RL). As Figure 4 shown, it can effectively improve the parameterization level of the Critic network and the Actor network.

[0155] S4.2. Set the number of training episodes to 0.

[0156] S4.3. Determine whether the number of training episodes has reached the first preset value. If so, output the parameters of the trained action network. If not, each agent obtains the initial state value of the environment, initializes the random noise, and sets the time step to 0.

[0157] S4.4. Determine whether the time step has reached the second preset value. If so, increment the number of training episodes by 1 and return to execute S4.3. If not, each agent selects the corresponding action according to the observation value and the action network, and the agent executes the action with the added random noise.

[0158] Among them, the action of the agent is obtained by the action estimation network. During the training process, to ensure the agent's exploration ability of the action and state space, noise is added to the deterministic action output by the network. Therefore, the action a i,t with the added random noise is specifically:

[0159]

[0160] In the formula, represents the policy output by the action estimation network according to the current action o i,t and represents a normal distribution with a mean of 0 and a variance of σ2 Gaussian noise

[0161] S4.5. Calculate the reward value after the action and obtain the observation value at the next time step.

[0162] S4.6. Store the state value, action value, reward value of each agent at the current time step and the state value at the next time step into the experience replay pool.

[0163] S4.7. Use the prioritized experience replay technique to sample a batch of state transition samples from the experience replay pool for updating the network parameters.

[0164] Among them, the state transition sample is:

[0165]

[0166] In the formula, o t =[o 1,t , o 2,t , …, o N,t represents the observation vector of all agents on the current environmental state, a t =[a 1,t , a 2,t , …, a N,t represents the joint action value of all agents on the current environmental state, r t represents the reward obtained by all agents after executing the action, o t+1 =[o 1,t+1 , o 2,t+1 , …, o N,t+1 represents the observation vector of all agents on the next state of the environment.

[0167] In the PER technique, the probability P i that a state transition sample is drawn is determined according to the absolute value of the TD-error, specifically:

[0168]

[0169] In the formula, β1 represents the weight for controlling the use of priority, p i = 1 / rank i represents the priority assigned to the state transition sample i, and rank i represents the ranking of the i-th state transition sample among all state transition samples.

[0170] The PER mechanism can optimize the learning process in the following two aspects by dynamically adjusting the priority of experience sampling: (1) Efficiently utilize high-value experiences: Replay positive experiences with significant TD-errors preferentially to accelerate the convergence of the policy towards the optimal direction; (2) Robustly handle negative experiences: Reasonably allocate the learning weights for negative TD-error experiences to avoid the degradation of the policy due to noise or outliers.

[0171] S4.8. Calculate the loss function of the evaluation network of each of the agents, and optimize and update the parameters of the evaluation estimation network by minimizing the loss function.

[0172] Among them, the loss function of the i-th agent The calculation is specifically as follows:

[0173]

[0174] In the formula, E[·] represents the expected value, y t represents the Q value of the evaluation target network of the agent, represents the joint state-action value at the current moment calculated by the evaluation estimation network of the i-th agent, γ represents the discount factor, which can generally be set to 0.99, represents the joint state-action value at the next moment calculated by the evaluation target network of the i-th agent, and the joint action a at the next moment t+1 is obtained by the action target network of the agent according to the observation at the next moment.

[0175] S4.9. Calculate the loss function of the action network of each of the agents, and update the parameters of the action estimation network using gradient backpropagation.

[0176] Among them, the gradient calculation of the i-th agent is specifically as follows:

[0177]

[0178] In the formula, represents the gradient ascent of the policy parameters of the i-th agent to optimize the objective function J, represents the gradient of the policy of the i-th agent with respect to its parameters i,t under the condition of the given observation o , represents the gradient of the Q function of the i-th agent with respect to the action a t under the condition of the given global observation o t and the action a t , represents the action selected by the i-th agent according to its policy π and the local observation o i,t at time t.

[0179] S4.10. Update the parameters of the evaluation target network and the parameters of the action target network.

[0180] Specifically, in order to stabilize the parameter update process of the agent target network, the parameters of the action target network and the evaluation target network of agent i both adopt the soft update method, specifically as follows:

[0181]

[0182] In the formula, τ represents the soft update coefficient, and τ << 1, and it can take the value of 0.01.

[0183] S4.11. After adding one to the time step, return to execute S4.4.

[0184] S4.12. Load the parameters of the trained action network into the active power filter device, obtain the observation values of the local nodes of the current power system, and execute actions according to the observation values and the trained action network, and assign the optimal action value to the active power filter device to complete harmonic suppression.

[0185] The action network only needs to perform a simple neural network forward propagation calculation to quickly generate a suitable harmonic suppression strategy online. Each APF device injects corresponding harmonic currents for compensation according to this strategy, and thus decentralized harmonic suppression considering the harmonic level of the entire network can be achieved. Among them, the observation values of the local nodes are the fundamental wave voltage and its phase, harmonic voltage and phase of the local nodes. The number of harmonic voltages observed is considered during the centralized training of the agent, and the number of harmonic voltages observed during decentralized execution is the same as that during centralized training.

[0186] In the centralized training stage of the present invention, a reward function is formulated with the goal of ensuring the solution of the harmonic problems of the entire network. The agents share state information and the reward function, and adopt a collaborative strategy to obtain the maximum reward; in the decentralized execution stage, only the trained action network needs to be deployed on each decentralized APF device in the distribution network, and the APF device can make a governance strategy considering the harmonic level of the entire distribution network in real time according to the local node observation information.

[0187] Please refer to Figure 2 , and the second embodiment of the present invention is:

[0188] A distribution network harmonic governance system includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it realizes each step in the distribution network harmonic governance method in the first embodiment.

[0189] In summary, the present invention provides a method and system for harmonic governance of a distribution network. An objective function is established with the minimum total harmonic distortion rate of all network nodes, and power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices are established. According to the objective function and constraint conditions, a harmonic governance model of the distribution network is obtained. A decentralized partially observable Markov decision process is established based on the harmonic governance model of the distribution network. The multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay is used to perform harmonic governance on the distribution network based on the decentralized partially observable Markov decision process. In this way, by introducing the PRE technology, the problem of low sample efficiency of the reinforcement learning algorithm is effectively solved, the stability of the training process and the algorithm convergence speed are promoted, and the effectiveness of the finally obtained harmonic governance strategy is ensured. The MADDPG algorithm is based on a centralized training-decentralized execution architecture, which can ensure that multiple active filter devices can make collaborative control strategies that take into account the harmonic problems of the entire distribution network in real time only relying on decentralized observations, thus realizing efficient harmonic collaborative governance of the entire network. In addition, the MADDPG algorithm adopts the framework form of Actor-Critic in the network architecture, and the Critic network and Actor network of the MADDPG algorithm are improved using D2RL, effectively improving the parameterization level of the Critic network and Actor network, thereby significantly enhancing the performance of the MADDPG algorithm in complex environments, and further improving the effectiveness of harmonic governance.

[0190] The above are only the embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in the relevant technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for harmonic governance of a distribution network, characterized in that, Including the steps: Establish an objective function with the minimum total harmonic distortion rate of all network nodes, and establish power flow constraints, harmonic distortion rate constraints, and constraints related to active filter devices; Obtain a harmonic management model for the distribution network according to the objective function, the power flow constraints, the harmonic distortion rate constraints, and the constraints related to the active filter devices; Establish a decentralized partially observable Markov decision process based on the harmonic management model for the distribution network; Use a multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic management on the distribution network based on the decentralized partially observable Markov decision process.

2. The harmonic governance method for a distribution network according to claim 1, wherein, The establishment of the objective function with the minimum total harmonic distortion rate of all network nodes includes: In the formula, f represents the objective function, T represents the set of optimization periods, I represents the set of all nodes in the distribution network, A represents phase A, B represents phase B, and C represents phase C. represents the total harmonic distortion rate of phase φ at node i at time t, H represents the maximum harmonic frequency considered. represents the h - th harmonic voltage of phase φ at node i at time t. represents the fundamental voltage of the h - th harmonic of phase φ at node i at time t.

3. A method for harmonic suppression in a distribution network according to claim 2, characterized in that, The power flow constraints are specifically: Wherein, represents the active power of phase φ of node i at time t, represents the reactive power of phase φ of node i at time t, represents the voltage of phase φ of node i at time t, represents the voltage of phase φ' of node j at time t, represents the line conductance between phase φ and phase φ' between node i and node j, represents the voltage phase angle difference between phase φ and phase φ' between node i and node j at time t, represents the line susceptance between phase φ and phase φ' between node i and node j, represents the h - th injected harmonic current of phase φ of node i at time t, represents the h - th harmonic coupling admittance between phase φ and phase φ' between node i and node j at time t, represents the h - th harmonic voltage of phase φ' of node i at time t; The harmonic distortion rate constraints are specifically: Wherein, represents the maximum allowable total harmonic distortion rate of the φ-phase of node i at time t; The constraints related to the active filter devices are specifically: Wherein, represents the harmonic voltage value of the φ phase of node i equipped with an active filter device at time t, represents the harmonic current value of the φ phase of node i equipped with an active filter device at time t, represents the capacity of the active filter device, represents the lower limit of the harmonic current value output by the active filter device, represents the upper limit of the harmonic current value output by the active filter device.

4. A method for harmonic suppression in a distribution network according to claim 1, characterized in that, The establishment of the decentralized partially observable Markov decision process based on the harmonic management model for the distribution network includes: Regard each active filter device as an agent, and regard the distribution network as the environment for the joint actions of the agents; Determine the number of the agents; Determine the state space according to all possible states in the environment; Determine the joint observation space according to the observation spaces of the respective agents; Determine the joint action space according to the action spaces of the respective agents; Determine the reward function according to the objective function, the harmonic distortion rate constraints, and the constraints related to the active filter devices; Determine the state transition function; Obtain a tuple according to the number of the agents, the state space, the joint observation space, the joint action space, the reward function, and the state transition function, and use the tuple to represent the decentralized partially observable Markov decision process.

5. A method for harmonic governance of a distribution network according to claim 4, characterized in that, The use of the multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic management on the distribution network based on the decentralized partially observable Markov decision process includes: Initialize the parameters of the evaluation network, the parameters of the action network, and the experience replay pool. The evaluation network and the action network are the evaluation network and the action network improved based on the deep dense reinforcement learning architecture; Set the number of training rounds to 0; Judge whether the number of training rounds reaches a first preset value. If so, output the parameters of the trained action network. If not, each agent obtains the initial state value of the environment, initializes random noise, and sets the time step to 0; Judge whether the time step reaches a second preset value. If so, add 1 to the number of training rounds and return to execute the step of judging whether the number of training rounds reaches the first preset value. If not, each agent selects a corresponding action according to the observation value and the action network, and the agent executes the action with the added random noise; Calculate the reward value after the action and obtain the observation value at the next time step; Store the state value, action value, reward value, and state value at the next time step of each agent at the current time step into the experience replay pool; Use the prioritized experience replay technique to extract a batch of state transition samples from the experience replay pool for updating the network parameters. Calculate the loss function of the evaluation network of each of the agents, and optimize and update the parameters of the evaluation estimation network by minimizing the loss function; Calculate the loss function of the action network of each of the agents, and update the parameters of the action estimation network using gradient backpropagation; Update the parameters of the evaluation target network and the parameters of the action target network; After incrementing the time step by one, return to execute the step of determining whether the time step reaches the second preset value; Load the parameters of the trained action network into the active power filter device, obtain the observed values of the local nodes of the current power system, and perform actions according to the observed values and the trained action network, and assign the optimal action value to the active power filter device to complete harmonic suppression.

6. A distribution network harmonic control system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Establish an objective function with the minimum total harmonic distortion rate of all network nodes, and establish power flow constraints, harmonic distortion rate constraints, and active power filter device-related constraints; Obtain a distribution network harmonic suppression model according to the objective function, the power flow constraints, the harmonic distortion rate constraints, and the active power filter device-related constraints; Establish a decentralized partially observable Markov decision process based on the distribution network harmonic suppression model; Use the multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic suppression on the distribution network based on the decentralized partially observable Markov decision process.

7. A harmonic governance system for a distribution network according to claim 6, characterized in that, The establishment of the objective function with the minimum total harmonic distortion rate of all network nodes includes: In the formula, f represents the objective function, T represents the set of optimization periods, I represents the set of all nodes in the distribution network, A represents phase A, B represents phase B, and C represents phase C. represents the total harmonic distortion rate of phase φ at node i at time t, H represents the maximum harmonic frequency considered. represents the h-th harmonic voltage of phase φ at node i at time t. represents the fundamental voltage of the h-th order of phase φ at node i at time t.

8. A harmonic governance system for a distribution network according to claim 7, characterized in that The power flow constraints are specifically: Wherein, represents the active power of phase φ of node i at time t, represents the reactive power of phase φ of node i at time t, represents the voltage of phase φ of node i at time t, represents the voltage of phase φ' of node j at time t, represents the line conductance between phase φ and phase φ' between node i and node j, represents the voltage phase angle difference between phase φ and phase φ' between node i and node j at time t, represents the line susceptance between phase φ and phase φ' between node i and node j, represents the h - th injected harmonic current of phase φ of node i at time t, represents the h - th harmonic coupling admittance between phase φ and phase φ' between node i and node j at time t, represents the h - th harmonic voltage of phase φ' of node i at time t; The harmonic distortion rate constraints are specifically: In the formula, represents the maximum allowable total harmonic distortion rate of the φ-phase of node i at time t; The active power filter device-related constraints are specifically: Wherein, represents the harmonic voltage value of phase φ at node i equipped with an active filter device at time t, represents the harmonic current value of phase φ at node i equipped with an active filter device at time t, represents the capacity of the active filter device, represents the lower limit of the harmonic current value output by the active filter device, represents the upper limit of the harmonic current value output by the active filter device.

9. The harmonic management system for a distribution network according to claim 6, wherein, The establishment of the decentralized partially observable Markov decision process based on the distribution network harmonic suppression model includes: Take each active power filter device as an agent, and take the distribution network as the environment for the joint actions of the agents; Determine the number of the agents; Determine the state space according to all possible states in the environment; Determine the joint observation space according to the observation spaces of the respective agents; Determine the joint action space according to the action spaces of the respective agents; Determine the reward function according to the objective function, the harmonic distortion rate constraints, and the active power filter device-related constraints; Determine the state transition function; Obtain a tuple according to the number of the agents, the state space, the joint observation space, the joint action space, the reward function, and the state transition function, and use the tuple to represent the decentralized partially observable Markov decision process.

10. A distribution network harmonic control system according to claim 9, characterized in that: The use of the multi-agent deep deterministic policy gradient algorithm combined with prioritized experience replay to perform harmonic suppression on the distribution network based on the decentralized partially observable Markov decision process includes: Initialize the parameters of the evaluation network, the parameters of the action network, and the experience replay pool, where the evaluation network and the action network are the evaluation network and the action network improved based on the deep dense reinforcement learning architecture; Set the number of training rounds to 0; Determine whether the number of training rounds has reached a first preset value. If so, output the parameters of the trained action network. If not, each of the agents obtains the initial state value of the environment, initializes random noise, and sets the time step to 0; Determine whether the time step has reached a second preset value. If so, increment the number of training rounds by one and then return to execute the step of determining whether the number of training rounds has reached the first preset value. If not, each of the agents selects corresponding actions according to the observation value and the action network, and the agent executes the action with the added random noise; Calculate the reward value after the action and obtain the observation value at the next time step; Store the state value, action value, reward value, and state value at the next time step of each of the agents at the current time step into the experience replay pool; Use the prioritized experience replay technique to extract a batch of state transition samples from the experience replay pool for updating the network parameters; Calculate the loss function of the evaluation network of each of the agents, and optimize and update the parameters of the evaluation estimation network by minimizing the loss function; Calculate the loss function of the action network of each of the agents, and update the parameters of the action estimation network using gradient backpropagation; Update the parameters of the evaluation target network and the parameters of the action target network; Increment the time step by one and then return to execute the step of determining whether the time step has reached the second preset value; Load the parameters of the trained action network into the active power filter device, obtain the observation value of the local node of the current power system, and execute an action according to the observation value and the trained action network, and assign the optimal action value to the active power filter device to complete harmonic suppression.

Images