Power distribution network voltage control method and system based on improved DDPG algorithm

Abstract

The invention discloses a power distribution network voltage control method and system based on an improved DDPG algorithm, and the method comprises the steps: carrying out the voltage control of a power distribution network through employing the improved DDPG algorithm in combination with a preset cost function when the voltage deviation of the power distribution network exceeds a preset threshold value, according to the improved DDPG algorithm, on the basis of the DDPG algorithm, a time sequence differential error is calculated for the experience stored in the experience playback buffer area to serve as a priority, and the experience is sampled from the experience playback buffer area according to the priority so that the utilization efficiency of the key experience with the higher priority can be improved. The invention aims to improve the computational efficiency of reinforcement learning, accelerate the convergence process and enhance the anti-noise capability, thereby providing a more accurate and more stable real-time voltage control solution to meet the dynamic adjustment demand of a power system in a complex environment.

Description

Technical Field

[0001] The present invention relates to the technical field of voltage control of power systems, and in particular to a distribution network voltage control method and system based on an improved DDPG algorithm. Background Art

[0002] Voltage control is an important guarantee for the stability and reliability of modern power systems. Traditional voltage control methods, such as centralized control, distributed control, local control, capacitor control and droop control, are widely used in power systems. However, these methods also have some significant disadvantages. First, traditional control methods often rely on preset models and parameters, and have poor adaptability to dynamic changes in the system, especially when facing complex situations such as grid load fluctuations and distributed energy access, and cannot flexibly adjust the control strategy. Secondly, these methods usually lack sufficient adaptive capabilities. When the system state changes drastically, it is difficult to respond effectively in time, resulting in large voltage fluctuations or unstable control. In addition, traditional voltage control methods have poor resistance to environmental noise and interference, especially when there are uncertainties and external disturbances in the power system, errors are prone to occur, affecting the stability of the system. With the development of smart grids, voltage control methods based on reinforcement learning have gradually become a research hotspot. Reinforcement learning has the memory and nonlinear fitting capabilities of other neural network structures such as recursive neural networks. In the control stage, the control action can be directly given without considering the distribution network model, saving the process of solving the optimal solution. As a reinforcement learning method, DDPG (Deep Deterministic Policy Gradient) can adaptively adjust the control strategy and optimize voltage regulation through interaction with the environment. It can automatically adjust the control strategy according to the real-time status of the power grid, thereby achieving precise voltage control. However, the traditional DDPG algorithm has high computational complexity and needs to process a large amount of empirical data and optimize multiple networks, resulting in large consumption of computing resources and unable to meet the requirements of real-time voltage control. Secondly, the algorithm converges slowly. Since the training process relies on a large amount of interactive experience and the gradient update is unstable, it is difficult to quickly optimize the control strategy in a dynamic environment. Finally, the traditional DDPG algorithm is sensitive to environmental noise, especially under dynamic changes in the power grid or external disturbances, it is easily interfered by noise, affecting control accuracy and system stability. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a distribution network voltage control method and system based on an improved DDPG algorithm are provided. The present invention aims to improve the computational efficiency of reinforcement learning, accelerate the convergence process, and enhance the anti-noise ability, thereby providing a more accurate and stable real-time voltage control solution to meet the dynamic adjustment needs of the power system in a complex environment.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A distribution network voltage control method based on an improved DDPG algorithm comprises the following steps: when the voltage deviation of the distribution network exceeds a preset threshold, the improved DDPG algorithm is used in combination with a preset cost function to control the distribution network voltage, the improved DDPG algorithm is based on the DDPG algorithm, a timing difference error is calculated for the experience stored in the experience playback buffer as a priority, and the experience is sampled from the experience playback buffer according to the priority to improve the utilization efficiency of the key experience with a higher priority, and the function expression for calculating the timing difference error is: , in, For experience The timing difference error, is the reward at the current time t calculated using the cost function; is the discount factor; is the target value network of the DDPG algorithm right , The estimated value of is the state at the next moment t+1, The target policy network of the DDPG algorithm right The selection action, The value network of the DDPG algorithm about The estimated value of They are respectively the state and action at the current time t, the state includes the voltage and reactive power at the corresponding time, and the action is to update the reactive power of the distribution network to adjust the voltage of the distribution network; the cost function is composed of the action cost and the voltage deviation, wherein the action cost is the L1 norm of the control input, and the control input includes part or all of reactive power compensation, reactive power injection and the dispatching power of distributed energy.

[0005] Optionally, the function expression of the preset cost function is: , in, For Node The reward at the current time t, For Node The voltage at the current time t, For Node The control input at the current time t is, is the voltage deviation, is the action cost, is the balance coefficient of voltage deviation, is the balance coefficient of action cost, For Node The target voltage value The average of , and: , , , in, is the voltage at the next moment t+1, is the conductivity matrix of the power grid, Reactive power injection for the next time t+1; The impact of the external environment on the voltage includes the reactive power injection of the node at the current time t and the next time t+1; and Node Reactive power injection at the current time t and the next time t+1, is the interval time, for The control strategy function is Is a node The policy parameters of the control policy function.

[0006] Optionally, when sampling experience from the experience playback buffer, any experience The probability of being sampled The calculation function expression is: , , in, For experience The priority of For experience The priority of For experience The index of For experience The timing difference error, is the priority adjustment factor, is the error value.

[0007] Optionally, the adopting of the improved DDPG algorithm in combination with a preset cost function to perform distribution network voltage control includes: S101, create and initialize the policy network in the DDPG algorithm , target strategy network , Value Network and target value network , where the policy network and target strategy network Same structure, value network and target value network Same structure, strategy network and value network Both include the first hidden layer, the second hidden layer and the output layer, and the policy network and value network The function expression of the first hidden layer is: , , in, and Strategy Network and value network The output of the first hidden layer is for Activation function, Strategy Network The weight matrix of the first hidden layer is and Value Network The weight matrix of the first hidden layer for states and actions, and Strategy Network and value network The bias vector of the first hidden layer, is the state of the normalized network at the current time t The processing result, is the action of the normalized network at the current time t The processing result of strategy network and value network The function expression of the second hidden layer is: , , in, and Strategy Network and value network The output of the second hidden layer is is the normalized network, and Strategy Network and value network The weight matrix of the second hidden layer is and Strategy Network and value network The bias vector of the second hidden layer of the policy network and value network The function expression of the output layer is: , , in, Strategy Network Output the action at the current time t, and Strategy Network The weight matrix and bias vector of the output layer, Value Network Output the state at the current time t and actions The estimated value of and Value Network The weight matrix and bias vector of the output layer; the function expression of the normalized network is: = , in, To normalize the network with respect to the input features The output result is, is a learnable scaling parameter, is a learnable offset parameter, , is the mean and variance of the mini-batch input features, To prevent a tiny constant with a zero denominator; S102, randomly initialize the policy network in the DDPG algorithm The parameters are assigned to the target policy network , Value Network The parameters are assigned to the target value network , initialize the experience playback buffer used to store experience; S103, executing the improved DDPG algorithm environment interaction, including: Based on the current moment Status Add Ornstein-Uhlenbeck noise to the output of to generate the current moment Action ; Execute the current moment Action Inject reactive power into the distribution network and obtain the current moment through the cost function Rewards and the state at the next moment t+1 ; will be used to represent the current moment Five-tuple of experience Stored in the experience replay buffer, where To mark the end; S104, sampling a batch of experiences from the experience playback buffer according to the priority, combining the sampled experiences with the target value network The target value is calculated according to the following formula : , in, For the current moment Rewards, is the discount factor, is the target value network of the DDPG algorithm right , The estimated value of is the state at the next moment t+1, The target policy network of the DDPG algorithm right The selected action; the value network is calculated using the preset loss function According to the current time Action and status Value estimation, target value The loss between them, and the value of the network is obtained by minimizing the loss Parameters; S105, using the optimization objective function shown in the following formula To update the policy network Parameters: , in, represents the mathematical expectation, Value Network about The estimated value of Strategy Network about and strategy, in which Strategy Network Parameters, is the weight parameter of entropy regularization, For strategy Entropy of S106, using a soft update mechanism to update the parameters of the target network according to the following formula: ， in, are the parameters of the updated target network, are the parameters of the target network before updating, is the current parameter of the target network corresponding to the main network, is the step size of the soft update, the target network includes the target strategy network and target value network , target policy network and target value network The corresponding main networks are the strategy networks and value network ; S107, determine whether the number of iterations reaches a preset threshold. If it has not reached the preset threshold, jump to step S103 to continue iterating; otherwise, end and exit.

[0008] Optionally, in step S102, the policy network Based on the current moment Status Add Ornstein-Uhlenbeck noise to the output of to generate the current moment Action The function expression is: , in, for The action of the moment, Strategy Network Based on the current moment Status The output, Strategy Network Parameters, For the current moment The update function of Ornstein-Uhlenbeck noise is: , in, and are the next time t+1 and the current time respectively The noise value, is the mean value of the noise; is the mean reversion speed, which is used to control the rate at which the noise gradually reverts to the mean; is the random amplitude of the noise, is white noise sampled from a standard normal distribution, Follow the distribution .

[0009] Optionally, the function expression of the loss function preset in step S103 is: , in, is the preset loss function, represents the mathematical expectation, is the sampling weight of experience j, Value Network Based on the current moment Status and actions The estimated value of Value Network Parameters, is the target value; where the function expression of the sampling weight of experience j is: , in, is the size of the experience replay buffer, For experience The probability of being sampled, is the parameter used to adjust the weight.

[0010] Optionally, the strategy in step S104 Entropy The calculation function expression is: , in, For action, is a random strategy.

[0011] In addition, the present invention also provides a distribution network voltage control system based on an improved DDPG algorithm, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm.

[0012] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm through a processor.

[0013] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm through a processor.

[0014] Compared with the prior art, the present invention mainly has the following advantages: the distribution network voltage control method based on the improved DDPG algorithm of the present invention includes that when the voltage deviation of the distribution network exceeds a preset threshold, the improved DDPG algorithm is used in combination with a preset cost function to perform distribution network voltage control, the improved DDPG algorithm is based on the DDPG algorithm, and the timing differential error is calculated for the experience stored in the experience replay buffer as a priority, and the experience is sampled from the experience replay buffer according to the priority to improve the utilization efficiency of key experience with higher priority, thereby improving the computational efficiency of reinforcement learning, accelerating the convergence process, and enhancing the anti-noise ability, thereby providing a more accurate and stable real-time voltage control solution to meet the dynamic regulation needs of the power system in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0016] Figure 2 Schematic diagram of the principle of improving the DDPG algorithm in an embodiment of the present invention.

[0017] Figure 3 Schematic diagram of a simulation environment in an embodiment of the present invention.

[0018] Figure 4 Schematic diagram of comparison of results between the improved DDPG algorithm and the original DDPG algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0020] like Figure 1 As shown, the distribution network voltage control method based on the improved DDPG algorithm of this embodiment includes the following steps: when the voltage deviation of the distribution network exceeds a preset threshold, the improved DDPG algorithm is used in combination with a preset cost function to control the distribution network voltage. The improved DDPG algorithm is based on the DDPG algorithm, and the temporal difference error is calculated for the experience stored in the experience playback buffer as a priority, and the experience is sampled from the experience playback buffer according to the priority to improve the utilization efficiency of the key experience with a higher priority. The function expression for calculating the temporal difference error is: , in, For experience The timing difference error, is the reward at the current time t calculated using the cost function; is the discount factor; is the target value network of the DDPG algorithm right , The estimated value of is the state at the next moment t+1, The target policy network of the DDPG algorithm right The selection action, The value network of the DDPG algorithm about The estimated value of are the state and action at the current time t, respectively. The state includes the voltage and reactive power at the corresponding time. The action is to update the reactive power of the distribution network to adjust the voltage of the distribution network. The calculation of the priority experience playback mechanism adopts the time difference error Weigh the importance of experience.

[0021] like Figure 1 As shown in the figure, before judging that the voltage deviation of the distribution network exceeds the preset threshold, it is necessary to collect the voltage information of the distribution network through the monitoring system, and calculate the absolute value of the voltage information and the rated voltage to obtain the voltage deviation of the distribution network. , the judgment condition that the voltage deviation of the distribution network exceeds the preset threshold is The most important thing about voltage control is to keep the voltage stable. Voltage stability is defined as the ability of the system voltage trajectory to return to an acceptable range after any disturbance. and , Converge to the set : , Meaning: , The distance is defined as: .

[0022] in, The influence of the external environment on the voltage includes the reactive power injection of the node at the current time t and the next time t+1; is the initial voltage, is the voltage at the current time t, represents a set of voltage values, n is the n-dimensional real number space, is the lower limit of voltage, is the actual voltage, is the upper limit of voltage, for The distance between For collection In the case of high penetration of distributed power grids, load and renewable energy generation often change rapidly in a short time scale, so it is very important to ensure that the designed controller meets the stability conditions. For the requirement of voltage stability, the optimal voltage control problem can be expressed as: , , , , in, is the objective function, which is used to measure the operating cost of the entire system. The goal of the voltage control problem is to reduce the time step From 0 to total cost. The parameters of the control strategy, that is, the parameters to be optimized (such as the policy network parameters of DDPG). is the total number of steps in the control time domain. Represents the discount factor, which indicates the degree of decay of future rewards and weighs short-term and long-term benefits. is the total number of grid nodes. For Node The reward at the current time t, For Node The voltage at the current time t, For Node The control input at the current time t is is the voltage at the next moment t+1, is the conductivity matrix of the power grid, Reactive power injection for the next time t+1; The impact of the external environment on the voltage includes the reactive power injection of the node at the current time t and the next time t+1; and Node Reactive power injection at the current time t and the next time t+1, is the interval time, for The control strategy function is Is a node The policy parameters of the control policy function. The cost function at the current time t is usually related to the node voltage and control input Related.

[0023] The cost function in this embodiment is composed of action cost and voltage deviation, where the action cost is the L1 norm of the control input, and the control input includes reactive power compensation (Q compensation), reactive power injection, and the dispatch power (partial or full) of distributed energy (such as photovoltaic, wind power, etc.). Specifically, the function expression of the preset cost function in this embodiment is: , in, For Node The reward at the current time t, For Node The voltage at the current time t, For Node The control input at the current time t is is the voltage deviation, is the action cost, is the balance coefficient of voltage deviation, is the balance coefficient of action cost, For Node The target voltage value The average of the action costs is used to control the input through the L1 norm constraint The size of the control strategy is to avoid excessive action and make the system more stable and robust; the voltage deviation is and The L2 norm of the two voltage deviations is used to meet the voltage stability requirement and is: , , , in, is the voltage at the next moment t+1, is the conductivity matrix of the power grid, Reactive power injection for the next time t+1; The impact of the external environment on the voltage includes the reactive power injection of the node at the current time t and the next time t+1; and Node Reactive power injection at the current time t and the next time t+1, is the interval time, for The control strategy function is Is a node The policy parameters of the control policy function. It is a collection of local strategy parameters. The ultimate goal is to control the voltage per unit value within the range of 0.95 to 1.05 to ensure the voltage stability and operation safety of the power grid system.

[0024] In this embodiment, when sampling experience from the experience playback buffer, any experience The probability of being sampled The calculation function expression is: , , in, For experience The priority of For experience The priority of For experience The index of For experience The timing difference error, is the priority adjustment factor, is the error value (even if the timing difference error is zero, the experience will still have a minimum priority and have a chance to be sampled). Priority is the sampling probability update, and sampling from the buffer is based on priority Experience ,experience The timing difference error The larger the value, the higher the priority. The higher the value, the higher the experience replay buffer is. The experience replay buffer improves the sample utilization efficiency by increasing the sampling probability of key experiences, making strategy optimization more efficient.

[0025] like Figure 2 As shown, in this embodiment, the improved DDPG algorithm combined with a preset cost function is used to control the voltage of the distribution network, including: S101, create and initialize the policy network in the DDPG algorithm , target strategy network , Value Network and target value network , where the policy network and target strategy network Same structure, value network and target value network Same structure, strategy network and value network Both include the first hidden layer, the second hidden layer and the output layer, and the policy network and value network The function expression of the first hidden layer is: , , in, and Strategy Network and value network The output of the first hidden layer is for Activation function, Strategy Network The weight matrix of the first hidden layer is and Value Network The weight matrix of the first hidden layer for states and actions, and Strategy Network and value network The bias vector of the first hidden layer, is the state of the normalized network at the current time t The processing result, is the action of the normalized network at the current time t The processing result; Policy Network and value network The function expression of the second hidden layer is: , , in, and Strategy Network and value network The output of the second hidden layer is is the normalized network, and Strategy Network and value network The weight matrix of the second hidden layer is and Strategy Network and value network The bias vector of the second hidden layer; Policy Network and value network The function expression of the output layer is: , , in, Strategy Network Output the action at the current time t, and Strategy Network The weight matrix and bias vector of the output layer, Value Network Output the state at the current time t and actions The estimated value of and Value Network The weight matrix and bias vector of the output layer; the function expression of the normalized network is: = , in, To normalize the network with respect to the input features The output result is, is a learnable scaling parameter, is a learnable offset parameter, , is the mean and variance of the mini-batch input features, A small constant to prevent the denominator from being zero; the policy network Responsible for generating the current state Deterministic actions under , the policy network consists of parameters Definition, its output is used to directly interact with the environment and determine actions such as reactive power injection, which can be expressed as: ; Value Network Responsible for assessing status With action The impact on the long-term return of the agent can be expressed as ; S102, randomly initialize the policy network in the DDPG algorithm The parameters are assigned to the target policy network , Value Network The parameters are assigned to the target value network , initialize the experience playback buffer for storing experience; value network Output Indicates that under the current strategy, from the state Start executing the action The cumulative reward that can be obtained, the value network is composed of parameters Definition. Target Network and To calculate the target value , whose weights are delayed copies of the main network parameters. Randomly initialize the main network and of and To ensure the stability of training, the target network is initialized to be consistent with the main network parameters, namely: , Through the above settings, the main network is used for strategy generation and value evaluation, while the target network provides a stable training target for the main network by slowly changing, and then the soft update mechanism is gradually adjusted; S103, executing the improved DDPG algorithm environment interaction, including: Based on the current moment Status Add Ornstein-Uhlenbeck noise to the output of to generate the current moment Action ; Execute the current moment Action Inject reactive power into the distribution network and obtain the current moment through the cost function Rewards and the state at the next moment t+1 ; will be used to represent the current moment Five-tuple of experience Stored in the experience replay buffer, where The end mark; in order to improve sampling efficiency and learning effect, the DDPG algorithm adopts a priority experience replay mechanism (Prioritized Replay Buffer). Under this mechanism, different experiences are assigned sampling priorities according to their importance (measured by TD error), so that key experiences are used more frequently. The storage form of its experience: After each interaction with the environment, a complete five-tuple is stored: ; S104, sampling a batch of experiences from the experience playback buffer according to the priority, combining the sampled experiences with the target value network The target value is calculated according to the following formula : , in, For the current moment Rewards, is the discount factor, is the target value network of the DDPG algorithm right , The estimated value of is the state at the next moment t+1, The target policy network of the DDPG algorithm right The selected action; the value network is calculated using the preset loss function According to the current time Action and status Estimated value, target value The loss between them, and the value of the network is obtained by minimizing the loss Parameters; S105, using the optimization objective function shown in the following formula To update the policy network Parameters: , in, represents the mathematical expectation, Value Network about The estimated value of Strategy Network about and strategy, in which Strategy Network Parameters, is the weight parameter of entropy regularization, For strategy Entropy of S106, using a soft update mechanism to update the parameters of the target network according to the following formula: ， in, are the parameters of the updated target network, are the parameters of the target network before updating, is the current parameter of the target network corresponding to the main network, is the step size of the soft update, the target network includes the target strategy network and target value network , target policy network and target value network The corresponding main networks are the strategy networks and value network ; S107, determine whether the number of iterations reaches a preset threshold. If it has not reached the preset threshold, jump to step S103 to continue iterating; otherwise, end and exit.

[0026] The actions generated by the policy network are deterministic and lack randomness. Therefore, in order to enhance the agent's exploration ability in the action space, the improved algorithm introduces the Ornstein-Uhlenbeck (OU) noise process. OU noise has time correlation, and the generated noise is smoother, which helps to achieve more effective exploration in continuous control tasks. Specifically, in step S102 of this embodiment, the policy network Based on the current moment Status Add Ornstein-Uhlenbeck noise to the output of to generate the current moment Action The function expression is: , in, for The action of the moment, Strategy Network Based on the current moment Status The output, Strategy Network Parameters, For the current moment The update function of Ornstein-Uhlenbeck noise is: , in, and are the next time t+1 and the current time respectively The noise value, is the mean value of the noise; is the mean reversion speed, which is used to control the rate at which the noise gradually reverts to the mean; is the random amplitude of the noise, is white noise sampled from a standard normal distribution, Follow the distribution . In this embodiment, by introducing the Ornstein-Uhlenbeck noise process, the noise resistance can be effectively enhanced, thereby providing a more accurate and stable real-time voltage control solution to meet the dynamic regulation needs of the power system in a complex environment. By introducing OU noise, the agent can conduct a large-scale exploration in the early stage of training, and the noise amplitude gradually decreases over time, which helps to focus on high-quality actions in the later stage of training.

[0027] When updating the value network in this embodiment, the optimization goal of the value network is to minimize the current value estimate With target value Specifically, the function expression of the loss function preset in step S103 of this embodiment is: , in, is the preset loss function (i.e., optimization objective), represents the mathematical expectation, is the sampling weight of experience j, Value Network Based on the current moment Status and actions The estimated value of Value Network Parameters, is the target value; where the function expression of the sampling weight of experience j is: , in, is the size of the experience replay buffer, For experience The probability of being sampled, is the parameter used to adjust the weight.

[0028] When updating the policy network, the optimization goal of the policy network is to maximize the state Next select action of value (value estimation), and at the same time encourage exploration behavior through entropy regularization of the strategy. Therefore, the optimization objective function shown in the following formula is adopted in step S105 To update the policy network Parameters: , in, represents the mathematical expectation, Value Network about The estimated value of Strategy Network about and strategy, in which Strategy Network Parameters, is the weight parameter of entropy regularization, For strategy The entropy of Entropy The calculation function expression is: , in, For action, is a random strategy.

[0029] In order to verify the distribution network voltage control method based on the improved DDPG algorithm in this embodiment, the Figure 3 The simulation environment shown is used to simulate the distribution network. The 220kV bus 650 is stepped down to 10kV by a transformer and then output to the distribution network. The distribution network includes multiple overhead lines. The black circles in the figure represent nodes, and B1 represents a circuit breaker. Figure 4 This is a schematic diagram showing the comparison of the results of the improved DDPG algorithm and the original DDPG algorithm in this embodiment. Figure 4 It can be seen that the improved DDPG algorithm and the original DDPG algorithm in this embodiment are compared and run in a 13-node voltage control system. The average reward of the improved DDPG algorithm in this embodiment is improved compared with the original DDPG algorithm in the training process, especially in the early stage of training, the convergence is faster, and the final performance is more stable. It can be seen that when the voltage deviation of the distribution network exceeds the preset threshold, the distribution network voltage control method based on the improved DDPG algorithm in this embodiment adopts the improved DDPG algorithm combined with the preset cost function to control the distribution network voltage. The improved DDPG algorithm is based on the DDPG algorithm. The time difference error is calculated for the experience stored in the experience replay buffer as the priority, and the experience is sampled from the experience replay buffer according to the priority to improve the utilization efficiency of the key experience with higher priority, so as to improve the computational efficiency of reinforcement learning, accelerate the convergence process, and enhance the anti-noise ability, thereby providing a more accurate and stable real-time voltage control solution to meet the dynamic regulation needs of the power system in a complex environment.

[0030] In addition, this embodiment also provides a distribution network voltage control system based on an improved DDPG algorithm, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm.

[0031] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm through a processor.

[0032] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm through a processor.

[0033] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0034] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A distribution network voltage control method based on an improved DDPG algorithm, characterized in that: The method comprises the following steps: when the voltage deviation of the distribution network exceeds a preset threshold, an improved DDPG algorithm is used in combination with a preset cost function to control the voltage of the distribution network, the improved DDPG algorithm is based on the DDPG algorithm, a timing differential error is calculated for the experience stored in the experience playback buffer as a priority, and the experience is sampled from the experience playback buffer according to the priority to improve the utilization efficiency of the key experience with a higher priority, and the function expression for calculating the timing differential error is: ， in, For experience The timing difference error, is the reward at the current time t calculated using the cost function; is the discount factor; is the target value network of the DDPG algorithm right , The estimated value of is the state at the next moment t+1, The target policy network of the DDPG algorithm right The selection action, The value network of the DDPG algorithm about The estimated value of They are respectively the state and action at the current time t, the state includes the voltage and reactive power at the corresponding time, and the action is to update the reactive power of the distribution network to adjust the voltage of the distribution network; the cost function is composed of the action cost and the voltage deviation, wherein the action cost is the L1 norm of the control input, and the control input includes part or all of reactive power compensation, reactive power injection and the dispatching power of distributed energy.

2. The distribution network voltage control method based on the improved DDPG algorithm according to claim 1 is characterized in that: The function expression of the preset cost function is: ， in, For Node The reward at the current time t, For Node The voltage at the current time t, For Node The control input at the current time t is is the voltage deviation, is the action cost, is the balance coefficient of voltage deviation, is the balance coefficient of action cost, For Node The target voltage value The average of , and: ， ， ， in, is the voltage at the next moment t+1, is the conductivity matrix of the power grid, Reactive power injection for the next time t+1; The influence of the external environment on the voltage includes the reactive power injection of the node at the current time t and the next time t+1; and Node Reactive power injection at the current time t and the next time t+1, is the interval time, for The control strategy function is Is a node The policy parameters of the control policy function.

3. The distribution network voltage control method based on the improved DDPG algorithm according to claim 2 is characterized in that: When sampling experience from the experience playback buffer, any experience The probability of being sampled The calculation function expression is: ， ， in, For experience The priority of For experience The priority of For experience The index of For experience The timing difference error, is the priority adjustment factor, is the error value.

4. The distribution network voltage control method based on the improved DDPG algorithm according to claim 1 is characterized in that: The improved DDPG algorithm combined with a preset cost function to control the distribution network voltage includes: S101, create and initialize the policy network in the DDPG algorithm , target strategy network , Value Network and target value network , where the policy network and target strategy network Same structure, value network and target value network Same structure, strategy network and value network Both include the first hidden layer, the second hidden layer and the output layer, and the policy network and value network The function expression of the first hidden layer is: ， ， in, and Strategy Network and value network The output of the first hidden layer is for Activation function, Strategy Network The weight matrix of the first hidden layer is and Value Network The weight matrix of the first hidden layer for states and actions, and Strategy Network and value network The bias vector of the first hidden layer, is the state of the normalized network at the current time t The processing result, is the action of the normalized network at the current time t The processing result of strategy network and value network The function expression of the second hidden layer is: ， ， in, and Strategy Network and value network The output of the second hidden layer is is the normalized network, and Strategy Network and value network The weight matrix of the second hidden layer is and Strategy Network and value network The bias vector of the second hidden layer of the policy network and value network The function expression of the output layer is: ， ， in, Strategy Network Output the action at the current time t, and Strategy Network The weight matrix and bias vector of the output layer, Value Network Output the state at the current time t and actions The estimated value of and Value Network The weight matrix and bias vector of the output layer; the function expression of the normalized network is: = ， in, To normalize the network with respect to the input features The output result is, is a learnable scaling parameter, is a learnable offset parameter, , is the mean and variance of the mini-batch input features, To prevent a tiny constant with a zero denominator; S102, randomly initialize the policy network in the DDPG algorithm The parameters are assigned to the target policy network , Value Network The parameters are assigned to the target value network , initialize the experience playback buffer used to store experience; S103, executing the improved DDPG algorithm environment interaction, including: Based on the current moment Status Add Ornstein-Uhlenbeck noise to the output of to generate the current moment Action ; Execute the current moment Action Inject reactive power into the distribution network and obtain the current moment through the cost function Rewards and the state at the next moment t+1 ; will be used to represent the current moment Five-tuple of experience Stored in the experience replay buffer, where To mark the end; S104, sampling a batch of experiences from the experience playback buffer according to the priority, combining the sampled experiences with the target value network The target value is calculated according to the following formula : ， in, For the current moment Rewards, is the discount factor, is the target value network of the DDPG algorithm right , The estimated value of is the state at the next moment t+1, The target policy network of the DDPG algorithm right The selected action; the value network is calculated using the preset loss function According to the current time Action and status Value estimation, target value The loss between them, and the value of the network is obtained by minimizing the loss Parameters; S105, using the optimization objective function shown in the following formula To update the policy network Parameters: ， in, represents the mathematical expectation, Value Network about The estimated value of Strategy Network about and strategy, in which Strategy Network Parameters, is the weight parameter of entropy regularization, For strategy Entropy of S106, using a soft update mechanism to update the parameters of the target network according to the following formula: ， in, are the parameters of the updated target network, are the parameters of the target network before updating, is the current parameter of the target network corresponding to the main network, is the step size of the soft update, the target network includes the target strategy network and target value network , target policy network and target value network The corresponding main networks are the strategy networks and value network ; S107, determine whether the number of iterations reaches a preset threshold. If it has not reached the preset threshold, jump to step S103 to continue iterating; otherwise, end and exit.

5. The distribution network voltage control method based on the improved DDPG algorithm according to claim 4 is characterized in that: In step S102, the policy network Based on the current moment Status Add Ornstein-Uhlenbeck noise to the output of to generate the current moment Action The function expression is: ， in, for The action of the moment, Strategy Network Based on the current moment Status The output, Strategy Network Parameters, For the current moment The update function of Ornstein-Uhlenbeck noise is: ， in, and are the next time t+1 and the current time respectively The noise value, is the mean value of the noise; is the mean reversion speed, which is used to control the rate at which the noise gradually reverts to the mean; is the random amplitude of the noise, is white noise sampled from a standard normal distribution, Follow the distribution .

6. The distribution network voltage control method based on the improved DDPG algorithm according to claim 4 is characterized in that: The function expression of the loss function preset in step S103 is: ， in, is the preset loss function, represents the mathematical expectation, is the sampling weight of experience j, Value Network Based on the current moment Status and actions The estimated value of Value Network Parameters, is the target value; where the function expression of the sampling weight of experience j is: ， in, is the size of the experience replay buffer, For experience The probability of being sampled, is the parameter used to adjust the weight.

7. The distribution network voltage control method based on the improved DDPG algorithm according to claim 4 is characterized in that: Strategy in step S104 Entropy The calculation function expression is: ， in, For action, is a random strategy.

8. A distribution network voltage control system based on an improved DDPG algorithm, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm as described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the distribution network voltage control method based on the improved DDPG algorithm as described in any one of claims 1 to 7 through a processor.

Images