Distribution network security constraint voltage control method and system based on deep reinforcement learning

Abstract

The present invention discloses a method and system for voltage control in a distribution network with safety constraints based on deep reinforcement learning. The method includes training using an improved DDPG algorithm model, and the improved DDPG algorithm model includes improving its policy network for the DDPG algorithm model. The improved policy network generates corresponding actions based on the distribution network state, including: generating a weight recovery matrix and a bias recovery matrix, calculating positive and negative weight matrices and bias matrices, and fusing them with the distribution network state at time t #imgabs0# to generate two control signals, and adding the two control signals to obtain a control signal for controlling reactive power injection in the distribution network. The present invention aims to address the problems of traditional deep reinforcement learning being sensitive to environmental noise, susceptible to noise interference under dynamic changes in the power grid or external disturbances, affecting control accuracy and system stability, and making it difficult to ensure voltage stability in complex scenarios.

Description

Technical Field

[0001] The present invention relates to voltage control technology in power systems, and in particular to a distribution network safety-constrained voltage regulation method and system based on deep reinforcement learning. Background Art

[0002] In power systems, voltage control is a critical component for ensuring system stability, improving power quality, and enhancing equipment safety. As modern power systems continue to expand in scale and complexity, voltage fluctuations, overvoltage, and undervoltage have become major factors impacting their stable operation. The research and application of voltage control technology is crucial for optimizing grid operation, reducing equipment losses, and improving power transmission efficiency. The goal of voltage control is to maintain voltage within a reasonable range to ensure the normal operation of power equipment and prevent damage and unstable power transmission caused by voltage anomalies. Excessively high or low voltage can adversely impact the system, affecting equipment lifespan, system safety, and overall operational efficiency. Therefore, effective voltage control methods can ensure economical, safe, and reliable operation of power systems. Traditional voltage control methods rely on static and dynamic equipment, such as transformers and reactive power compensation devices. However, with the integration of renewable energy, the increase in distributed power sources, and the growing complexity of power loads, voltage control faces new challenges. Modern voltage control must not only account for static load fluctuations but also account for transient fluctuations and other unexpected events caused by the addition of renewable energy sources. Furthermore, the high-dimensional nature of large-capacity power systems complicates the real-time voltage control problem. Rapid and optimal implementation of effective control becomes a significant challenge, particularly in the face of severe but unknown disturbances. As grid structures become increasingly complex, traditional approaches based on precise mathematical models are no longer effective for emergency control. Therefore, it is necessary to develop a new real-time control paradigm that can effectively and rapidly support grid operators in executing autonomous and efficient control actions.

[0003] Traditional voltage control methods, such as centralized control, distributed control, local control, capacitor control, and droop control, are widely used in power systems. However, these methods also have some significant disadvantages. First, traditional control methods often rely on preset models and parameters, and have poor adaptability to dynamic changes in the system. In particular, they are unable to flexibly adjust the control strategy when faced with complex situations such as grid load fluctuations and distributed energy access. Secondly, these methods usually lack sufficient adaptive capabilities. When the system state changes drastically, it is difficult to respond effectively and in a timely manner, resulting in large voltage fluctuations or unstable control. In addition, traditional voltage control methods have poor resistance to environmental noise and interference. Especially when there are uncertainties and external disturbances in the power system, errors are prone to occur, affecting the stability of the system.

[0004] However, the traditional DDPG algorithm has high computational complexity, requiring the processing of large amounts of empirical data and the optimization of multiple networks. This consumes significant computing resources and cannot meet the requirements of real-time voltage control. Secondly, the algorithm converges slowly. Since the training process relies on a large amount of interactive experience and gradient updates are unstable, it is difficult to quickly optimize the control strategy in dynamic environments. Finally, the traditional DDPG algorithm is sensitive to environmental noise, especially under dynamic changes in the power grid or external disturbances. It is easily affected by noise, which affects control accuracy and system stability, making it difficult to ensure voltage stability in complex scenarios. Summary of the Invention

[0005] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems in the prior art, a method and system for voltage control based on safety constraints in distribution networks is provided. The present invention aims to solve the problems that traditional deep reinforcement learning is sensitive to environmental noise, easily affected by noise interference under dynamic changes or external disturbances in the power grid, affecting control accuracy and system stability, and making it difficult to ensure voltage stability in complex scenarios.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A distribution network security-constrained voltage control method based on deep reinforcement learning includes the following steps: using an improved DDPG algorithm model for training, wherein the improved DDPG algorithm model includes improving its policy network for the DDPG algorithm model; the improved policy network is used to generate corresponding actions according to the distribution network state, wherein the distribution network state refers to the node voltage state of the distribution network, and the action refers to a control signal for controlling reactive power injection of the distribution network. The improved policy network generates corresponding actions according to the distribution network state, including:

[0008] S101: Generate a weight recovery matrix and a bias recovery matrix according to the following formula:

[0009] ,

[0010] ,

[0011] in, is the weight recovery matrix, It means to generate an upper triangular matrix, where the elements on the diagonal and below are 0 and the elements above are 1; It means to generate an upper triangular matrix, where the elements of the diagonal and the two rows below it are 0, and the elements above it are 1; is a 2-fold identity matrix, that is, the elements on the diagonal are 2 and the other elements are 0; is the operation of upper triangular matrices, is the offset from the main diagonal, is the identity matrix, is the bias recovery matrix;

[0012] S102, restore the weight matrix The forward weight matrix is ​​calculated according to the following formula and the negative weight matrix :

[0013] ,

[0014] in, is the forward weight matrix, is the negative weight matrix, is a parameter;

[0015] S103, calculate the forward bias matrix according to the following formula and the negative bias matrix :

[0016] ,

[0017] ,

[0018] in, is the forward bias matrix, is the negative bias matrix, is the bias reference matrix, and are the upper and lower voltage limits, is a matrix of all 1s;

[0019] S104, the forward weighted matrix and the distribution network status at time t After multiplication, the Relu activation function is activated and the L1 norm of the result is calculated, and the negative weighted matrix and the distribution network status at time t Multiply and add the result of L1 norm calculation as the first control signal; then calculate the difference between the positive bias matrix, negative bias matrix and bias reference matrix respectively, and multiply the two differences by the distribution network state at time t. The second control signal is obtained by multiplying the two control signals, and the two control signals are added to obtain the final control signal for controlling the reactive power injection of the distribution network.

[0020] Optionally, the improved DDPG algorithm model includes improving the value network of the DDPG algorithm model, where the improved value network is used to generate a corresponding Q value according to the distribution network state and the action generated by the policy network. The improved value network includes a normalization function, a first hidden layer, a second hidden layer, and an output layer, and the function expression for generating the corresponding Q value is:

[0021] ,

[0022] in, Represents the value network according to the distribution network status at time t ,action The generated Q value, is the weight matrix of the output layer; is the bias vector of the output layer, is the output of the second hidden layer, and:

[0023] ,

[0024] ,

[0025] ,

[0026] in, is the Leaky ReLU activation function, is the normalization operation, is the weight matrix of the second hidden layer; is the bias vector of the second hidden layer, is the output of the first hidden layer, The bias vector of the first hidden layer, is the weight matrix of the state input, is the weight matrix of the action input, is the normalized function of the distribution network state at time t The normalized result of is the action of the normalized function at time t The normalized result of is the slope parameter of the negative semi-axis.

[0027] Optionally, the training using the improved DDPG algorithm model includes:

[0028] S201, initialize the policy network, value network and their respective target networks. The parameters of the target networks are the same as those of the original networks; initialize the experience replay buffer;

[0029] S202, by interacting with the distribution network environment to obtain the current state of the distribution network, generate corresponding actions based on the current state of the distribution network through the policy network, obtain the reward of the action and the next state, and put the current state, action, reward and next state into the experience replay buffer as an experience; the function expression for obtaining the reward of the action is:

[0030] ,

[0031] in, For nodes The reward at time t, For nodes The voltage deviation at time t is For nodes The control input at time t is is the voltage deviation, is the action cost, is the balance coefficient of voltage deviation, is the balance coefficient of action cost, For nodes The target voltage value The average of , and:

[0032] ,

[0033] ,

[0034] ,

[0035] in, is the voltage at time t+1, is the conductivity matrix of the power grid, is the reactive power injection at time t+1; is the impact of the external environment on the voltage, which includes the reactive power injection of the node at time t and time t+1; and Node Reactive power injection at time t and time t+1, is the interval time, for The control strategy function, is a node Strategy parameters of the control strategy function;

[0036] S203, based on the priority experience playback mechanism, experience sampling is performed from the experience playback buffer, including sampling N experiences from the experience playback buffer according to priority, and the probability of each experience being sampled is:

[0037] ,

[0038] ,

[0039] ,

[0040] in, is the probability that experience i is sampled, is the priority value of experience i, is the priority adjustment factor, is the error value, is the TD error of experience i, is the reward for experience i; is a discount factor used to control the impact of future rewards on current value; Target Value Network The state at time t+1 and target strategies The Q-value of the generated action, The value network is based on the distribution network status at time t and actions The obtained Q value;

[0041] S204, combining the sampled experience with the loss function to update the parameters of the policy network and the value network through back propagation;

[0042] S205, soft-update the parameters of the target networks of the strategy network and the value network respectively;

[0043] S206, jump to step S203 until the preset training end condition is reached.

[0044] Optionally, in step S204, the function expression of the loss function used to update the parameters of the value network is:

[0045] ,

[0046] ,

[0047] ,

[0048] in, The loss function used to update the parameters of the value network, is the expected value, is the importance sampling weight of experience i, is the probability that experience i is sampled, is the size of the experience replay buffer, For the value network based on the current state and actions and parameters The obtained Q value is is the target value of the current experience, is the parameter used to adjust the weight; is a discount factor used to control the impact of future rewards on current actions; The target network of the value network is based on the state at time t+1 and actions The obtained Q value is The state of the policy network at time t+1 Generated actions.

[0049] Optionally, in step S204, the function expression of the loss function used to update the parameters of the policy network is:

[0050] ,

[0051] ,

[0052] in, The loss function used to update the parameters of the policy network, is the expected value, The value network is based on the distribution network status at time t and actions The obtained Q value is For the policy network based on the current state and parameters The action obtained; is the weight parameter of entropy regularization, which is used to adjust the trade-off between exploration and utilization in strategy optimization; is the entropy of the strategy, is the distribution network state of the strategic network at time t The probability distribution of taking action a.

[0053] Optionally, in step S205, the function expressions of the parameters of the target networks of the soft update strategy network and the value network are:

[0054] ,

[0055] in, are the parameters of the target network after soft update, For soft updates, is the parameter of the target network corresponding to the strategy network or value network, are the parameters of the target network before soft update; is the step size of the soft update, which controls the update speed of the target network.

[0056] Optionally, when initializing the policy network, the value network, and their respective target networks in step S201, noise is added to the output action of the policy network to generate exploratory actions. The function expression is:

[0057] ,

[0058] ,

[0059] in, Current status The deterministic action under For the strategy network, and They are and The noise value at the moment, is the mean of the noise; is the mean reversion speed, which is used to control the rate at which the noise gradually returns to the mean; is the random amplitude of the noise; Refers to white noise sampled from a standard normal distribution.

[0060] In addition, the present invention also provides a distribution network safety constraint voltage control system based on deep reinforcement learning, including an interconnected microprocessor and a memory, and the microprocessor is programmed or configured to execute the distribution network safety constraint voltage control method based on deep reinforcement learning.

[0061] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the distribution network safety constraint voltage control method based on deep reinforcement learning through a processor.

[0062] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the distribution network safety constraint voltage control method based on deep reinforcement learning through a processor.

[0063] Compared with the prior art, the present invention can mainly achieve the following beneficial effects: the present invention includes the training of an improved DDPG algorithm model, and the improved DDPG algorithm model includes improving its strategy network for the DDPG algorithm model, and the improved strategy network generates corresponding actions according to the distribution network state, including: generating a weight recovery matrix and a bias recovery matrix, calculating the positive and negative weight matrices and bias matrices and comparing them with the distribution network state at time t Two control signals are generated by fusion and added together to obtain the final control signal used to control reactive power injection in the distribution network. Through this approach, structurally controllable weight and bias recovery matrices are introduced into the improved strategy network, thereby explicitly regulating the mapping process between state and action, ensuring that the generated control instructions can effectively guide the voltage to the desired range (e.g., 0.95-1.05 pu). The node voltage state of the distribution network is used as the input of the strategy network. After being processed by the weighted matrix and bias matrix constructed by the upper triangular structure, a directional weight matrix and a corresponding bias recovery matrix are generated. The above matrices are used to model the control behavior when the voltage is higher than the upper limit and lower than the lower limit, respectively, thereby outputting a suppressive reactive injection signal and an enhanced reactive injection signal, respectively. When the node voltage is higher than the upper limit, the positive weight and bias recovery matrix are activated, so that the positive control amount takes effect and a signal for lowering the voltage is generated; when the node voltage is lower than the lower limit, the negative weight and bias recovery matrix are activated, so that the negative control amount takes effect and a signal for increasing the voltage is generated; if the voltage is in the safe range, the outputs of the two control signals are close to 0, and the action tends to be stable or no injection. Therefore, the present invention can solve the problems that traditional deep reinforcement learning is sensitive to environmental noise, easily affected by noise interference under dynamic changes in the power grid or external disturbances, affecting control accuracy and system stability, and difficult to ensure voltage stability in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of the network structure of the improved policy network in an embodiment of the present invention.

[0065] Figure 2 Schematic diagram of the network structure of the improved value network in an embodiment of the present invention.

[0066] Figure 3 Schematic diagram of the workflow of the improved DDPG algorithm model in an embodiment of the present invention.

[0067] Figure 4 Schematic diagram of the working principle of the improved DDPG algorithm model in an embodiment of the present invention.

[0068] Figure 5 Comparison of the average rewards of the improved DDPG algorithm model and the original DDPG algorithm model in this embodiment of the present invention.

[0069] Figure 6 This is the voltage control effect of the improved DDPG algorithm model in the embodiment of the present invention and the existing safe-DDPG algorithm model on node 2.

[0070] Figure 7 This is the voltage control effect of the improved DDPG algorithm model in the embodiment of the present invention and the existing safe-DDPG algorithm model on node 7.

[0071] Figure 8 This is the voltage control effect of the improved DDPG algorithm model in the embodiment of the present invention and the existing safe-DDPG algorithm model on node 9. DETAILED DESCRIPTION

[0072] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention. This embodiment provides a distribution network safety constraint voltage control method based on deep reinforcement learning, comprising the following steps: using an improved DDPG algorithm model for training, and the improved DDPG algorithm model includes improving its policy network for the DDPG algorithm model; the improved policy network is used to generate corresponding actions according to the distribution network state, the distribution network state refers to the node voltage state of the distribution network, and the action refers to the control signal for controlling the reactive power injection of the distribution network.

[0073] The improved strategy network structure optimization in this embodiment mainly focuses on introducing security constraints and enhancing network flexibility to meet the requirements of voltage control, aiming to ensure stability and efficiency in the voltage control process. Figure 1 As shown, the improved strategy network in this embodiment generates corresponding actions according to the distribution network status, including:

[0074] S101: Generate a weight recovery matrix and a bias recovery matrix according to the following formula:

[0075] ,

[0076] ,

[0077] in, is the weight recovery matrix, It means to generate an upper triangular matrix, where the elements on the diagonal and below are 0 and the elements above are 1; It means to generate an upper triangular matrix, where the elements of the diagonal and the two rows below it are 0, and the elements above it are 1; is a 2-fold identity matrix, that is, the elements on the diagonal are 2 and the other elements are 0; is the operation of upper triangular matrices, is the offset from the main diagonal, is the identity matrix, is the bias recovery matrix;

[0078] S102, restore the weight matrix The forward weight matrix is ​​calculated according to the following formula and the negative weight matrix :

[0079] ,

[0080] in, is the forward weight matrix, is the negative weight matrix, is a parameter;

[0081] S103, calculate the forward bias matrix according to the following formula and the negative bias matrix :

[0082] ,

[0083] ,

[0084] in, is the forward bias matrix, is the negative bias matrix, is the bias reference matrix, and are the upper and lower voltage limits, is a matrix of all 1s;

[0085] S104, the forward weighted matrix and the distribution network status at time t After multiplication, the Relu activation function is activated and the L1 norm of the result is calculated, and the negative weighted matrix and the distribution network status at time t Multiply and add the result of L1 norm calculation as the first control signal; then calculate the difference between the positive bias matrix, negative bias matrix and bias reference matrix respectively, and multiply the two differences by the distribution network state at time t. The second control signal is obtained by multiplying the two control signals, and the two control signals are added to obtain the final control signal for controlling the reactive power injection of the distribution network.

[0086] The improved strategy network in this embodiment firstly calculates the distribution network status at time t based on the traditional strategy network. Introducing the weight recovery matrix and the bias recovery matrix , The goal is to restore the monotonicity of the network by adjusting the matrix structure, thereby ensuring that the voltage control does not generate negative feedback, By adjusting the bias term, the effectiveness and rationality of the control signal are improved. and Matrix to obtain the forward weight matrix and the negative weight matrix , and the forward bias matrix and the negative bias matrix The goal is to control the monotonicity of the network through a special matrix construction, thereby making the output more stable, ensuring that the voltage control does not generate negative feedback, and avoiding unreasonable control signals by adjusting the bias term. On this basis, the ReLU activation function is used to process the state, further combining the weight matrix and bias output. Applied to the weighted results, negative values ​​are clipped to zero, ensuring non-negative output of the network. The ReLU activation function effectively handles the nonlinear characteristics of the network, avoids the gradient vanishing problem, and enhances the network's expressive power. The ReLU activation function can be expressed as: ,in The input matrix, i.e. the previous series of weight matrices and bias matrices, is activated by the ReLU function, and the output value is non-negative. The function expression of the L1 norm of the calculated result can be expressed as:

[0087] ,

[0088] in, is the output result, C is the input result, It is an operation that limits the tensor to a given range. It is used in this network to ensure that the input result C does not have negative or excessive values, and is normalized by L1 norm regularization to ensure that the output is within a reasonable range. It is a hyperparameter used to scale the bias term and controls the size of the bias term. is the L1 norm of the input result C. Ensure that the magnitude of the bias term is within a reasonable range after normalization. Through the above operations, the bias term in the network will remain within a reasonable range during each update, preventing them from being too large or too small.

[0089] As an optional implementation method, the improved DDPG algorithm model in this embodiment includes improving the value network of the DDPG algorithm model. The improved value network is used to generate corresponding Q values ​​according to the state of the distribution network and the actions generated by the policy network. Layer normalization is added to the layers of the original value network in combination with residual connections. Layer normalization is a regularization technology that avoids gradient vanishing or gradient explosion problems by standardizing the activation values ​​of each layer. It is applied to the output of each layer to ensure that the output of each layer remains within a suitable range during training, thereby accelerating training and stabilizing the learning process. Residual connections are then superimposed on the basis of layer normalization. The main purpose of residual connections is to solve the gradient vanishing problem in deep network training and accelerate convergence. It introduces a "skip connection" to allow the input to flow directly to subsequent layers. As Figure 2As shown, the improved value network in this embodiment includes a normalization function, a first hidden layer, a second hidden layer, and an output layer, and the function expression for generating the corresponding Q value is:

[0090] ,

[0091] in, Represents the value network according to the distribution network status at time t ,action The generated Q value, is the weight matrix of the output layer; is the bias vector of the output layer, is the output of the second hidden layer, and:

[0092] ,

[0093] ,

[0094] ,

[0095] in, is the Leaky ReLU activation function, is the normalization operation, is the weight matrix of the second hidden layer; is the bias vector of the second hidden layer, is the output of the first hidden layer, The bias vector of the first hidden layer, is the weight matrix of the state input, is the weight matrix of the action input, is the normalized function of the distribution network state at time t The normalized result of is the action of the normalized function at time t The normalized result of is the slope parameter of the negative half axis. The purpose of the value network is to evaluate , first normalize the input state:

[0096] , .

[0097] The hidden layer of the value network then uses a residual connection with layer normalization, and on top of this, a Leaky ReLU activation function is applied. This function addresses the "dead neuron" problem that can occur with traditional ReLU activation functions. The Leaky ReLU activation function introduces a small negative slope to avoid "dead zones" in negative regions, allowing neurons to maintain a certain gradient even when the input is negative:

[0098] ,

[0099] in is the slope of the negative half-axis (typically a small constant). The Leaky ReLU activation function is applied to each layer of the network to ensure a small slope on the negative side, thus avoiding the dead neuron problem. Finally, the corresponding Q value is generated through the first hidden layer, the second hidden layer, and the output layer.

[0100] like Figure 3 As shown, the training of the improved DDPG algorithm model in this embodiment includes:

[0101] S201, initialize the policy network, value network and their respective target networks. The parameters of the target networks are the same as those of the original networks; initialize the experience replay buffer;

[0102] S202, obtaining the current state of the distribution network by interacting with the distribution network environment, generating corresponding actions based on the current state of the distribution network through the policy network, obtaining the reward and next state of the action, and storing the current state, action, reward, and next state as an experience in the experience replay buffer;

[0103] S203, based on the priority experience playback mechanism, experience sampling is performed from the experience playback buffer, including sampling N experiences from the experience playback buffer according to priority, and the probability of each experience being sampled is:

[0104] ,

[0105] ,

[0106] ,

[0107] in, is the probability that experience i is sampled, is the priority value of experience i, is the priority adjustment factor, is the error value, is the TD error of experience i, is the reward for experience i; is a discount factor used to control the impact of future rewards on current value; Target Value Network The state at time t+1 and target strategies The Q-value of the generated action, The value network is based on the distribution network status at time t and actions The Q value obtained; the experience replay buffer improves the sample utilization efficiency by increasing the sampling probability of key experiences, making the strategy optimization more efficient;

[0108] S204, combining the sampled experience with the loss function to update the parameters of the policy network and the value network through back propagation;

[0109] S205, soft-update the parameters of the target networks of the strategy network and the value network respectively;

[0110] S206, jump to step S203 until the preset training end condition is reached.

[0111] like Figure 4 As shown, the strategy network, value network and their respective target networks initialized in step S201 of this embodiment are respectively represented as: strategy network , strategic target network , Value Network and value target network ,in Indicates status, Representing actions, policy networks Responsible for generating the distribution network status at time t Deterministic actions under , policy network By parameters Definition, Action Directly interact with the environment to determine the reactive power injection action of the distribution network; value network Responsible for evaluating the distribution network status at time t and action Impact on the agent's long-term returns (Q-value), value network The output of represents the distribution network state at time t under the current strategy. Start executing the action The cumulative rewards that can be obtained, the value network By parameters Definition; Policy Target Network By parameters Definition; Value Goal Network By parameters Define and implement a strategic network The output action of is used to add noise to generate exploratory actions; the network is randomly initialized. Parameters And assign it to the strategy target network Parameters , Value Network Parameters And assign it to the value target network Parameters To ensure the stability of training, it can be expressed as: . Policy Target Network and value target network To calculate the target value , whose weights serve as delayed copies of the main network's parameters (soft updates). With this setup, the main network is used for policy generation and value evaluation, while the target network provides a stable training target for the main network by slowly changing. This then enables the soft update mechanism to gradually adjust.

[0112] To improve sampling efficiency and learning effectiveness, this embodiment employs a prioritized replay buffer (PRB) mechanism in addition to the DDPG algorithm. This mechanism prioritizes sampling of different experiences based on their importance (measured by TD error), allowing key experiences to be used more frequently. The experience is stored as follows: After each interaction with the environment, a complete five-tuple is stored:

[0113] ,

[0114] in, is the distribution network status at time t, including voltage and reactive power , The actions generated by the current policy network, such as reactive power adjustment, The immediate reward calculated according to the cost function, is the state of the distribution network at time t+1, that is, the next state of the system after the action is executed, A flag indicating whether the process is finished.

[0115] In step S202 of this embodiment, the function expression for obtaining the reward of the action is:

[0116] ,

[0117] in, For nodes The reward at time t, For nodes The voltage deviation at time t is For nodes The control input at time t is is the voltage deviation, is the action cost, is the balance coefficient of voltage deviation, is the balance coefficient of action cost, For nodes The target voltage value The average of , and:

[0118] ,

[0119] ,

[0120] ,

[0121] in, is the voltage at time t+1, is the conductivity matrix of the power grid, is the reactive power injection at time t+1; is the impact of the external environment on the voltage, which includes the reactive power injection of the node at time t and time t+1; and Node Reactive power injection at time t and time t+1, is the interval time, for The control strategy function, is a node Strategy parameters of the control strategy function;

[0122] The function expression of the loss function used to update the parameters of the value network in step S204 of this embodiment is:

[0123] ,

[0124] ,

[0125] ,

[0126] in, The loss function used to update the parameters of the value network, is the expected value, is the importance sampling weight of experience i, is the probability that experience i is sampled, is the size of the experience replay buffer, For the value network based on the current state and actions and parameters The obtained Q value is is the target value of the current experience, is the parameter used to adjust the weight; is a discount factor used to control the impact of future rewards on current actions; The target network of the value network is based on the state at time t+1 and actions The obtained Q value is The state of the policy network at time t+1 Generated actions.

[0127] In step S204 of this embodiment, the function expression of the loss function used to update the parameters of the policy network is:

[0128] ,

[0129] ,

[0130] in, The loss function used to update the parameters of the policy network, is the expected value, The value network is based on the distribution network status at time t and actions The obtained Q value is For the policy network based on the current state and parameters The action obtained; is the weight parameter of entropy regularization, which is used to adjust the trade-off between exploration and utilization in strategy optimization; is the entropy of the strategy, is the distribution network state of the strategic network at time t The probability distribution of taking action a.

[0131] like Figure 4 As shown, in step S205 of this embodiment, the function expressions of the parameters of the target networks of the soft update strategy network and the value network are:

[0132] ,

[0133] in, are the parameters of the target network after soft update, For soft updates, is the parameter of the target network corresponding to the strategy network or value network, are the parameters of the target network before soft update; is the step size of the soft update, which controls the update speed of the target network.

[0134] The actions generated by the policy network are deterministic and lack randomness. Therefore, in order to enhance the agent's ability to explore the action space, the improved algorithm introduces the Ornstein-Uhlenbeck (OU) noise process. OU noise has time correlation and the generated noise is smoother, which helps to achieve more effective exploration in continuous control tasks. Figure 4As shown, in step S201 of this embodiment, when initializing the policy network, the value network, and their respective target networks, noise is added to the output action of the policy network to generate exploratory actions. The function expression is:

[0135] ,

[0136] ,

[0137] in, Current status The deterministic action under For the strategy network, and They are and The noise value at the moment, is the mean of the noise; is the mean reversion speed, which is used to control the rate at which the noise gradually returns to the mean; is the random amplitude of the noise; Refers to white noise sampled from a standard normal distribution. By introducing OU noise, the agent (the improved DDPG algorithm model) can conduct extensive exploration in the early stages of training. Meanwhile, the noise amplitude gradually decreases over time, helping to focus on high-quality actions in the later stages of training.

[0138] Considering that in the voltage control problem, the design of the reward function has an important impact on the performance of the reinforcement learning algorithm. In order to verify the effectiveness of the improved DDPG algorithm in voltage control, this embodiment compares and analyzes the reward functions of the improved DDPG algorithm model and the original DDPG algorithm model. Based on the original DDPG algorithm model, the improved DDPG algorithm model optimizes the design of the reward function for the voltage control scenario, so that it can better balance voltage stability and control efficiency. Specifically, the reward function of the improved DDPG algorithm model not only takes into account the penalty of voltage deviation, but also introduces additional constraints such as voltage fluctuation smoothness and the economy of control actions, thereby more comprehensively reflecting the needs of actual power grid operation. By comparing the reward function performance of the two algorithms under different voltage interference scenarios, the advantages of the improved DDPG algorithm model in voltage control tasks can be intuitively evaluated. Figure 5This is a comparison of the average rewards of the improved DDPG algorithm model of this embodiment and the original DDPG algorithm model, which shows the comparison of the reward function convergence, stability and final control effect between the improved DDPG algorithm model of this embodiment and the original DDPG algorithm model (DDPG). In the voltage control problem, the DDPG algorithm is usually used to train the intelligent agent so that it can effectively control the voltage to achieve the goal. It is observed that the reward value of the original DDPG (blue curve) performs poorly in the early stage, but as the training progresses, the reward value gradually increases and stabilizes. The optimized DDPG (orange curve) shows better performance at the beginning of training, and the reward value grows more steadily, and eventually tends to a higher level. From Figure 5 Several advantages of the improved DDPG algorithm can be reflected in the following: (1) Better initial performance: The improved DDPG algorithm model in this embodiment quickly improved the reward value in the initial 12 training rounds, indicating that through algorithm optimization, the intelligent agent can adapt to the environment more quickly and take appropriate actions in the voltage control task. This is particularly important for real-time control in power systems and can effectively reduce system fluctuations and instability. (2) Improved stability: The improved DDPG algorithm model shows a smoother reward curve. This shows that the optimized algorithm can learn the voltage control strategy more stably, avoid the volatility of the original algorithm in some training rounds, and thus improve the reliability of the system. (3) Higher final reward: The reward value of the improved DDPG algorithm model in the later stage of training is significantly higher than that of the original DDPG algorithm model, starting from the first step. This shows that the optimized algorithm can not only achieve better performance in the initial stage, but also can continuously improve its control strategy, thereby obtaining better control effects in the voltage control task. (4) Higher convergence speed: The improved DDPG algorithm model in this embodiment seems to converge faster and achieves higher reward values ​​in fewer training rounds. This means that through optimization measures, the improved DDPG algorithm model in this embodiment can achieve a more efficient voltage control strategy in a shorter time. This shows that in voltage control problems, the improved DDPG algorithm model can help improve system stability and control efficiency, reduce voltage fluctuations, and respond more quickly to external changes, thereby improving the overall performance of the power system.

[0139] To verify the performance advantages of the deep reinforcement learning-based distribution network security-constrained voltage control method in voltage control, this embodiment conducted experiments on the IEEE 13-bus test feeder, focusing on the voltage control effects at nodes 2, 7, and 9. Figure 6 This is the voltage control effect of the improved DDPG algorithm model of this embodiment and the existing safe-DDPG algorithm model on node 2. Figure 7This is the voltage control effect of the improved DDPG algorithm model of this embodiment and the existing safe-DDPG algorithm model on node 7. Figure 8 The voltage control effect of the improved DDPG algorithm model of this embodiment and the existing safe-DDPG algorithm model on node 9. Figures 6 to 8 In the graph for node 9, the optimized algorithm ensures that the voltage remains below the specified upper limit (1.05 pu), while the safe-DDPG algorithm model exhibits some initial voltage overshoot, particularly in the first 200 time steps. This demonstrates that the optimized algorithm better controls voltage and reduces the risk of system overshoot, which is crucial for voltage control, especially in preventing equipment damage or system instability. The improved DDPG algorithm model in this embodiment stabilizes the voltage and maintains it near the target voltage, demonstrating its greater stability in dynamic control. While the original algorithm ultimately stabilizes the voltage, its convergence process is relatively slow and prone to fluctuations. In summary, the improved DDPG algorithm model of this embodiment converges to the target voltage more quickly than the safe-DDPG algorithm model in voltage control tasks; maintains the voltage more stably within the set range, reducing the risk of overshoot; and improves system stability and control accuracy, particularly when controlling multi-node voltage. This demonstrates that the improved DDPG algorithm model of this embodiment, through optimized policy network, value network, and reward function design, enables it to more quickly and stably control voltage within the safe range (i.e., below 1.05 pu). Under the same high-voltage scenario, the improved DDPG algorithm model of this embodiment is able to adjust the voltages of nodes 2, 7, and 9 from exceeding 1.05 pu to within the safe range more quickly than the safe-DDPG algorithm model, while also demonstrating better convergence and stability. This demonstrates the significant advantages of the improved DDPG algorithm model of this embodiment in voltage control tasks.

[0140] In summary, the distribution network safety constraint voltage control method based on deep reinforcement learning in this embodiment provides higher reliability and stability guarantees for voltage control by introducing safety constraints and stability optimization mechanisms. The core of our method is to ensure that the control signal is always within the safety range through improved policy network design and dynamic adjustment mechanism, while significantly improving the convergence speed and adaptability of the algorithm. Experimental results show that the distribution network safety constraint voltage control method based on deep reinforcement learning in this embodiment can not only effectively deal with the voltage fluctuation problems caused by high-proportion photovoltaic penetration and load fluctuations, but also shorten the response time by more than one third, while always achieving voltage stability. The distribution network safety constraint voltage control method based on deep reinforcement learning in this embodiment provides an efficient and reliable solution for power system voltage control, and lays the foundation for future research combining other control theory optimization principles.

[0141] In addition, the present invention also provides a distribution network safety constraint voltage control system based on deep reinforcement learning, including an interconnected microprocessor and a memory, and the microprocessor is programmed or configured to execute the distribution network safety constraint voltage control method based on deep reinforcement learning.

[0142] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the distribution network safety constraint voltage control method based on deep reinforcement learning through a processor.

[0143] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the distribution network safety constraint voltage control method based on deep reinforcement learning through a processor.

[0144] Those skilled in the art should understand that the technical solutions that can be provided by the embodiments of the present invention may be in the form of methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment that combines software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0145] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A distribution network security constraint voltage control method based on deep reinforcement learning, characterized in that: The method comprises the following steps: adopting an improved DDPG algorithm model for training, wherein the improved DDPG algorithm model comprises improving a strategy network for the DDPG algorithm model, wherein the improved strategy network is used to generate corresponding actions according to a distribution network state, wherein the distribution network state refers to a node voltage state of the distribution network, and the action refers to a control signal for controlling reactive power injection of the distribution network. The improved strategy network generates corresponding actions according to the distribution network state, including: S101: Generate a weight recovery matrix and a bias recovery matrix according to the following formula: ， ， in, is the weight recovery matrix, It means to generate an upper triangular matrix, where the elements on the diagonal and below are 0 and the elements above are 1; It means to generate an upper triangular matrix, where the elements of the diagonal and the two rows below it are 0, and the elements above it are 1; is a 2-fold identity matrix, that is, the elements on the diagonal are 2 and the other elements are 0; is the operation of upper triangular matrices, is the offset from the main diagonal, is the identity matrix, is the bias recovery matrix; S102, restore the weight matrix The forward weight matrix is ​​calculated according to the following formula and the negative weight matrix : ， in, is the forward weight matrix, is the negative weight matrix, is a parameter; S103, calculate the forward bias matrix according to the following formula and the negative bias matrix : ， ， in, is the forward bias matrix, is the negative bias matrix, is the bias reference matrix, and are the upper and lower voltage limits, is a matrix of all 1s; S104, the forward weighted matrix and the distribution network status at time t After multiplication, the Relu activation function is activated and the L1 norm of the result is calculated, and the negative weighted matrix and the distribution network status at time t Multiply and add the result of L1 norm calculation as the first control signal; then calculate the difference between the positive bias matrix, negative bias matrix and bias reference matrix respectively, and multiply the two differences by the distribution network state at time t. The second control signal is obtained by multiplying the two control signals, and the two control signals are added to obtain the final control signal for controlling the reactive power injection of the distribution network.

2. The distribution network safety constraint voltage control method based on deep reinforcement learning according to claim 1 is characterized in that: The improved DDPG algorithm model includes improving its value network for the DDPG algorithm model. The improved value network is used to generate a corresponding Q value according to the distribution network state and the action generated by the policy network. The improved value network includes a normalization function, a first hidden layer, a second hidden layer, and an output layer. The function expression for generating the corresponding Q value is: ， in, Represents the value network according to the distribution network status at time t ,action The generated Q value, is the weight matrix of the output layer; is the bias vector of the output layer, is the output of the second hidden layer, and: ， ， ， in, is the Leaky ReLU activation function, is the normalization operation, is the weight matrix of the second hidden layer; is the bias vector of the second hidden layer, is the output of the first hidden layer, The bias vector of the first hidden layer, is the weight matrix of the state input, is the weight matrix of the action input, is the normalized function of the distribution network state at time t The normalized result of is the action of the normalized function at time t The normalized result of is the slope parameter of the negative semi-axis.

3. The distribution network safety constraint voltage control method based on deep reinforcement learning according to claim 1 is characterized in that: The training of the improved DDPG algorithm model includes: S201, initialize the policy network, value network and their respective target networks. The parameters of the target networks are the same as those of the original networks; initialize the experience replay buffer; S202, by interacting with the distribution network environment to obtain the current state of the distribution network, generate corresponding actions based on the current state of the distribution network through the policy network, obtain the reward of the action and the next state, and put the current state, action, reward and next state into the experience replay buffer as an experience; the function expression for obtaining the reward of the action is: ， in, For nodes The reward at time t, For nodes The voltage deviation at time t is For nodes The control input at time t is is the voltage deviation, is the action cost, is the balance coefficient of voltage deviation, is the balance coefficient of action cost, For nodes Target voltage value The average of , and: ， ， ， in, is the voltage at time t+1, is the conductivity matrix of the power grid, is the reactive power injection at time t+1; is the impact of the external environment on the voltage, which includes the reactive power injection of the node at time t and time t+1; and Node Reactive power injection at time t and time t+1, is the interval time, for The control strategy function, is a node Strategy parameters of the control strategy function; S203, based on the priority experience playback mechanism, experience sampling is performed from the experience playback buffer, including sampling N experiences from the experience playback buffer according to priority, and the probability of each experience being sampled is: ， ， ， in, is the probability that experience i is sampled, is the priority value of experience i, is the priority adjustment factor, is the error value, is the TD error of experience i, is the reward for experience i; is a discount factor used to control the impact of future rewards on current value; Target Value Network The state at time t+1 and target strategies The Q-value of the generated action, The value network is based on the distribution network status at time t and actions The obtained Q value; S204, combining the sampled experience with the loss function to update the parameters of the policy network and the value network through back propagation; S205, soft-update the parameters of the target networks of the strategy network and the value network respectively; S206, jump to step S203 until the preset training end condition is reached.

4. The distribution network security constraint voltage control method based on deep reinforcement learning according to claim 3 is characterized in that: In step S204, the loss function used to update the parameters of the value network is expressed as follows: ， ， ， in, The loss function used to update the parameters of the value network, is the expected value, is the importance sampling weight of experience i, is the probability that experience i is sampled, is the size of the experience replay buffer, For the value network based on the current state and actions and parameters The obtained Q value is is the target value of the current experience, is the parameter used to adjust the weight; is a discount factor used to control the impact of future rewards on current actions; The target network of the value network is based on the state at time t+1 and actions The obtained Q value is The state of the policy network at time t+1 Generated actions.

5. The distribution network security constraint voltage control method based on deep reinforcement learning according to claim 3 is characterized in that: In step S204, the function expression of the loss function used to update the parameters of the policy network is: ， ， in, The loss function used to update the parameters of the policy network, is the expected value, The value network is based on the distribution network status at time t and actions The obtained Q value is For the policy network based on the current state and parameters The action obtained; is the weight parameter of entropy regularization, which is used to adjust the trade-off between exploration and utilization in strategy optimization; is the entropy of the strategy, is the distribution network state of the strategic network at time t The probability distribution of taking action a.

6. The method for voltage control in a distribution network with safety constraints based on deep reinforcement learning according to claim 3 is characterized in that: In step S205, the function expressions of the parameters of the target networks of the soft update strategy network and the value network are: ， in, are the parameters of the target network after soft update, For soft updates, is the parameter of the target network corresponding to the strategy network or value network, are the parameters of the target network before soft update; is the step size of the soft update, which controls the update speed of the target network.

7. The method for voltage control in a distribution network with safety constraints based on deep reinforcement learning according to claim 3 is characterized in that: When initializing the policy network, the value network, and their respective target networks in step S201, noise is added to the output action of the policy network to generate exploratory actions. The function expression is: ， ， in, Current status The deterministic action under For the strategy network, and They are and The noise value at the moment, is the mean of the noise; is the mean reversion speed, which is used to control the rate at which the noise gradually returns to the mean; is the random amplitude of the noise; Refers to white noise sampled from a standard normal distribution.

8. A distribution network security constraint voltage control system based on deep reinforcement learning, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the distribution network security constraint voltage control method based on deep reinforcement learning as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the distribution network security constraint voltage control method based on deep reinforcement learning as described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the distribution network security constraint voltage control method based on deep reinforcement learning as described in any one of claims 1 to 7 through a processor.

Images