Low-voltage distribution network voltage control method and system based on deep reinforcement learning

Abstract

The application discloses a low-voltage distribution network voltage control method and system based on deep reinforcement learning, comprising the following steps: embedding a trained Transform-SAC model into a low-voltage distribution network, making the Transform-SAC model interact with the environment of the low-voltage distribution network to form an action strategy, and issuing the action strategy to a switch or a smart controller on the user side to control the switching of power and reactive power compensators of a distributed voltage, wherein the distributed power supply equipment and the reactive power compensator execute actions after receiving the command and form new environment data to feed back to the model for self-optimization in real time. The application adopts an SAC algorithm to optimize the voltage control strategy of the low-voltage distribution network, improves the learning efficiency through experience replay and strategy optimization, feeds back the interaction between the model and the environment of the low-voltage distribution network, and continuously optimizes the control strategy and adjusts the power grid equipment of an agent, so that the voltage regulation effect is continuously improved.

Description

Technical Field

[0001] This invention relates to the field of power system automation control technology, specifically to a voltage control method and system for low-voltage distribution networks based on deep reinforcement learning. Background Technology

[0002] With the large-scale integration of distributed renewable energy sources, voltage control in low-voltage distribution networks faces new challenges. Low-voltage distribution networks typically handle the electricity demands of residential, commercial, and small industrial users, and these loads exhibit high dynamism and uncertainty. Simultaneously, the increasing integration of distributed generation units such as photovoltaics and electric vehicles further complicates and unpredicts the operating environment of low-voltage distribution networks.

[0003] Existing voltage control technologies typically rely on pre-defined rules and empirical models, often exhibiting low robustness and accuracy when faced with high-dimensional state spaces and complex network topologies. Particularly in low-voltage distribution networks, due to the highly dynamic nature of the network topology and the stochasticity of power flow, traditional voltage control methods are no longer sufficient to meet the operational requirements of the current power grid.

[0004] Reinforcement learning, through continuous interaction with the environment, can continuously optimize control strategies and gradually achieve adaptive adjustments, thus possessing certain advantages in handling complex and uncertain low-voltage distribution network environments. However, reinforcement learning algorithms also face some challenges in practical applications. First, the state space of distribution networks typically has high dimensionality, and traditional deep reinforcement learning methods may struggle to effectively handle this high-dimensional data, leading to low training efficiency and slow convergence. Second, the topology of low-voltage distribution networks is often complex and dynamically changing, which makes control strategies based on classical models lack sufficient flexibility and generalization ability.

[0005] Existing multi-timescale distribution network voltage optimization control methods (such as Chinese patent document CN202410803630.6) and multi-timescale voltage regulation methods based on deep reinforcement learning (such as Chinese patent document CN202411696765.3) construct voltage control models using conventional deep learning methods. However, when faced with distribution network load fluctuations or sudden events, such as distribution network topology changes and equipment failures, they cannot fully capture the complex nonlinear characteristics of the system, and the voltage regulation effect needs to be further improved. Summary of the Invention

[0006] The purpose of this invention is to provide a voltage control method and system for low-voltage distribution networks based on deep reinforcement learning. This method optimizes the voltage regulation problem caused by the large-scale integration of new energy sources into low-voltage distribution networks by introducing Transformer networks and the SAC algorithm. Specifically, global features are extracted using Transformer networks, and the control strategy is optimized using the SAC algorithm to achieve optimized voltage control of the low-voltage distribution network. As a powerful sequence modeling tool, the Transformer network can extract effective global features from high-dimensional data, overcoming the shortcomings of traditional deep learning methods in handling complex low-voltage distribution network states. The SAC algorithm enhances the system's exploration capability and stability by optimizing policy entropy and expected reward, thereby improving the accuracy and robustness of voltage control and providing a more efficient solution for low-voltage distribution network voltage control.

[0007] The present invention is achieved by at least one of the following technical solutions.

[0008] A voltage control method for low-voltage distribution networks based on deep reinforcement learning includes the following steps:

[0009] The trained Transformer-SAC model is embedded into the voltage control system of the low-voltage distribution network, so that the Transformer-SAC model interacts with the environment of the low-voltage distribution network to form action strategies.

[0010] The action strategy is sent to the execution equipment to control the power of the distributed voltage and the number of reactive power compensators switched on and off;

[0011] After receiving a command, the device executes an action, and the low-voltage distribution network forms new state variables. The new state variables are fed back to the Transformer-SAC model in real time for optimization.

[0012] The Transformer-SAC model interacts with the low-voltage distribution network environment to form action strategies, including the following steps:

[0013] S1. Use Transformer network to extract the correlation feature matrix of state variables of low voltage distribution network, and output the feature matrix after processing by Transformer network.

[0014] S2. Input the feature matrix output by the Transformer network into the SAC algorithm to generate the action policy.

[0015] Further, in step S1, a Transformer network is used to extract the correlation feature matrix from the state variables of the low-voltage distribution network. The correlation feature matrix is ​​then processed by the Transformer network to output a feature matrix, including the following steps:

[0016] S1.1 Extract the node voltage, active power load, reactive power load, and node voltage sensitivity from the state variables of the low-voltage distribution network as the correlation feature matrix;

[0017] S1.2 The correlation feature matrix calculates the attention weight of each attention head through a multi-head attention mechanism;

[0018] S1.3. Concatenate the weights of all attention heads to obtain a concatenated feature matrix and perform linear mapping; add the concatenated feature matrix to the associated feature matrix to achieve residual connection, and then process it through normalization to obtain the normalized feature matrix.

[0019] S1.4 Input the normalized feature matrix into the feedforward neural network to obtain the feature matrix processed by the feedforward neural network.

[0020] Further, in step S2, the SAC algorithm includes:

[0021] 1) The policy optimization objective of the SAC algorithm is to maximize the expected reward and policy entropy:

[0022] 2) Experience playback: Stores the interaction data between the agent and the environment, including the state, actions, and rewards of the environment feedback;

[0023] 3) Update Network: By minimizing Bellman error update of the function network;

[0024] 4) Update the policy network by maximizing the policy entropy and sum. Value update strategy network.

[0025] Furthermore, the reward function Calculated by the following formula:

[0026] ;

[0027] In the formula for Penalty value for voltage exceeding limit at any time. for Always control costs. for It is used to punish power losses in low-voltage distribution networks. for Time node voltage sensitivity penalty value, , , , These are the weights of the corresponding sub-items; In order to be in Decision-making actions at any moment for State variables of the low-voltage distribution network at any given time.

[0028] Furthermore, the training of the Transformer-SAC model includes the following steps:

[0029] (1) Based on historical data of low-voltage distribution network, a virtual environment of low-voltage distribution network is constructed in the computer. The Transformer network obtains state variables through the virtual environment, extracts the correlation feature matrix, and generates new state variables.

[0030] (2) Input the new state variables into the action policy obtained by the SAC algorithm and output the action;

[0031] (3) The execution equipment performs actions to change the state of the virtual environment of the low-voltage distribution network, and the changed state variables are fed back to the Transformer-SAC model;

[0032] (4) The Transformer-SAC model uses the changed state variables for optimization;

[0033] (5) Optimized Transformer-SAC model update action strategy;

[0034] Repeat steps (3) to (5) to iterate the Transformer-SAC model learning until convergence, thus completing the Transformer-SAC model training.

[0035] Furthermore, the optimization of the Transformer-SAC model includes parameter optimization of the Transformer network, using a loss function that minimizes the mean squared error, backpropagation via the chain rule, and an adaptive moment estimation optimizer to update the parameters of the Transformer network.

[0036] Furthermore, the trained Transformer-SAC model is embedded into the voltage control system of the low-voltage distribution network, enabling the Transformer-SAC model to interact with the low-voltage distribution network environment to form action strategies, including the following steps:

[0037] 1.1. Based on the real-time power grid status, the trained Transformer-SAC model generates action strategies and issues action commands to the execution equipment of the low-voltage distribution network;

[0038] 1.2 The execution device executes the selected control action to change the state of the actual low-voltage distribution network environment, and feeds the changed network state back to the trained Transformer-SAC model;

[0039] 1.3 The Transformer network performs feature processing on the input power grid state and optimizes the weights and biases of the Transformer network. The Transformer network extracts the latest correlation feature matrix based on the real-time updated low-voltage distribution network state and inputs it into the SAC algorithm to generate control strategies, thus completing the real-time iterative training of the Transformer-SAC model and the voltage regulation control of the low-voltage distribution network.

[0040] The system for implementing the low-voltage distribution network voltage control method based on deep reinforcement learning includes a low-voltage distribution network voltage control system and a Transformer-SAC control module.

[0041] The Transformer-SAC control module includes a Transformer-SAC model for extracting features from the power grid state and generating a control strategy. The Transformer network is used for feature extraction, optimizing the input features of the control strategy. The SAC algorithm is used to generate the optimized control strategy, which is then processed through the policy network and... Training is conducted via the network;

[0042] The voltage control system of a low-voltage distribution network includes: a distribution network condition monitoring system, a central controller, and actuators;

[0043] The power distribution network condition monitoring system monitors the state of the power grid in real time through sensors. The sensors feed back the real-time data to the Transformer-SAC model to evaluate the effectiveness of the control strategy.

[0044] The central controller sends the control commands generated by the Transformer-SAC model to the execution device through the communication network. After receiving the control commands, the execution device adjusts its operating state according to the commands. The environmental state during the execution process is fed back to the Transformer-SAC model in real time through sensors. The Transformer-SAC model realizes voltage control and optimizes the Transformer network parameters within the Transformer-SAC model based on the feedback.

[0045] Furthermore, the execution device includes a reactive power compensator for regulating the reactive power output of the power grid and a distributed power supply device for adjusting the active power output according to control commands.

[0046] Furthermore, the communication network adopts the IEC 61850 standardized communication protocol to transmit control commands from the central controller to each substation, and then the substation forwards them to the equipment.

[0047] This invention combines Transformer networks and the SAC algorithm to address the problem of voltage exceeding forward and reverse limits when massive renewable energy sources are connected to low-voltage distribution networks, achieving optimized voltage control in low-voltage distribution networks. It exhibits significant advantages in global feature extraction, robustness and adaptability, real-time feedback mechanisms, and adaptability to topology changes when handling low-voltage distribution network voltage optimization problems.

[0048] Compared with existing technologies, the beneficial effects of the present invention are specifically as follows:

[0049] (1) By introducing the Transformer network for global feature extraction and utilizing its self-attention mechanism, it is possible to identify the complex dependencies between different nodes in the low-voltage distribution network. Thus, when considering various factors such as voltage, power, and load of the low-voltage distribution network, global information between each node can be captured, thereby improving the accuracy and adaptability of the voltage control strategy.

[0050] (2) By introducing entropy regularization, this invention enables the agent to balance exploration and utilization during the optimization process, thereby improving the robustness of the system. In particular, it can maintain a relatively stable control effect when facing the uncertainty of power grid operating conditions.

[0051] (3) Through real-time interactive feedback with the low-voltage distribution network environment, the agent can dynamically adjust and optimize control decisions based on immediate rewards and new states. This adaptive learning process enables the agent to adjust control strategies in real time under constantly changing grid operating conditions, ensuring the best voltage regulation effect and further guaranteeing the stable operation of the grid. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a low-voltage distribution network voltage control method based on deep reinforcement learning, as an example.

[0053] Figure 2 The image shows the convergence curve of the reward function for the operation of a low-voltage distribution network voltage control system based on deep reinforcement learning, as an example.

[0054] Figure 3 The image shows the loss curve of the value function of a low-voltage distribution network voltage control system based on deep reinforcement learning, as an example.

[0055] Figure 4 The figure shows the node voltage exceedance curve of a low-voltage distribution network under the operation of a voltage control system based on deep reinforcement learning, as an example. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] like Figure 1 As shown in the figure, the low-voltage distribution network voltage control method based on deep reinforcement learning in this embodiment includes the following steps:

[0058] The trained Transformer-SAC model is embedded in the low-voltage distribution network; the Transformer-SAC model interacts with the environment of the low-voltage distribution network to form action strategies; the action strategies are sent to the intelligent controllers on the switch or user side to control the power of the distributed voltage and the number of reactive power compensators switched on and off; the distributed power generation equipment and reactive power compensators execute actions after receiving commands and generate new environmental data; the new environmental data is fed back to the Transformer-SAC model in real time for self-optimization.

[0059] The action strategy formed by the interaction between the Transformer-SAC model and the low-voltage distribution network environment includes the following steps:

[0060] S1. Extract the correlation feature matrix from the state variables of the low-voltage distribution network using a Transformer network. After processing by the Transformer network, the feature matrix is ​​output. The specific steps include:

[0061] S1.1 Extraction State variables of low-voltage distribution network at time t Mid-node voltage Active power load reactive power load Node voltage sensitivity As the correlation feature matrix :

[0062] ;

[0063] in, , express The real space of dimension 1 Indicates the number of nodes. As the feature number, this embodiment has 4 features, namely .

[0064] The state variables of the low-voltage distribution network in this embodiment Including: node voltage Active power load reactive power load Network topology connection matrix Node voltage sensitivity and distributed energy generation capacity Capacitor switching amount .

[0065] Wherein, the node voltage is The unit is per unit. For the first Each node at time... voltage value, This indicates the number of nodes. The voltage of each node, after being converted to a per-unit value, ranges from [0.95 to 1.05].

[0066] Active power load is The unit is kilowatt. Indicates the first Each node at time... Active power load; typical base load range is [20 kW, 500 kW].

[0067] Reactive power load is The unit is kilovolt-amperes. Indicates the first Each node at time... The reactive power load.

[0068] Distributed energy generation capacity is The unit is kilowatt. Indicates the first A distributed power source at time Distributed energy generation capacity, The number of distributed power sources; the power generation range of distributed energy sources is [0 kW, 250 kW].

[0069] Capacitor switching capacity is The unit is kilovolt-amperes. Indicates capacitor At any moment The amount of reactive power cut.

[0070] Network topology matrix Used to represent the connection relationships between nodes, matrix elements Represents a node With nodes Connected, 0 represents a node With nodes Not connected.

[0071] As one embodiment, the connections between nodes in this embodiment are based on the IEEE 33-node low-voltage distribution network standard topology, and the network topology matrix... Includes connection information for 33 nodes. Node voltage sensitivity. , Indicates at time No. The sensitivity of the node voltage to changes in active power is calculated by the following formula at time t. No. Sensitivity of node voltage to changes in active power :

[0072] ;

[0073] in, For the first Each node and The per-unit voltage difference between the two time points. For the first Each node and The difference in active power between the two time points.

[0074] S1.2, Correlation Feature Matrix The attention weight of each attention head is calculated using a multi-head attention mechanism:

[0075] ;

[0076] ;

[0077] in For the first Each attention head weight, , For the number of heads; These represent the query matrix, key matrix, and value matrix, respectively. express The real space of dimension 1 It is the dimension of each attention head. By Input features at time step Obtained by linear transformation, ; This indicates a normalization operation; , , It is the weight matrix of the linear transformation, which is used as parameters adaptively obtained during the training of the Transformer network; express The real space of dimension 1 It is the characteristic number.

[0078] S1.3, concatenate the weights of all attention points to obtain the concatenated feature matrix. And perform a linear mapping:

[0079] ;

[0080] in, The function concatenates the output matrices of multiple attention heads horizontally along the channel dimension to form a complete concatenated feature matrix. ， For the total number of attention heads, For the first One's attention, It is the output weight matrix of the attention head, concatenated with the feature matrix. The correlation feature matrix with the input The residuals are joined by summation and then processed by normalization.

[0081] ;

[0082] in This is a normalization operation; This represents the normalized characteristic matrix.

[0083] S1.4, normalize the characteristic matrix Through feedforward neural networks:

[0084] ;

[0085] ;

[0086] in, for The feature matrix after residual connection processing This is the matrix output after processing by the feedforward neural network. , These are the weight matrices for different layers of the feedforward network. For the intermediate layer dimensions of the feedforward network, ; It is the first layer of the feedforward network, with a size of The bias vector is added to the linear transformation result of the first layer; It is the second layer of the feedforward network, with a size of The bias vector is added to the linear transformation result of the second layer; The activation function represents the application of rectified linear units to the result of the linear transformation; the matrix output by the feedforward neural network after processing... as well as Normalization:

[0087] ;

[0088] in, This represents the output characteristics of the Transformer network.

[0089] S2. Input the feature matrix output by the Transformer network into the SAC algorithm to generate the action policy. The SAC algorithm includes:

[0090] 1) The policy optimization objective of the SAC algorithm is to maximize the expected reward and policy entropy:

[0091] ;

[0092] in, Here is the entropy regularization coefficient. For policy entropy, For the reward function, In order to be in Decision-making actions at any moment These are state variables for low-voltage distribution networks; It is a strategy in reinforcement learning, used to represent the rules for taking actions on a low-voltage distribution network; Indicates in state variables The probability distribution of the next action selection; This serves as a discount factor for expected returns, ensuring the decay of long-term rewards. Indicates policy-based Expected value operation; Representation Strategy The objective function; For an infinite number of steps.

[0093] 2) Experience Replay: This stores the interaction data between the agent and the environment, including environmental feedback on states, actions, and rewards. During training, the agent interacts with the low-voltage power distribution network environment, collecting information such as states, actions, and rewards from the environment. This information forms experience learning samples. Within a certain training period, the agent stores these experience learning samples in an experience replay pool and randomly samples from it (i.e., retrieves samples from the storage pool for learning) to increase sample diversity and avoid training bias during reinforcement learning.

[0094] 3) By minimizing Bellman error update of the function network:

[0095] ;

[0096] in, for The discount factor of the function express Time-state variables Next action of The value reflects the expected reward obtained after taking this action. Indicates the next strategy action Expectations for Error-weighted average of the network; Indicates in Below The network error is weighted and averaged. yes Output features of the Transformer network at any time Time state variables, yes Transformer network output at time The state variable at that time, yes Momentary rewards yes Actions performed by the intelligent agent at all times yes The actions taken by the intelligent agent at any given moment.

[0097] 4) Update the policy network by maximizing the policy entropy and sum. Value update strategy network:

[0098] ;

[0099] In the formula, Let the loss function be the policy network. Indicates in state variables Weighted average; In strategy Below, actions based on strategy selection Expected value For state variables Next, strategy Select Action The probability of.

[0100] reward function Calculated by the following formula:

[0101] ;

[0102] In the formula for Penalty value for voltage exceeding limit at any time. for Always control costs. for It is used to punish power losses in low-voltage distribution networks. for Time node voltage sensitivity penalty value, , , , These are the weights of the corresponding sub-items; In order to be in Decision-making actions at any moment for State variables of the low-voltage distribution network at any given time.

[0103] The voltage over-limit penalty value represents the penalty for a node voltage exceeding the allowable range, and the calculation formula is as follows:

[0104] ;

[0105] In the formula The penalty value. For reference voltage, For the maximum allowable deviation, for Time of the first The per-unit voltage value of each node. This represents the number of nodes.

[0106] Control cost represents the execution cost of penalizing control actions, and the calculation formula is:

[0107] ;

[0108] In the formula To control the action The cost coefficient, Total execution cost.

[0109] The formula for calculating power loss in a low-voltage distribution network is as follows:

[0110] ;

[0111] In the formula For the first Power loss of each node, This represents the total power loss value.

[0112] The node voltage sensitivity penalty is used to guide the agent to prioritize adjusting nodes that affect the overall voltage. The calculation formula is as follows:

[0113] ;

[0114] In the formula For the first The change in reactive power at each node For the first Each node at time... Sensitivity, This is the voltage sensitivity penalty value.

[0115] This embodiment trains a Transformer-SAC model based on time-series load history data, topology parameters, distributed generation parameters, and capacitor parameters of a low-voltage distribution network, including the following steps:

[0116] (1) Based on historical data of low-voltage distribution network, a virtual environment of low-voltage distribution network is constructed. The Transformer network obtains state variables through the virtual environment, extracts the correlation feature matrix, and generates new state variables.

[0117] As one embodiment, this embodiment uses PyTorch / TensorFlow to construct a virtual environment for a low-voltage distribution network. The simulation uses an IEEE 33-node low-voltage distribution network and simulates different load fluctuations and distributed energy access scenarios. The historical data for the low-voltage distribution network is relevant historical data from the actual engineering environment, including time-series load history data, network topology matrix parameters, distributed power source parameters, and capacitor parameters. The simulation environment is mathematically modeled from the topology matrix and historical active and reactive load information. Then, through power flow calculations, the operating state of the simulation environment under different instructions or control strategies given by the deep reinforcement learning DRL is obtained, including voltage and current.

[0118] The virtual environment is used to provide the state variables of the low-voltage distribution network required by the Transformer-SAC model and to receive the control strategy generated by the Transformer-SAC model. Next action The execution result.

[0119] (2) Action strategy obtained by SAC algorithm based on the initial virtual environment state The Transformer-SAC model outputs action commands. ,action This includes discrete control actions for adjusting reactive power compensators and continuous control actions for adjusting the power generation of distributed energy sources.

[0120] (3) The reactive power compensator and distributed power source perform actions, causing the virtual environment of the low-voltage distribution network to change. The changed state variables are fed back to the Transformer-SAC model for iterative optimization.

[0121] (4) The Transformer-SAC model is optimized using the changed state variables and iteratively trained through loss calculation, backpropagation, and parameter update.

[0122] (5) Optimized Transformer-SAC model update action strategy.

[0123] Repeat steps (3)-(5) to iterate the Transformer-SAC model learning until convergence, thus completing the training of the Transformer-SAC model.

[0124] Optimization of the Transformer-SAC model includes parameter optimization of the Transformer network, using a loss function that minimizes the mean squared error, backpropagation via the chain rule, and an adaptive moment estimation optimizer to update the Transformer network parameters.

[0125] The loss function is to minimize the mean squared error:

[0126]

[0127] in For loss function, It is the first Predicted values ​​of node features It is the first The true value of each node's feature. This refers to the number of nodes. Backpropagation calculates the gradient of the loss function with respect to the Transformer network parameters (including the weights and biases of the query, key, and value matrices) using the chain rule:

[0128] ;

[0129] in It is the output of each layer of the Transformer network. The gradient is propagated backward through each layer of the Transformer network to calculate the gradient of each parameter. For including , , , , The parameters.

[0130] The parameters of the Transformer network are updated using an adaptive moment estimator based on the calculated gradients. The optimization rule is as follows:

[0131] ;

[0132] ;

[0133]

[0134]

[0135] in, and yes Momentum and squared gradient at time intervals. and yes Time bias correction It's the learning rate. It is a constant used to prevent division by zero errors. 、 The momentum decay coefficients are given for the two adaptive moment estimators.

[0136] The trained Transformer-SAC model is embedded into the voltage control system of a low-voltage distribution network to form a Transformer-SAC model that interacts with the environment, as detailed below:

[0137] 1.1 The real-time grid state variables are input into the trained Transformer-SAC model to generate action strategies and issue action commands to the distributed power sources and reactive power compensators in the low-voltage distribution network.

[0138] 1.2 The reactive power compensator and distributed power source execute the selected control actions to change the state of the actual low-voltage distribution network environment, and feed the changed network state back to the trained Transformer-SAC model.

[0139] 1.3 The Transformer network performs feature processing on the input power grid state and optimizes the weights and biases of the Transformer network based on step (4). The Transformer network extracts the latest correlation feature matrix based on the real-time updated low-voltage distribution network state and inputs it into the SAC algorithm to generate control strategies, thus completing the real-time iterative training of the Transformer-SAC model and the voltage regulation control of the low-voltage distribution network.

[0140] This embodiment also provides a system for implementing the deep reinforcement learning-based low-voltage distribution network voltage control method, including a low-voltage distribution network voltage control system and a Transformer-SAC control module.

[0141] The Transformer-SAC control module includes a Transformer-SAC model composed of a Transformer network and a SAC algorithm, used to extract features from the power grid state and generate a control strategy. The Transformer network is used for feature extraction, optimizing the input features of the control strategy. The SAC algorithm is used to generate the optimized control strategy, which is then processed through the policy network and... Training is conducted via the internet.

[0142] The voltage control system for low-voltage distribution networks includes:

[0143] The power distribution network condition monitoring system monitors the state of the power grid (including voltage, power, etc.) in real time through sensors (such as voltage and current sensors). The sensors feed back the real-time data to the Transformer-SAC model to evaluate the effectiveness of the control strategy and guide the next action.

[0144] The central controller, installed on the low-voltage distribution side, obtains the power grid control strategy based on the Transformer-SAC model and issues control commands through a communication protocol.

[0145] The communication network adopts the IEC 61850 standardized communication protocol to transmit control commands from the central controller to each substation, and then the substation forwards them to the execution devices.

[0146] The execution equipment includes reactive power compensators for regulating the reactive power output of the power grid and distributed power generation equipment that adjusts its active power output according to control commands.

[0147] The workflow of the system implementing the deep reinforcement learning-based low-voltage distribution network voltage control method is as follows:

[0148] The central controller on the distribution transformer side sends the control commands generated by the Transformer-SAC model (including the adjustment of the capacitor switching of the reactive power compensator in the low-voltage distribution network and the active power output of the distributed power generation equipment) to each substation of the low-voltage distribution network through the standardized communication protocol IEC 61850. The substation, as an intermediate layer, is responsible for forwarding the received control commands to the corresponding reactive power compensators and distributed power generation equipment.

[0149] After receiving control commands, the reactive power compensator and distributed power supply equipment adjust their operating status according to the commands, changing the power output of the distributed power supply equipment and the capacitor switching of the reactive power compensator; the environmental status during the equipment execution process is fed back to the Transformer-SAC model in real time through sensors.

[0150] The Transformer-SAC model achieves voltage control and self-optimization of Transformer parameters within the model based on environmental feedback.

[0151] The control effect of this invention is shown in the accompanying drawings, wherein the reward function curve of the Transformer-SAC model is shown in the figure below. Figure 2 As shown, the value function loss curve is as follows: Figure 3 As shown, the Transformer-SAC model began to converge after 82 iterations. The final node voltage control effect is as follows: Figure 4 As shown, the node out-of-bounds situation disappeared after 63 iterations.

[0152] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any modifications, alterations, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A voltage control method for low-voltage distribution networks based on deep reinforcement learning, characterized in that, Includes the following steps: The trained Transformer-SAC model is embedded into the voltage control system of the low-voltage distribution network, so that the Transformer-SAC model interacts with the environment of the low-voltage distribution network to form action strategies. The action strategy is sent to the execution equipment to control the power of the distributed voltage and the number of reactive power compensators switched on and off; After receiving a command, the device executes an action, and the low-voltage distribution network forms new state variables. The new state variables are fed back to the Transformer-SAC model in real time for optimization. The Transformer-SAC model interacts with the low-voltage distribution network environment to form action strategies, including the following steps: S1. The Transformer network is used to extract the correlation feature matrix from the state variables of the low-voltage distribution network. The correlation feature matrix is ​​then processed by the Transformer network to output the feature matrix. This specifically includes the following steps: S1.1 Extracting the state variables S of the low-voltage distribution network at time t. t Mid-node voltage V t Active power load P t Reactive power load Q t Node voltage sensitivity Sen t As the correlation feature matrix X t : in, Let N represent an N×d dimensional real space, where N represents the number of nodes and d is the number of features, i.e., d = 4; State variables S of low-voltage distribution networks t Includes: node voltage V t Active power load P t Reactive power load Q t Network topology connection matrix T, node voltage sensitivity Sen t and distributed energy generation power P DER,t Capacitor switching quantity Q Cap,t ; Wherein, the node voltage is V t =[V t,1 ,…,V t,N ], the unit is per unit, V t,N Let N be the voltage value of the Nth node at time t, where N represents the number of nodes. The voltage of each node, after being converted to a per-unit value, ranges from [0.95 to 1.05]. The active power load is P t =[P t,1 ,…,P t,N ], the unit is kilowatt, P t,N This represents the active power load of the Nth node at time t; the typical base load range is [20kW, 500kW]. The reactive power load is Q t =[Q t,1 ,…,Q t,N The unit is kilovolt-ampere (Q). t,N This represents the reactive power load of the Nth node at time t; Distributed energy generation capacity is P DER,t =[P DER,t,1 ,…,P DER,t,M ], the unit is kilowatt, P DER,t,M This represents the distributed energy generation power of the Mth distributed power source at time t, where M is the number of distributed power sources; the distributed energy generation power range is [0kW, 250kW]. The switching capacitance of the capacitor is Q. Cap,t =[Q Cap, t,1,…,Q Cap,t,l The unit is kilovolt-ampere (Q). Cap,t,l This represents the reactive power input / output of capacitor l at time t; The network topology matrix T is used to represent the connection relationships between nodes, and the matrix elements T i,j =1 indicates that node i is connected to node j, T i,j =0 indicates that node i and node j are not connected; The connections between nodes are based on the IEEE 33-node low-voltage distribution network standard topology, and the network topology matrix T contains connection information for 33 nodes; the node voltage sensitivity Sen t =[Sen t,1 ,…,Sen t,N Sen t,N Let Sen represent the sensitivity of the voltage at node N to changes in active power at time t. The sensitivity Sen of the voltage at node i to changes in active power at time t is calculated using the following formula: t,i : in, Let be the per-unit voltage difference of the i-th node before and after time t and t+1. Let be the difference in active power of the i-th node before and after time t and t+1; S1.2, Correlation Feature Matrix X t The attention weight of each attention head is calculated using a multi-head attention mechanism: A=XW A ,B=XW B ,C=XW C ; Where H f Let A, B, and F be the weights of the f-th attention heads, where f ∈ F and F is the number of attention heads. These represent the query matrix, key matrix, and value matrix, respectively. Represents N×d k The real space of dimension d k These are the dimensions of each attention head, A, B, and C, obtained by linear transformation of the input feature X at time t, where X ∈ x. t ; softmax(.) represents the normalization operation; W A W B W C It is the weight matrix of the linear transformation, which is used as parameters adaptively obtained during the training of the Transformer network; Represents d×d k A real space of dimension d, where d is the characteristic number; S1.3, concatenate the weights of all attention heads to obtain the concatenated feature matrix H, and perform linear mapping: H=Concat(H 1 ,H 2 ,…,H F )W F ; The Concat(.) function concatenates the output matrices of multiple attention heads horizontally along the channel dimension to form a complete concatenated feature matrix H, where F is the total number of attention heads and H is the sum of the input matrices and output matrices of the input matrices. F For the Fth attention head, W F It is the output weight matrix of the attention head, concatenated with the feature matrix H and the input correlation feature matrix X. t The residuals are joined by summation and then processed by normalization. H′=LayerNorm(H+X t ); Where LayerNorm(·) is the normalization operation; H′ represents the normalized feature matrix; S1.

4. Pass the normalized feature matrix H′ through a feedforward neural network: H″=ReLU(H′W1+b1); X t ′=H″W2+b2; Where H″ is the feature matrix of H′ after residual connection processing, and X′ t This is the matrix output after processing by the feedforward neural network. These are the weight matrices for different layers of the feedforward network, d ff d represents the intermediate layer dimension of the feedforward network. ff >d; It is the first layer of the feedforward network with a size of d. ff The bias vector is added to the linear transformation result of the first layer; This is the bias vector of size d in the second layer of the feedforward network, which is added to the linear transformation result of the second layer; ReLU(·) is the activation function, indicating that a rectified linear unit is applied to the linear transformation result; the output matrix X′ after processing by the feedforward neural network is... t And H″ normalization: X″ t =LayerNorm(X′ t +H″); Where, X″ t This represents the output characteristics of the Transformer network; S2. Input the feature matrix output by the Transformer network into the SAC algorithm to generate an action policy; the SAC algorithm includes: 1) The policy optimization objective of the SAC algorithm is to maximize the expected reward and policy entropy: Where α is the entropy regularization coefficient, G(π(·∣s) t R(s) represents the policy entropy. t ,a t ) is the reward function, a t For the decision action at time t, s t π represents the state variables of the low-voltage distribution network; π is the policy in reinforcement learning, used to represent the rules for taking actions on the low-voltage distribution network; π(·∣s t ) indicates that in the state variable s t The probability distribution of the next action selection; τ∈(0,1) is the discount factor for expected return, ensuring the decay of long-term rewards; J(π) represents the expected value operation based on policy π; J(π) represents the objective function of policy π; ∞ represents infinite steps; 2) Experience Replay: Stores the interaction data between the agent and the environment, including the state, actions, and rewards fed back by the environment. During the training process, the agent interacts with the environment of the low-voltage distribution network and collects information such as the state, actions, and rewards fed back by the environment. This information forms experience learning samples. Within a certain training period, the agent stores these experience learning samples in the experience replay pool and randomly samples from them to help increase the diversity of samples and avoid training bias in the reinforcement learning process. 3) Update the Q-network by minimizing the Bellman error of the Q-function: Where γ is the discount factor of the Q function, Q(s) t ′,a t ) represents the state variable s at time t. t 'Execute action a' t The Q value reflects the expected reward obtained after taking this action. Indicates the next strategy action a t+1 Expectations, L Q Weighted average of the errors of the Q network; Indicates in s t ,a t ,R t ,s t+1′ The following is a weighted average of the Q-network error, s t ′ is the output feature X″ of the Transformer network at time t. t The state variable s t+1′ The Transformer network output X″ at time t+1 t+1 The state variable at time, R t It is the reward at time t, a t It is the action performed by the agent at time t, a t+1 It refers to the action taken by the agent at time t+1; 4) Update the policy network by maximizing the policy entropy and Q-value: In the formula, L π Let the loss function be the policy network. This indicates that in the state variable s t Weighted average under ′; Let a be the action a based on policy selection under policy π. t The expected value, π(a) t |s t ′) represents the state variable s t Under ', strategy π selects action a t The probability of; Reward function R(s) t ,a t ) Calculated by the following formula: R(s t ,a t )=ω1R volt,t +ω2R cost,t +ω3R loss,t +ω4R sens,t ; In the formula R volt,t R is the voltage over-limit penalty value at time t. cost,t To control costs at time t, R loss,t R is used to penalize power losses in the low-voltage distribution network at time t. sens,t Let be the node voltage sensitivity penalty value at time t, and ω1, ω2, ω3, and ω4 be the weights of the corresponding sub-items; a t For the decision action at time t, s t Let t be the state variables of the low-voltage distribution network at time t; The voltage over-limit penalty value represents the penalty for a node voltage exceeding the allowable range, and the calculation formula is as follows: In the formula R volt V is the penalty value. ref For reference voltage, ΔV max For the maximum allowable deviation, V t,i Let N be the per-unit voltage value of the i-th node at time t, and N be the number of nodes. Control cost represents the execution cost of penalizing control actions, and the calculation formula is: In the formula c i To control action a t,i Cost coefficient, R cost Total execution cost; The formula for calculating power loss in a low-voltage distribution network is as follows: In the formula P loss,i R is the power loss of the i-th node. loss This represents the total power loss value. The node voltage sensitivity penalty is used to guide the agent to prioritize adjusting nodes that affect the overall voltage. The calculation formula is as follows: In the formula ΔQ t,i Sen represents the change in reactive power at the i-th node. t,i Let R be the sensitivity of the i-th node at time t. sens This is the voltage sensitivity penalty value.

2. The low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 1, characterized in that, Step S1 specifically includes the following steps: S1.1 Extract the node voltage, active power load, reactive power load, and node voltage sensitivity from the state variables of the low-voltage distribution network as the correlation feature matrix; S1.2 The correlation feature matrix calculates the attention weight of each attention head through a multi-head attention mechanism; S1.

3. Concatenate the weights of all attention heads to obtain a concatenated feature matrix and perform linear mapping; add the concatenated feature matrix to the associated feature matrix to achieve residual connection, and then process it through normalization to obtain the normalized feature matrix. S1.4 Input the normalized feature matrix into the feedforward neural network to obtain the feature matrix processed by the feedforward neural network.

3. The low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 1, characterized in that, Training the Transformer-SAC model includes the following steps: (1) Based on historical data of low-voltage distribution network, a virtual environment of low-voltage distribution network is constructed in the computer. The Transformer network obtains state variables through the virtual environment, extracts the correlation feature matrix, and generates new state variables. (2) Input the new state variables into the action strategy obtained by the SAC algorithm and output the action; (3) The execution equipment performs actions to change the state of the virtual environment of the low-voltage distribution network, and the changed state variables are fed back to the Transformer-SAC model. (4) The Transformer-SAC model uses the changed state variables for optimization; (5) Optimized Transformer-SAC model update action strategy; Repeat steps (3) to (5) to iterate the Transformer-SAC model learning until convergence, thus completing the Transformer-SAC model training.

4. The low-voltage distribution network voltage control method based on deep reinforcement learning according to any one of claims 1 to 3, characterized in that, Optimization of the Transformer-SAC model includes parameter optimization of the Transformer network, using a loss function that minimizes the mean squared error, backpropagation via the chain rule, and an adaptive moment estimation optimizer to update the Transformer network parameters.

5. The low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 4, characterized in that, The trained Transformer-SAC model is embedded into the voltage control system of a low-voltage distribution network, enabling the Transformer-SAC model to interact with the low-voltage distribution network environment to form action strategies. Includes the following steps: 1.

1. Based on the real-time power grid status, the trained Transformer-SAC model generates action strategies and issues action commands to the execution equipment of the low-voltage distribution network; 1.2 The execution device executes the selected control action to change the state of the actual low-voltage distribution network environment, and feeds the changed network state back to the trained Transformer-SAC model; 1.3 The Transformer network performs feature processing on the input power grid state and optimizes the weights and biases of the Transformer network. The Transformer network extracts the latest correlation feature matrix based on the real-time updated low-voltage distribution network state and inputs it into the SAC algorithm to generate control strategies, thus completing the real-time iterative training of the Transformer-SAC model and the voltage regulation control of the low-voltage distribution network.

6. A system for implementing the low-voltage distribution network voltage control method based on deep reinforcement learning as described in claim 1, characterized in that, This includes voltage control systems for low-voltage distribution networks and Transformer-SAC control modules; The Transformer-SAC control module includes a Transformer-SAC model, which is used to extract features from the power grid state and generate control strategies; wherein the Transformer network is used for feature extraction and to optimize the input features of the control strategy. The SAC algorithm is used to generate optimized control policies, and is trained through a policy network and a Q network. The voltage control system of a low-voltage distribution network includes: a distribution network condition monitoring system, a central controller, and actuators; The power distribution network status monitoring system monitors the status of the power grid in real time through sensors. The sensors feed back the real-time data to the Transformer-SAC model to evaluate the effectiveness of the control strategy. The central controller sends the control commands generated by the Transformer-SAC model to the execution device through the communication network. After receiving the control commands, the execution device adjusts its operating state according to the commands. The environmental state during the execution process is fed back to the Transformer-SAC model in real time through sensors. The Transformer-SAC model realizes voltage control and optimizes the Transformer network parameters within the Transformer-SAC model based on the feedback.

7. The system of the low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 6, characterized in that, The execution equipment includes a reactive power compensator for regulating the reactive power output of the power grid and a distributed power supply device for adjusting the active power output according to control commands.

8. The system of the low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 6, characterized in that, The communication network adopts the IEC 61850 standardized communication protocol to transmit control commands from the central controller to each substation, and then the substation forwards them to the execution devices.

Images