Low-voltage distribution network voltage control method and system based on deep reinforcement learning

Abstract

The invention discloses a low-voltage power distribution network voltage control method and system based on deep reinforcement learning, and the method comprises the following steps: embedding a trained Transform-SAC model into a low-voltage power distribution network, enabling the Transform-SAC model to interact with the environment of the low-voltage power distribution network to form an action strategy, transmitting the action strategy to a switch or an intelligent controller at a user side, and enabling the intelligent controller to control the low-voltage power distribution network according to the action strategy. And the control module is used for controlling the power of the distributed voltage and the switching of the reactive power compensator, the distributed power supply equipment and the reactive power compensator execute actions after receiving the command, and new environment data is formed and fed back to the model in real time for self-optimization. According to the invention, the SAC algorithm is adopted to optimize the voltage control strategy of the low-voltage distribution network, the learning efficiency is improved through experience playback and strategy optimization, and the continuous improvement of the voltage regulation effect is realized through the interactive feedback of the model and the low-voltage distribution network environment, the intelligent agent continuously optimizes the control strategy and regulates the power grid equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of power system automation control, and in particular to a low-voltage distribution network voltage control method and system based on deep reinforcement learning. Background Art

[0002] With the large-scale access of distributed renewable energy, voltage control of low-voltage distribution networks faces new challenges. Low-voltage distribution networks usually carry power demands from households, businesses and small industrial users, and these loads have high dynamic variability and uncertainty. At the same time, more and more distributed generation units such as photovoltaics and electric vehicles are connected, making the operating environment of low-voltage distribution networks more complex and unpredictable.

[0003] Existing voltage control technologies usually rely on preset rules and empirical models, and often show low robustness and accuracy when faced with high-dimensional state spaces and complex network topologies. Especially in low-voltage distribution networks, due to the highly dynamic network topology and the randomness of power flow, traditional voltage control methods can no longer meet the current power grid operation requirements.

[0004] Reinforcement learning can continuously optimize control strategies and gradually achieve adaptive adjustments through continuous interaction with the environment. Therefore, it has certain advantages in dealing with complex and uncertain low-voltage distribution network environments. However, reinforcement learning algorithms also face some challenges in practical applications. First, the state space of the distribution network usually has a high dimension. Traditional deep reinforcement learning methods may have difficulty in effectively processing these high-dimensional data, resulting in low training efficiency and slow convergence. Second, the topological structure of the low-voltage distribution network is often complex and dynamically changing, which makes the control strategy based on the classical model lack sufficient flexibility and generalization ability.

[0005] Existing multi-time-scale distribution network voltage optimization control methods (such as Chinese patent document CN202410803630.6) and multi-time-scale voltage regulation methods based on deep reinforcement learning (such as Chinese patent document CN202411696765.3) construct voltage control models through conventional deep learning methods. When facing distribution network load fluctuations or sudden events, such as distribution network topology changes, equipment failures and other uncertain conditions, they cannot fully capture the complex nonlinear characteristics of the system, and the voltage regulation effect needs to be further improved. Summary of the invention

[0006] The purpose of the present invention is to provide a low-voltage distribution network voltage control method and system based on deep reinforcement learning. The method optimizes the problem of voltage difficulty in regulating caused by large-scale access of new energy to the low-voltage distribution network by introducing the Transformer network and the SAC algorithm. Specifically, the global features are extracted by the Transformer network, and the control strategy is optimized in combination with the SAC algorithm to achieve optimal control of the voltage of the low-voltage distribution network. As a powerful sequence modeling tool, the Transformer network can extract effective global features from high-dimensional data, solving the shortcomings of traditional deep learning methods in dealing with complex low-voltage distribution network states. The SAC algorithm enhances the exploration ability and stability of the system by optimizing the strategy entropy and expected return, thereby improving the accuracy and robustness of voltage control, and can provide a more efficient solution for voltage control of low-voltage distribution networks.

[0007] The present invention is achieved by at least one of the following technical solutions.

[0008] The voltage control method of a low-voltage distribution network based on deep reinforcement learning includes the following steps: The trained Transformer-SAC model is embedded into the voltage control system of the low-voltage distribution network, so that the Transformer-SAC model interacts with the environment of the low-voltage distribution network to form an action strategy; The action strategy is sent to the execution device to control the power of the distributed voltage and the number of switching on and off of the reactive power compensator; After receiving the command, the execution device executes the action, and the low-voltage distribution network forms new state variables, which are fed back to the Transformer-SAC model in real time for optimization; The Transformer-SAC model interacts with the environment of the low-voltage distribution network to form an action strategy, including the following steps: S1. Use the Transformer network to extract the associated feature matrix from the state variables of the low-voltage distribution network. After the associated feature matrix is ​​processed by the Transformer network, the feature matrix is ​​output; S2. Input the feature matrix output by the Transformer network into the SAC algorithm to generate an action strategy.

[0009] Furthermore, in step S1, a Transformer network is used to extract a correlation feature matrix from the state variables of the low-voltage distribution network, and the correlation feature matrix is ​​processed by the Transformer network to output a feature matrix, which includes the following steps: S1.1, extracting node voltage, active power load, reactive power load, and node voltage sensitivity from the state variables of the low-voltage distribution network as a correlation feature matrix; S1.2, the associated feature matrix calculates the attention weight of each attention head through the multi-head attention mechanism; S1.3, concatenate the weights of all attention heads to obtain a concatenated feature matrix, and perform linear mapping; add the concatenated feature matrix to the associated feature moment to realize residual connection, and then process it through normalization to obtain a normalized feature matrix; S1.4. Input the normalized feature matrix into the feedforward neural network to obtain the feature matrix processed by the feedforward neural network.

[0010] Further, in step S2, the SAC algorithm includes: 1) The strategy optimization goal of the SAC algorithm is to maximize the expected return and strategy entropy: 2) Experience replay: store the interaction data between the agent and the environment, including the state, action, and reward of the environment feedback; 3) Update Network: By minimizing Bellman error update of a function network; 4) Update the policy network by maximizing policy entropy and Value update policy network.

[0011] Furthermore, the reward function Calculated by the following formula: ; In the formula for The voltage over-limit penalty value at the moment, for Always control costs. for It is always used to punish the power loss of low voltage distribution network. for The node voltage sensitivity penalty value at the moment, , , , are the weights of the corresponding sub-items respectively; For Moment-by-moment decision making, for State variables of the low-voltage distribution network at each moment.

[0012] Furthermore, the training of the Transformer-SAC model includes the following steps: (1) Based on the historical data of the low-voltage distribution network, a virtual environment of the low-voltage distribution network is constructed in the computer. The Transformer network obtains state variables through the virtual environment to extract the associated feature matrix and generate new state variables; (2) The new state variable is input into the action strategy obtained by the SAC algorithm and the action is output; (3) The execution device performs actions to change the state of the virtual environment of the low-voltage distribution network, and the changed state variables are fed back to the Transformer-SAC model; (4) The Transformer-SAC model uses the changed state variables for optimization; (5) Update action strategy of the optimized Transformer-SAC model; Repeat steps (3) to (5) to iterate the Transformer-SAC model learning until convergence, completing the Transformer-SAC model training.

[0013] Furthermore, the Transformer-SAC model is optimized including the parameter optimization of the Transformer network, using the loss function of minimizing the mean square error, back propagation through the chain rule, and adaptive moment estimation optimizer to optimize and update the parameters of the Transformer network.

[0014] Furthermore, the trained Transformer-SAC model is embedded in the voltage control system of the low-voltage distribution network, so that the Transformer-SAC model interacts with the environment of the low-voltage distribution network to form an action strategy, including the following steps: 1.1. Generate action strategies based on the trained Transformer-SAC model based on the real-time power grid status and issue action instructions to the execution devices of the low-voltage distribution network; 1.2. The execution device executes the selected control action to change the state of the actual low-voltage distribution network environment, and feeds the changed grid state back to the trained Transformer-SAC model; 1.3. The Transformer network performs feature processing on the input power grid state and optimizes the weights and bias items of the Transformer network. The Transformer network extracts the latest associated feature matrix based on the real-time updated low-voltage distribution network state and inputs it into the SAC algorithm to generate a control strategy, completing the real-time iterative training of the Transformer-SAC model and the voltage regulation control of the low-voltage distribution network.

[0015] A system for implementing the low-voltage distribution network voltage control method based on deep reinforcement learning, comprising a voltage control system of the low-voltage distribution network and a Transformer-SAC control module; The Transformer-SAC control module includes a Transformer-SAC model, which is used to extract features from the power grid state and generate a control strategy; wherein the Transformer network is used to extract features and optimize the input features of the control strategy; the SAC algorithm is used to generate the optimized control strategy, through the strategy network and Network for training; The voltage control system of the low voltage distribution network includes: distribution network status monitoring system, central controller, and execution equipment; The distribution network status monitoring system monitors the status of the power grid in real time through sensors. The sensors feed real-time data back to the Transformer-SAC model to evaluate the effectiveness of the control strategy. The central controller sends the control instructions formed by the Transformer-SAC model to the execution device through the communication network; after receiving the control instructions, the execution device adjusts the operating status according to the instructions, and the environmental status of the equipment during execution is fed back to the Transformer-SAC model in real time through sensors; the Transformer-SAC model realizes voltage control and optimization of the Transformer network parameters within the Transformer-SAC model based on the feedback.

[0016] Furthermore, the execution device includes a reactive power compensator for regulating reactive power output of the power grid and a distributed power supply device for adjusting active power output according to control instructions.

[0017] Furthermore, the communication network adopts the IEC 61850 standardized communication protocol to transmit control instructions from the central controller to each substation, and the substation forwards the control instructions to the device.

[0018] The present invention combines the Transformer network and the SAC algorithm to address the problem of voltage over-limit in both positive and reverse directions when massive new energy sources are connected to the low-voltage distribution network, and realizes the optimal control of the voltage of the low-voltage distribution network. When dealing with the voltage optimization problem of the low-voltage distribution network, it has obvious advantages in terms of global feature extraction, robustness and adaptability, real-time feedback mechanism, and adaptability to topology changes.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) By introducing the Transformer network for global feature extraction and utilizing its self-attention mechanism, the complex dependencies between different nodes in the low-voltage distribution network can be identified. In this way, the global information between each node can be captured when considering multiple factors such as voltage, power, and load of the low-voltage distribution network, thereby improving the accuracy and adaptability of the voltage control strategy.

[0020] (2) By introducing entropy regularization, the present invention enables the intelligent agent to balance exploration and utilization during the optimization process, thereby improving the robustness of the system. In particular, when facing the uncertainty of power grid operating conditions, it can maintain a relatively stable control effect.

[0021] (3) Through real-time interactive feedback with the low-voltage distribution network environment, the intelligent agent can dynamically adjust and optimize control decisions based on immediate rewards and new states. This adaptive learning process enables the intelligent agent to adjust control strategies in real time when the grid operating conditions are constantly changing, ensuring the best voltage regulation effect and further ensuring the stable operation of the grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The present invention is a flowchart of a low-voltage distribution network voltage control method based on deep reinforcement learning according to an embodiment of the present invention.

[0023] Figure 2 This is a graph showing the convergence of a reward function for the operation of a low-voltage distribution network voltage control system based on deep reinforcement learning in an embodiment.

[0024] Figure 3 The figure is a loss curve diagram of the value function of the operation of the low-voltage distribution network voltage control system based on deep reinforcement learning in the embodiment.

[0025] Figure 4 This is a graph showing the voltage exceeding the limit at the nodes of a low-voltage distribution network when a low-voltage distribution network voltage control system based on deep reinforcement learning is running. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, the low-voltage distribution network voltage control method based on deep reinforcement learning in this embodiment includes the following steps: The trained Transformer-SAC model is embedded in the low-voltage distribution network; the Transformer-SAC model interacts with the environment of the low-voltage distribution network to form an action strategy; the action strategy is sent to the intelligent controller on the switch or user side to control the power of the distributed voltage and the number of reactive compensators switched on and off; the distributed power supply equipment and reactive compensators execute actions after receiving the commands and form new environmental data; the new environmental data is fed back to the Transformer-SAC model in real time for self-optimization.

[0028] The action strategy formed by the interaction between the Transformer-SAC model and the low-voltage distribution network environment includes the following steps: S1. Use the Transformer network to extract the associated feature matrix from the state variables of the low-voltage distribution network, and output the feature matrix after processing by the Transformer network, which specifically includes the following steps: S1.1 Extraction The state variables of the low voltage distribution network at time Mid-node voltage , Active power load , reactive power load , node voltage sensitivity As the correlation feature matrix : ; in, , express dimensional real space, Indicates the number of nodes, is the number of features, and this embodiment has 4 features, namely .

[0029] The state variables of the low voltage distribution network in this embodiment Includes: Node voltage , Active power load , reactive power load , Network topology connection matrix , node voltage sensitivity And distributed energy generation power , capacitor switching capacity .

[0030] The node voltage is , the unit is per unit value, For the Nodes at time The voltage value, Represents the number of nodes. The voltage of each node is converted to a per-unit value and has a value range of [0.95 ~1.05].

[0031] The active power load is , in kilowatts, Indicates Nodes at time Active power load of ; typical base load range is [20 kW, 500 kW].

[0032] The reactive power load is , in kilovolt-amperes, Indicates Nodes at time reactive power load.

[0033] The distributed energy generation power is , in kilowatts, Indicates A distributed power source at time The distributed energy generation power, is the number of distributed generation sources; the power generation range of distributed energy is [0 kW, 250 kW].

[0034] The capacitor switching capacity is , in kilovolt-amperes, Indicates capacitor At the moment The reactive power switching amount.

[0035] Network topology matrix Used to represent the connection relationship between nodes, matrix elements Representation Node With Node Connected, 0 indicates a node With Node Not connected.

[0036] As one embodiment, the connection between nodes in this embodiment is based on the IEEE 33-node low-voltage distribution network standard topology, and the network topology matrix Contains connection information of 33 nodes. Node voltage sensitivity , Indicates at time No. The sensitivity of the node voltage to the change of active power is calculated by the following formula at time No. Sensitivity of node voltage to active power changes : ; in, For the Nodes in and The voltage per unit value difference before and after, For the Nodes in and The active power difference between the previous and next moments.

[0037] S1.2. Correlation feature matrix The attention weight of each attention head is calculated through the multi-head attention mechanism: ; ; in For the The attention head weights, , is the number of attention heads; denote the query matrix, key matrix and value matrix respectively, express dimensional real space, is the dimension of each attention head, By Input features at time Linear transformation yields, ; Represents a normalization operation; , , is the weight matrix of the linear transformation, which is used as the parameter adaptively acquired during the training process of the Transformer network; express dimensional real space, is the feature number.

[0038] S1.3. Concatenate the weights of all attention heads to obtain the concatenated feature matrix , and perform a linear mapping: ; in, The function concatenates the output matrices of multiple attention heads horizontally along the channel dimension to form a complete concatenated feature matrix ， is the total number of attention heads, For the Attention head, is the output weight matrix of the attention head, concatenated feature matrix The associated feature matrix with the input Add them together to realize residual connection, and then process them by normalization: ; in It is a normalization operation; Represents the normalized feature matrix.

[0039] S1.4, the normalized feature matrix Through a feed-forward neural network: ; ; in, for The feature matrix processed by residual connection, is the matrix output after feedforward neural network processing, , are the weight matrices of different layers of the feedforward network, is the intermediate layer dimension of the feedforward network, ; is the first layer of the feedforward network, with a size of The bias vector is added to the linear transformation result of the first layer; is the second layer of the feedforward network, with a size of The bias vector is added to the linear transformation result of the second layer; is the activation function, which means applying the rectified linear unit to the linear transformation result; the matrix output after the feedforward neural network processing as well as Normalization: ; in, Represents the output features of the Transformer network.

[0040] S2. Input the feature matrix output by the Transformer network into the SAC algorithm to generate an action strategy. The SAC algorithm includes: 1) The strategy optimization goal of the SAC algorithm is to maximize the expected return and strategy entropy: ; in, is the entropy regularization coefficient, is the policy entropy, is the reward function, For Moment-by-moment decision making, is the state variable of the low voltage distribution network; It is a strategy in reinforcement learning, used to represent the rules for taking actions on the low-voltage distribution network; Indicated in the state variable The probability distribution of the selected action; is the discount factor for the expected return, ensuring the decay of long-term rewards; Policy-based Expected value operation; Representation strategy The objective function of For infinite steps.

[0041] 2) Experience replay: Store the interaction data between the agent and the environment, including the state, action, and reward of the environment feedback; during the training process, the agent interacts with the environment of the low-voltage distribution network and collects information such as the state, action, and reward feedback from the environment, which form experience learning samples. Within a certain training cycle, the agent stores these experience learning samples in the experience replay pool and randomly samples from them (i.e., takes samples from the storage pool for learning) to help increase the diversity of samples and avoid training bias in the reinforcement learning process.

[0042] 3) By minimizing Bellman error update of a function network: ; in, for The discount factor of the function, express State variables at a moment Next action of Value, reflecting the expected return after taking this action, Indicates the next policy action expectations, for Weighted average of the network’s errors; Indicated in Next pair The network error is weighted averaged. yes Output features of the Transformer network at this moment When the state variable, yes Transformer network output at this moment The state variables at yes The rewards of the moment, yes The action performed by the agent at each moment, yes The action taken by the agent at each moment.

[0043] 4) Update the policy network by maximizing policy entropy and Value update strategy network: ; In the formula, is the loss function of the policy network, Indicated in the state variable The weighted average of For the strategy Next, the action selected based on the strategy The expected value of For the state variable Next, strategy Select Action probability.

[0044] Reward Function Calculated by the following formula: ; In the formula for The voltage over-limit penalty value at the moment, for Always control costs. for It is always used to punish the power loss of low voltage distribution network. for The node voltage sensitivity penalty value at the moment, , , , are the weights of the corresponding sub-items respectively; For Moment-by-moment decision making, for State variables of the low-voltage distribution network at each moment.

[0045] Among them, the voltage limit penalty value indicates that the penalty node voltage exceeds the allowable range, and the calculation formula is: ; In the formula is the penalty value, is the reference voltage, is the maximum allowable deviation, for Moment The voltage per unit value of each node is is the number of nodes.

[0046] The control cost represents the execution cost of the penalty control action, and the calculation formula is: ; In the formula To control the action The cost factor, is the total execution cost.

[0047] The power loss calculation formula of the penalty low voltage distribution network is: ; In the formula For the The power loss of each node is is the total power loss value.

[0048] The node voltage sensitivity penalty is used to guide the agent to prioritize the adjustment of nodes that affect the overall voltage. The calculation formula is: ; In the formula For the The reactive power change of each node, For the Nodes at time The sensitivity of is the voltage sensitivity penalty value.

[0049] This embodiment trains a Transformer-SAC model based on time-series load history data, topology parameters, distributed power supply parameters, and capacitor parameters of a low-voltage distribution network, including the following steps: (1) A virtual environment of the low-voltage distribution network is constructed based on the historical data of the low-voltage distribution network. The Transformer network obtains state variables through the virtual environment, extracts the associated feature matrix, and generates new state variables.

[0050] As one of the embodiments, this embodiment uses PyTorch / TensorFlow to build a virtual environment for a low-voltage distribution network, simulates the use of an IEEE 33-node low-voltage distribution network, and simulates different load fluctuations and distributed energy access conditions. The historical data of the low-voltage distribution network is the relevant historical data of the actual engineering environment, such as historical load data with time series, network topology matrix parameters, distributed power supply parameters, and capacitor parameters. The simulation environment is mathematically modeled by the topology matrix and historical active load and reactive load information, and then the operating status of the simulation environment under different instructions or control strategies given by deep reinforcement learning DRL is obtained through flow calculation, including voltage and current.

[0051] The virtual environment is used to provide the state variables of the low-voltage distribution network required by the Transformer-SAC model and receive the control strategy generated by the Transformer-SAC model Next action The execution result of .

[0052] (2) The action strategy obtained by the SAC algorithm based on the initial virtual environment state , Transformer-SAC model outputs action instructions ,action It includes discrete control actions for adjusting reactive power compensators and continuous control actions for adjusting the generated power of distributed energy sources.

[0053] (3) The reactive power compensator and distributed generation perform actions, causing the state of the virtual environment of the low-voltage distribution network to change. The changed state variables are fed back to the Transformer-SAC model for iterative optimization.

[0054] (4) The Transformer-SAC model uses the changed state variables for optimization and iterative training through loss calculation, back propagation, and parameter update.

[0055] (5) Update action strategy of the optimized Transformer-SAC model.

[0056] Repeat steps (3) to (5) to iterate the Transformer-SAC model learning until convergence, and the training of the Transformer-SAC model is completed.

[0057] The optimization of the Transformer-SAC model includes optimizing the parameters of the Transformer network, using the loss function that minimizes the mean square error, back propagation through the chain rule, and adaptive moment estimation optimizer to optimize and update the parameters of the Transformer network.

[0058] The loss function is to minimize the mean square error:

[0059] in is the loss function, It is The predicted value of node features, It is The true value of node features, is the number of nodes. Backpropagation calculates the gradient of the loss function with respect to the Transformer network parameters (including the weights and biases of the query, key, and value matrices) by the chain rule: ; in is the output of each layer of the Transformer network. The gradient is propagated backward through each layer of the Transformer network to calculate the gradient of each parameter. To include , , , , Parameters.

[0060] Use the adaptive moment estimation optimizer to optimize and update the parameters of the Transformer network according to the calculated gradient. The optimization rule is: ; ;

[0061]

[0062] in, and yes The momentum and squared gradient of the moment gradient, and yes Time bias correction, is the learning rate, is a constant to prevent division by zero errors, 、 Momentum decay coefficients for the two adaptive moment estimation optimizers.

[0063] The trained Transformer-SAC model is embedded into the voltage control system of the low-voltage distribution network to form a Transformer-SAC model that interacts with the environment, as follows: 1.1. The trained Transformer-SAC model is input with real-time grid state variables to generate action strategies and issue action instructions to distributed power sources and reactive power compensators in the low-voltage distribution network.

[0064] 1.2. The reactive power compensator and distributed generation execute the selected control actions to change the state of the actual low-voltage distribution network environment, and feed back the changed grid state to the trained Transformer-SAC model.

[0065] 1.3. The Transformer network performs feature processing on the input power grid state and optimizes the weights and bias terms of the Transformer network based on step (4). The Transformer network extracts the latest associated feature matrix based on the real-time updated low-voltage distribution network state and inputs it into the SAC algorithm to generate a control strategy, completing the real-time iterative training of the Transformer-SAC model and the voltage regulation control of the low-voltage distribution network.

[0066] This embodiment also provides a system for implementing the low-voltage distribution network voltage control method based on deep reinforcement learning, including a voltage control system of the low-voltage distribution network and a Transformer-SAC control module.

[0067] The Transformer-SAC control module includes a Transformer-SAC model composed of a Transformer network and a SAC algorithm, which is used to extract features from the power grid state and generate a control strategy. The Transformer network is used to extract features and optimize the input features of the control strategy. The SAC algorithm is used to generate the optimized control strategy. network for training.

[0068] The voltage control system of the low voltage distribution network includes: The distribution network status monitoring system monitors the status of the power grid (including voltage, power, etc.) in real time through sensors (such as voltage and current sensors). The sensors feed back real-time data to the Transformer-SAC model to evaluate the effectiveness of the control strategy and to guide the next action.

[0069] The central controller is a control system installed on the low-voltage distribution side. It obtains the control strategy of the power grid based on the Transformer-SAC model and issues control instructions through the communication protocol.

[0070] The communication network uses the IEC 61850 standardized communication protocol to transmit control instructions from the central controller to each substation, and then forwarded by the substation to the execution device.

[0071] The execution equipment includes a reactive power compensator for regulating the reactive power output of the power grid and a distributed power supply device for adjusting its active power output according to control instructions.

[0072] The working process of the system for implementing the low-voltage distribution network voltage control method based on deep reinforcement learning is as follows: The central controller on the distribution transformer side sends the control instructions formed by the Transformer-SAC model (the control instructions include the capacitor switching adjustment of the reactive power compensator in the low-voltage distribution network and the active power output of the distributed power supply equipment) to each substation of the low-voltage distribution network through the standardized communication protocol IEC 61850. The substation, as the middle layer, is responsible for forwarding the received control instructions to the corresponding reactive power compensator and distributed power supply equipment. After receiving the control instructions, the reactive power compensator and distributed power supply equipment adjust their operating status according to the instructions, changing the power output of the distributed power supply equipment and the switching amount of the reactive power compensator capacitor. The environmental status of the equipment during execution is fed back to the Transformer-SAC model in real time through sensors. The Transformer-SAC model realizes voltage control and self-optimization of Transformer parameters within the Transformer-SAC model based on environmental feedback.

[0073] The control effect of the present invention is shown in the accompanying drawings of the specification, where the reward function curve of the Transformer-SAC model is as follows: Figure 2 As shown, the value function loss curve is as follows Figure 3 As shown in Figure 2, the Transformer-SAC model has begun to converge at 82 iterations. The final node voltage control effect is shown in Figure 2. Figure 4 As shown in Figure 2, the node out-of-limit situation disappears after 63 iterations.

[0074] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other modifications, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and should be included in the protection scope of the present invention.

Claims

1. A low-voltage distribution network voltage control method based on deep reinforcement learning, characterized in that: The following steps are involved: The trained Transformer-SAC model is embedded into the voltage control system of the low-voltage distribution network, so that the Transformer-SAC model interacts with the environment of the low-voltage distribution network to form an action strategy; The action strategy is sent to the execution device to control the power of the distributed voltage and the number of switching on and off of the reactive power compensator; After receiving the command, the execution device executes the action, and the low-voltage distribution network forms new state variables, which are fed back to the Transformer-SAC model in real time for optimization; The Transformer-SAC model interacts with the environment of the low-voltage distribution network to form an action strategy, including the following steps: S1. Use the Transformer network to extract the associated feature matrix from the state variables of the low-voltage distribution network. After the associated feature matrix is ​​processed by the Transformer network, the feature matrix is ​​output; S2. Input the feature matrix output by the Transformer network into the SAC algorithm to generate an action strategy.

2. The low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 1 is characterized in that: Step S1 specifically includes the following steps: S1.1, extracting node voltage, active power load, reactive power load, and node voltage sensitivity from the state variables of the low-voltage distribution network as a correlation feature matrix; S1.2, the associated feature matrix calculates the attention weight of each attention head through the multi-head attention mechanism; S1.3, concatenate the weights of all attention heads to obtain a concatenated feature matrix, and perform linear mapping; add the concatenated feature matrix to the associated feature moment to realize residual connection, and then process it through normalization to obtain a normalized feature matrix; S1.

4. Input the normalized feature matrix into the feedforward neural network to obtain the feature matrix processed by the feedforward neural network.

3. The low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 2 is characterized in that: The SAC algorithm in step S2 includes: The strategy optimization goal of the SAC algorithm is to maximize the expected return and strategy entropy: Experience replay: stores the interaction data between the agent and the environment, including the state, action, and reward of the environment feedback; renew Network: By minimizing Bellman error update of a function network; Update the policy network by maximizing the policy entropy and Value update policy network.

4. The low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 3 is characterized in that: The reward function used by the SAC algorithm is for: ； In the formula for The voltage over-limit penalty value at the moment, for Always control costs. for It is always used to punish the power loss of low voltage distribution network. for The node voltage sensitivity penalty value at the moment, , , , are the weights of the corresponding sub-items respectively; For Moment-by-moment decision making, for State variables of the low-voltage distribution network at each moment.

5. The low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 1 is characterized in that: The training of the Transformer-SAC model includes the following steps: (1) Based on the historical data of the low-voltage distribution network, a virtual environment of the low-voltage distribution network is constructed in the computer. The Transformer network obtains state variables through the virtual environment to extract the associated feature matrix and generate new state variables; (2) The new state variable is input into the action strategy obtained by the SAC algorithm and the action is output; (3) The execution device performs actions to change the state of the virtual environment of the low-voltage distribution network, and the changed state variables are fed back to the Transformer-SAC model; (4) The Transformer-SAC model uses the changed state variables for optimization; (5) Update action strategy of the optimized Transformer-SAC model; Repeat steps (3) to (5) to iterate the Transformer-SAC model learning until convergence, completing the Transformer-SAC model training.

6. The low-voltage distribution network voltage control method based on deep reinforcement learning according to any one of claims 1 to 5, characterized in that: The optimization of the Transformer-SAC model includes optimizing the parameters of the Transformer network, using the loss function that minimizes the mean square error, back propagation through the chain rule, and adaptive moment estimation optimizer to optimize and update the parameters of the Transformer network.

7. The low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 6 is characterized in that: The trained Transformer-SAC model is embedded in the voltage control system of the low-voltage distribution network, so that the Transformer-SAC model interacts with the environment of the low-voltage distribution network to form an action strategy. The following steps are involved: 1.

1. Generate action strategies based on the trained Transformer-SAC model based on the real-time power grid status and issue action instructions to the execution devices of the low-voltage distribution network; 1.

2. The execution device executes the selected control action to change the state of the actual low-voltage distribution network environment, and feeds the changed grid state back to the trained Transformer-SAC model; 1.

3. The Transformer network performs feature processing on the input power grid state and optimizes the weights and bias items of the Transformer network. The Transformer network extracts the latest associated feature matrix based on the real-time updated low-voltage distribution network state and inputs it into the SAC algorithm to generate a control strategy, completing the real-time iterative training of the Transformer-SAC model and the voltage regulation control of the low-voltage distribution network.

8. A system for implementing the low-voltage distribution network voltage control method based on deep reinforcement learning as described in claim 1, characterized in that: Including voltage control system and Transformer-SAC control module of low voltage distribution network; The Transformer-SAC control module includes a Transformer-SAC model, which is used to extract features from the power grid state and generate a control strategy; wherein the Transformer network is used to extract features and optimize the input features of the control strategy; the SAC algorithm is used to generate the optimized control strategy, through the strategy network and Network for training; The voltage control system of the low voltage distribution network includes: distribution network status monitoring system, central controller, and execution equipment; The distribution network status monitoring system monitors the status of the power grid in real time through sensors. The sensors feed real-time data back to the Transformer-SAC model to evaluate the effectiveness of the control strategy. The central controller sends the control instructions formed by the Transformer-SAC model to the execution device through the communication network; after receiving the control instructions, the execution device adjusts the operating status according to the instructions, and the environmental status of the equipment during execution is fed back to the Transformer-SAC model in real time through sensors; the Transformer-SAC model realizes voltage control and optimization of the Transformer network parameters within the Transformer-SAC model based on the feedback.

9. The system of the low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 8 is characterized in that: The execution device includes a reactive power compensator for regulating the reactive power output of the power grid and a distributed power supply device for adjusting the active power output according to a control instruction.

10. The system of the low-voltage distribution network voltage control method based on deep reinforcement learning according to claim 8 is characterized in that: The communication network adopts the IEC 61850 standardized communication protocol to transmit control instructions from the central controller to each substation, and the substation forwards the control instructions to the execution device.

Images