A reactive power optimization method for distribution networks based on data augmentation and hybrid augmented intelligence

Abstract

This invention discloses a reactive power optimization method for distribution networks based on data augmentation and hybrid augmented intelligence. The method includes the following steps: establishing a diffusion model to expand the optical load training dataset; establishing a Markov decision model for coordinated active and reactive power optimization of the distribution network; constructing a T-MAAC offline centralized training model based on the Markov decision model, inputting the optical load training dataset to train the T-MAAC offline centralized training model offline, obtaining the trained T-MAAC offline centralized training model; deploying the trained T-MAAC offline centralized training model to a platform, generating optimization strategies online in real time, and executing the optimization strategies to achieve reactive power optimization of the distribution network. This invention can provide real-time voltage control strategies, adapt to source-load uncertainties, and has good real-time performance and control economy.

Description

Technical Field

[0001] This invention relates to the fields of power system operation optimization control and computer technology, specifically to a method for coordinated optimization of active and reactive power in distribution networks based on reinforcement learning. Background Technology

[0002] Following the introduction of the dual-carbon strategy and the county-wide rooftop photovoltaic policy, the penetration rate of distributed rooftop photovoltaic (PV) in distribution networks has been continuously increasing, posing a significant challenge to the operation and control of distribution systems. The large-scale integration of PV has altered the power flow distribution of traditional distribution networks, leading to backflow and significantly impacting voltage quality and power system reliability. Due to the fluctuating, random, and intermittent nature of PV output, voltage quality is a particularly noteworthy technical challenge as PV penetration continues to rise. Voltage quality is a crucial indicator of power system quality, and among various power quality issues, excessive voltage fluctuations cause the most widespread damage: directly affecting the performance of electrical equipment and hindering the stable and safe operation of the system. With the rapid development of distributed energy and electric vehicles, distribution networks, as the primary support for the integration of distributed energy and electric vehicle charging stations, also face significant challenges to economic, safe, and stable operation. Reactive power optimization and voltage control, as key technologies for distribution network operation optimization, optimize the overall reactive power flow of the power grid by adjusting reactive power compensation equipment in the distribution network. This plays a vital role in optimizing voltage distribution, mitigating voltage limits, reducing line losses, and improving power quality. Given the current high proportion of renewable energy integration and the participation of diverse adjustable equipment, the accuracy and real-time performance of reactive power and voltage optimization control technology are crucial to ensuring the integration and consumption of renewable energy and maintaining the economical and safe operation of the distribution network.

[0003] Traditional distribution network voltage control primarily relies on reactive power control, mainly adjusting the tap positions of on-load tap changers (OLTCs) and the number of capacitor banks switched on and off to regulate the distribution network voltage. However, OLTCs and capacitor banks have slow response times and high maintenance costs, making them unsuitable for frequent operation and difficult to apply to voltage fluctuations caused by large-scale distributed generation (DG) power fluctuations. Furthermore, due to the high resistance and reactance ratio (R / X) of distribution lines, simple reactive power control is insufficient to address voltage fluctuations caused by DG and load active power fluctuations.

[0004] Traditional mathematical optimization methods, intelligent optimization algorithms, and stochastic programming methods cannot effectively cope with highly uncertain environments and suffer from problems such as large computational load, easy getting trapped in local optima, dependence on accurate parameters and models, and difficulty in achieving online control.

[0005] The power system contains massive amounts of data, providing data support for data-driven artificial intelligence methods. The core of data-driven technology is to analyze massive historical data using machine learning methods to extract high-density valuable information, providing decision support for the planning and operation of the distribution network. Deep reinforcement learning (DRL) combines the perceptual advantages of deep learning with the decision-making advantages of reinforcement learning. As a type of sequential decision algorithm, it has a high structural fit with the reactive power optimization problem of the distribution network. DRL learns generalizable policies by interacting with complex environments, without needing to adjust analytical models or resolve specific decisions.

[0006] However, DRL requires a large amount of data samples, including numerous environmental interaction samples. If the sample size is insufficient, the application effect will be significantly reduced. Furthermore, the neural networks in deep reinforcement learning methods have limited ability to extract features, and because control decisions are directly provided by the neural network, the underlying mechanisms are unclear, exhibiting a certain black-box nature. DRL uses historical data to train agents offline, and after training, the agents are deployed online. During system operation, extreme situations that were not encountered during the agent's training phase may occur. In such cases, the feasibility and security of the solutions provided by the agent cannot be guaranteed, limiting its application in practical power engineering.

[0007] The existing technologies, such as Chinese patent documents CN113363998A, CN113363997A, CN111799808A, CN111884213A; and the prior art solutions disclosed in the paper (Liao Wenlong, Yu Yun, Wang Yusen, et al. Reactive power optimization of distribution network based on graph convolutional network [J]. Power System Technology, 2021, 45(6): 2150-2160.), have the following disadvantages:

[0008] 1) Traditional mathematical optimization methods, intelligent optimization algorithms and stochastic programming methods cannot effectively cope with highly uncertain environments, and have problems such as large computational load, easy to get trapped in local optima, dependence on accurate parameters and models, and difficulty in achieving online control;

[0009] 2) Supervised learning-based methods require a certain number of optimization methods to find the optimal decision as the "label" during the offline training phase. This process usually takes a long time and has a high computational cost.

[0010] 3) Reinforcement learning methods such as DDPG and DQN treat the regulating device as a single agent, which makes it difficult to scale to larger systems. As the number of regulating devices increases, the dimension of the action space grows explosively, making it difficult to effectively provide a better policy. MADDPG, a typical representative of multi-agent reinforcement learning algorithms, suffers from problems such as overestimation of q-values ​​and training instability. While algorithms such as multi-agent twin delayed deep deterministic policy gradient (MATD3) have suppressed the overestimation problem to some extent, the input information of the network also increases as the number of agents increases. When scaled to large-scale systems with many agents, the algorithm becomes inefficient and performs poorly.

[0011] 4) Reinforcement learning, as a data-driven method, requires a large number of samples, has high data collection costs, and low sampling efficiency. The coordinated optimization of active and reactive power in power distribution networks typically faces complex operating scenarios, such as load fluctuations and drastic changes in photovoltaic output. Existing reinforcement learning methods suffer from low-quality training datasets, uneven data distribution, and difficulty in covering various source-load output conditions. Although historical data can provide some data sources, the constantly changing structure and scale of the power system result in a limited number of effective data points and low sampling efficiency, making reinforcement learning models poorly adaptable to variable environments. Furthermore, traditional data generation methods based on generative adversarial networks are prone to pattern collapse during training, leading to training failure.

[0012] 5) Existing reinforcement learning methods have limited representation capabilities in neural network architectures. When facing large-scale distribution networks, they are often difficult to optimize effectively due to the large state space and long-term dependencies. Furthermore, distribution networks often face changes in environment and load in practical applications, while traditional reinforcement learning methods often lack generalization ability when dealing with diverse and rapidly changing environments.

[0013] 6) Existing reinforcement learning methods struggle to handle complex agent action constraints. When formulating reward functions, they typically guide the agent to find feasible solutions by adding penalty terms, a method that cannot guarantee the feasibility of the solution. Furthermore, since control decisions are directly provided by the neural network with unclear mechanisms, existing reinforcement learning methods generally employ an "offline training, online execution" framework. After offline training, the agent is deployed online. During system operation, extreme situations that were not encountered during the agent's training phase may arise, making it difficult to guarantee the security of the solutions provided by the agent. Summary of the Invention

[0014] To address the aforementioned issues, this invention proposes a method for coordinated active and reactive power optimization in distribution networks based on data augmentation and hybrid augmented intelligence. First, future scheduling scenario data is expanded using a generative diffusion model, and historical operational data is aggregated to form a high-quality training dataset. Then, a partially observable Markov decision process (POMDP) ​​is designed based on the distribution network active and reactive power coordinated optimization model, and a Transformer-based Multi-agent actor-attention-critic (T-MAAC) algorithm embedded with a Transformer is used to solve the model. Finally, using the trained model, an action safety verification module is introduced during the online execution phase based on a hybrid augmented intelligence paradigm to strictly ensure the safety of the output actions. The diffusion model improves the quality and diversity of the data, while the Transformer model helps improve the DRL representation capability, significantly enhancing the algorithm's stability and learning efficiency. Through an offline training-online execution hybrid intelligent framework, coordinated active and reactive power optimization control of photovoltaic inverters and distributed energy storage in distribution networks is achieved. The proposed method can provide real-time voltage control strategies, adapt to source-load uncertainties, and exhibits good real-time performance and control economy.

[0015] The objective of this invention is achieved by at least one of the following technical solutions.

[0016] A reactive power optimization method for distribution networks based on data augmentation and hybrid augmented intelligence includes the following steps:

[0017] S1. Establish a diffusion model to expand the photocharge training dataset;

[0018] S2. Establish a Markov decision model for the coordinated optimization of active and reactive power in the distribution network;

[0019] S3. Construct a T-MAAC offline centralized training model based on the Markov decision model, input the photocharge training dataset to train the T-MAAC offline centralized training model offline, and obtain the trained T-MAAC offline centralized training model.

[0020] S4. Using the trained T-MAAC offline centralized training model, online optimization strategies are generated in real time, and the optimization strategies are executed to achieve reactive power optimization of the distribution network.

[0021] Further, step S1 includes the following steps:

[0022] S1.1 Data Acquisition and Preprocessing:

[0023] Collect power distribution network operation data over a period of time, including photovoltaic active power output, load active power, and load reactive power; standardize the collected data to ensure that the data can be used for training on the same scale;

[0024] S1.2 Forward diffusion process:

[0025] Noise is gradually added to the prepared data to simulate different environmental or system disturbances; the noise addition process at each step can be represented by multiple time steps to gradually blur the data; a set of data is randomly sampled from the preprocessed photovoltaic active power output data sample, load active power data sample, and load reactive power data sample as the initial sample. , , , The initial values ​​for photovoltaic active power output, load active power, and load reactive power are given, and the number of diffusion steps is set. and noise parameters At every step Add noise to the sample step by step , Samples were obtained after being subjected to different levels of noise. , , and The first The photovoltaic active power, load active power, and load reactive power of the step;

[0026] The noise addition process is shown in the following formula:

[0027] ;

[0028] in, For the first Samples after step diffusion hour , This is the initial sample; For the first The standard Gaussian noise in the step diffusion is resampled at each step to ensure the randomness and diversity of each step; For the first The noise parameters of the step diffusion change with time step. Linearly increasing, with a growth rate of k ;

[0029] S1.3, Reverse Generation Process:

[0030] Through a reverse denoising process, these noisy samples are gradually restored to more realistic and diverse samples; a denoising neural network model is trained, with the noisy data samples as input. That is, the samples generated in the forward diffusion process in step S1.2, the output of which is the predicted noise. The training objective is to minimize the denoising error, that is, to minimize the difference between the predicted noise and the actual noise at each time step. During training, the network learns how to recover the original samples from the noisy samples by minimizing this objective and predicting the added noise as much as possible. This enables the denoising neural network model to accurately recover the original samples from the noisy data at each step. The clear photovoltaic power output and load power data are recovered, and the optimization objective function of the training process is optimized. as follows:

[0031] ;

[0032] in, For the first step in the forward diffusion process Standard Gaussian noise in step diffusion, The first prediction of the denoising neural network model The noise component added in the step, Indicates different initial samples and noise The expectation of a random sample is and The average of the squares of the differences are taken as the Euclidean distance (L2 norm) to fully account for the diversity of the samples;

[0033] After training, the trained denoising neural network model is used to analyze the first... Samples after step diffusion Gradual noise reduction is used to generate diverse photovoltaic power output and load power data, as shown in the following formula:

[0034] ;

[0035] Among them, the Samples after step diffusion As initial noise sample ; For the trained denoising neural network model;

[0036] S1.4, Generate training dataset:

[0037] The diverse photovoltaic power output and load power data generated in step S1.3 are added to the original dataset to construct a training dataset that includes diverse combinations of photovoltaic active power output, load active power, and load reactive power data.

[0038] Furthermore, in step S2, a Markov decision model for the coordinated optimization of active and reactive power in the distribution network is established. Coordinated optimization is achieved by optimizing the reactive power output of distributed photovoltaic power and the active power of distributed energy storage in the distribution network, as detailed below:

[0039] A distributed control framework is adopted, with the distribution network comprising multiple sub-regions and multiple nodes within each sub-region. Each node represents the location for setting up distributed photovoltaic (PV) or distributed energy storage within that sub-region. The local controllers of each sub-region act as intelligent agents, controlling the distributed PV and distributed energy storage within their respective sub-regions. The distribution network serves as a reinforcement learning environment, and the Markov Decision Model (POMDP) ​​for the distribution system is defined as follows: ,in For the number of agents, Discount factor;

[0040] For the node feature matrix and adjacency matrix The state set constituted, wherein the characteristic matrix Represents the characteristic information of all nodes. , , , , , and These represent the node's active power, reactive power, photovoltaic active power output, photovoltaic reactive power output, state of charge of distributed energy storage, and node voltage amplitude, respectively; node set. Represents all nodes in the distribution network topology. Indicates the first 1 node This represents the total number of nodes in the distribution network. include Voltage amplitude of each node; adjacency matrix Adjacency matrix represents the connectivity relationships in the topology. have OK Columns, adjacency matrix The first in Line number Column elements Indicates the first Nodes With the Nodes The connection relationship, when the first Nodes With the Nodes When there is a connection between them, the adjacency matrix The Middle Line number Column elements Otherwise, it is 0;

[0041] Observation set It represents the set of observation information of all agents. Each agent can only observe the state variables of the sub-region where the agent is located. The observation set includes the observations of all agents. The observations of each agent include the observations of all nodes in the sub-region where the agent is located.

[0042] Action Set It represents the set of actions of all intelligent agents, including the reactive power output of photovoltaic inverters and the charging and discharging power of distributed energy storage set up by all nodes in the sub-region where each intelligent agent is located;

[0043] T This invention is a state transition function that represents the probability of an agent transitioning from its current state to the next state after performing an action. It belongs to a model-free reinforcement learning method, which does not require a definite state transition probability. Furthermore, random state transitions can effectively cope with multiple uncertainties such as source load in the distribution network, and are suitable for the active and reactive power co-optimization problem of the distribution network.

[0044] Let the reward function represent the reward obtained by the agent after performing an action; based on the active and reactive power coordinated optimization model of the distribution network, the reward function is... as follows:

[0045] ;

[0046] ;

[0047] in, and These are the node voltage offsets. and network loss Additional penalty coefficient; To take a positive function, For the first Nodes voltage amplitude, and These are the upper and lower safety limits for node voltage, respectively.

[0048] Furthermore, in step S2, at each time step In this process, each agent makes the optimal action based on observations through an action network, and then performs the action according to the reward function. Each agent receives an immediate reward; the goal of each agent is to learn the optimal policy to maximize the cumulative reward; after all agents have performed the optimal action, the POMDP (Power Distribution System) will transition to the next state.

[0049] Furthermore, in step S3, in the T-MAAC offline centralized training model, the transformer module is embedded in the action network architecture of the agent, and the attention weight mechanism is embedded in the agent evaluation network. The action network and the evaluation network work together, and the agents are in a cooperative relationship. A homogeneous agent parameter sharing mechanism is adopted, that is, the action network and evaluation network of each agent are the same, so as to improve the performance and efficiency of the algorithm.

[0050] Furthermore, in the transformer module, the original state features are mapped to the input space to generate embedded representations for each agent:

[0051] ;

[0052] ;

[0053] in, For the first Embedded information of each intelligent agent; For the first The first subregion where the agent is located Observations of each node The projection, the projection layer This is a linear transformation that converts the original observations into an embedding vector; For the first The number of nodes in the sub-region where each agent is located. ; The original input matrix;

[0054] Based on the original input matrix This maps the embedded information into a space suitable for computational attention:

[0055] ;

[0056] in, It is the index of the transformer layer in the transformer module; , , They represent the first The Query matrix, Key matrix, and Value matrix of the transformer layer; , , These represent the learnable weight matrices used to calculate the Query matrix, Key matrix, and Value matrix, respectively. For the first Input matrix of the transformer layer;

[0057] A masked self-attention mechanism is used to establish the correlation between nodes within a region; the formula for calculating masked self-attention is as follows:

[0058] ;

[0059]

[0060] in, For the first The mask self-attention calculation value of the transformer layer; The function representing the self-attention mechanism computation with a masked matrix; For activation functions; This is the transpose of the Key matrix. This is the scaling factor, and its value is the dimension of the Query matrix; This is a mask matrix where each element is either 0 or 1. Adjacency matrix This ensures that attention weights are calculated only between topologically connected nodes, thereby avoiding interference from irrelevant nodes, making it more adaptable to the physical characteristics of the distribution network, and improving computing performance and efficiency.

[0061] Layer normalization and linear transformation are applied to the output of each transformer layer to update the embedded features, iteratively constructing higher-level embedded features, as follows:

[0062] ;

[0063] in, For the first Embedded features output by the transformer layer;

[0064] After multi-layer masking self-attention computation, an aggregation embedding module is used to aggregate the embedding features of all nodes:

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] in, , , These are the Query, Key, and Value matrices of the aggregation embedding module, respectively. , , These are the learnable parameters of the Query, Key, and Value matrices of the aggregation embedding module, respectively. The global features are weighted representations of the global features obtained using the standard attention computation mechanism; The global embedding features are obtained through further processing via residual connections and layer normalization; The local neighborhood features extracted through MaskAttention are the first... The embedded features output by the transformer layer fuse local and global features, ensuring that local features are not lost during the global weighting process, thus improving the stability and representational ability of the T-MAAC offline centralized training model.

[0070] The SelectEmbedding module is used to select all nodes that are configured for distributed photovoltaic or distributed energy storage for embedding, thereby obtaining global information from the perspective of the agent corresponding to the sub-region where the node is located:

[0071] ;

[0072] in, Indicates from the first Global information extracted from the perspective of an individual agent. For the first The first subregion where the agent is located One node;

[0073] Using a gated recurrent unit (GRU) module to Mapped to the first Action value of an agent :

[0074] .

[0075] Furthermore, the agent evaluation network The formula for calculating the value is as follows:

[0076] ;

[0077] in, For the first The current observation action of the agent Below value, To evaluate network parameters, To evaluate the multilayer perceptron in the network, To evaluate a single-layer perceptron in a network, For the first Observations of an intelligent agent For other intelligent agents to the first The weighted sum of the values ​​of each agent is as follows:

[0078] ;

[0079] in, For the first The first agent on the first Attention weights of each agent; For the first Embedded functions of an agent For activation function, For shared matrices;

[0080] ;

[0081] in, , The first is calculated by comparing their similarity. The first agent on the first Attention weights for each agent.

[0082] Furthermore, in step S3, the actual operating environment of the distribution network is simulated through power flow calculation; an experience pool is constructed using an experience replay mechanism to store samples obtained from the interaction between the agent and the environment, so as to improve training stability and learning efficiency; the action network is copied to construct the target action network; and the evaluation network is copied to construct the target evaluation network, so as to further improve the stability of the algorithm.

[0083] Furthermore, in step S3, offline training specifically includes the following steps:

[0084] S3.1 Randomly initialize the action network parameters of the T-MAAC offline centralized training model. and target action network parameters Evaluate network parameters and target evaluation network parameters ;

[0085] S3.2 Initialize the distribution network environment status Read the input photocharge training dataset and obtain the initial observations for each agent. Set the number of training steps to "step" and the number of training rounds to "episode".

[0086] S3.3 Generate the agent's actions based on the agent's action network. ;

[0087] S3.4, The agent executes actions Then, the reward value of the agent is returned. ;

[0088] S3.5, All agents execute actions. Afterwards, the power distribution network environment transitions to the next state. The agent acquires the observations at the next moment. ;

[0089] S3.6, Take the sample Store in experience pool In, among them, , For the first The initial observations of each agent, , For the first The action value of each agent. , For the first The reward value for each agent. , For the first The next observation value of each agent;

[0090] S3.7 Repeat steps S3.3 to S3.6 until the experience pool is reached. The sample size in the sample reaches the set threshold. No need to wait for the experience pool Fill the experience pool to accelerate the training process and improve efficiency; in subsequent training rounds, the experience pool will be filled. The sample size has reached its limit. Then, the experience pool is updated on a rolling basis using a FIFO (First In, First Out) strategy. ;

[0091] S3.8, The agent in the experience pool Randomly select a small batch of data, with the sample size being the batch size;

[0092] S3.9 Update the evaluation network according to the following loss minimization formula:

[0093] ;

[0094] in, For the first The update function of the evaluation network for each agent; In the experience pool The expected benefit of mid-sampling is the benefit of the current observation action pair. Next Evaluation network of individual agents value With the The objective function value of each agent square of the difference The average value is used to obtain the result. For the first The objective function values ​​for each agent are as follows:

[0095] ;

[0096] in, As a discount factor, To observe the action pair at the next moment Next A target evaluation network for individual agents value, To balance the parameters of maximum entropy and reward, No. Target Action Network of Individual Agents In the current observation Select action The probability of; for and The weighted average of the differences represents the first... Target Action Network of Individual Agents The weighted average of all possible actions observed at the next moment;

[0097] S3.10. Update the action network according to the gradient descent formula:

[0098] ;

[0099] ;

[0100] in, For the first Action network of individual agents The gradient of the update function; Let be the policy gradient part, representing the th... Action network of individual agents In the current observation Select action The gradient of the probability; This serves as a baseline to reduce the variance of the policy gradient. Indicates except the first Joint actions of other intelligent agents besides the individual intelligent agent;

[0101] S3.11. Update the target action network parameters and target evaluation network parameters for all agents:

[0102] ;

[0103] ;

[0104] in, To update the weights;

[0105] S3.12. Repeat steps S3.8 to S3.11 until all training steps are completed in this round.

[0106] S3.13. Repeat steps S3.2 to S3.12 until all rounds of training are completed, and obtain the trained T-MAAC offline centralized training model.

[0107] Furthermore, in step S4, the trained T-MAAC offline centralized training model is used to intelligently generate optimization strategies online, and the safety of the output actions is strictly guaranteed by introducing a hybrid augmented intelligence paradigm action safety verification module. The specific process is as follows:

[0108] S4.1 Load the action network of the agent in the T-MAAC offline centralized training model trained in step S3, and set the simulation duration. ;

[0109] S4.2. Real-time reading of optical load data from the distribution network to obtain the observation values ​​of the intelligent agent. ;

[0110] S4.3. Generate the current time step in real time based on the trained T-MAAC offline centralized training model. Action value of the agent ;

[0111] S4.4 Determine the action performed by the intelligent agent. The later Check whether the voltage levels of all nodes where photovoltaic inverters are located in the sub-region where the intelligent agent is located are within the safety constraints. If the constraints are met, execute the action. If the conditions are not met, the action will be corrected based on the voltage exceeding the limit, and then the action will be executed again. If the first Nodes with photovoltaic inverters are set up in the sub-region where each intelligent agent is located. The higher the voltage level, the more the PV action correction formula is shown below:

[0112] ;

[0113] in, and They are time points According to the Actions of an agent The reactive power output of the photovoltaic inverters in the corresponding sub-regions before and after PV operation correction. For a moment No. The voltage level of all photovoltaic inverter nodes in the sub-region corresponding to each intelligent agent. This is the upper limit of the node voltage. For a moment No. The voltage reactive power sensitivity of the node where the photovoltaic inverter is located in the sub-region corresponding to each intelligent agent represents the sensitivity of reactive power regulation to the voltage.

[0114] The node voltage safety constraints are as follows:

[0115] ;

[0116] in, This is the lower limit of the node voltage.

[0117] S4.5, Determine the action to be performed. The later If the state of charge (SOC) of all distributed energy storage systems (DESS) in the sub-region corresponding to each agent is within the constraints, and if not, the DESS action is corrected according to the limit exceedance; if the first agent... Each intelligent agent has a distributed energy storage (DESS) node set up in its sub-region. If the state of charge (SOC) exceeds the upper limit, the correction formula is as follows:

[0118] ;

[0119] in, For a moment According to the Actions of an agent The charging and discharging power of the distributed energy storage DESS in the corresponding sub-region after DESS action correction. This represents the upper limit of the DESS state of charge. For a moment No. The state of charge of distributed energy storage (DESS) in the sub-region where each agent is located. For DESS charging efficiency. This is the upper limit of DESS capacity;

[0120] The DESS safety constraints are as follows:

[0121] ;

[0122] in, and These are the upper and lower limits of the DESS state of charge, respectively.

[0123] The energy constraints for DESS are as follows:

[0124] ;

[0125] ;

[0126] ;

[0127] in, For DESS at time Energy levels; The discharge efficiency of DESS. and These are the upper and lower limits of the DESS capacity;

[0128] S4.6 Repeat steps S4.2 to S4.5 for all agents to obtain the results including time intervals. The corresponding time of all agents' actions Optimization strategy ;

[0129] S4.7 Repeat steps S4.2 to S4.6 until the entire online execution time is completed. The results include online execution time. Online execution time corresponding to the actions of all agents within the system Optimization strategy .

[0130] Compared with the prior art, the advantages of the present invention are:

[0131] To improve the data quality of the training set for reinforcement learning methods, this invention proposes a data augmentation method based on a generative diffusion model. This method uses a diffusion model to simulate the process of adding and removing noise, generating new samples that cover a wider range of photovoltaic power output and load fluctuation scenarios, thereby increasing the diversity of training samples and improving the generalization and robustness of the algorithm.

[0132] To improve the generalization ability of MADRL to diverse environments and its scalability to large systems, this invention integrates the advantages of deep learning and reinforcement learning, proposing a T-MAAC-based intelligent collaborative voltage control method for photovoltaic inverters. Specifically, a Transformer model is used to represent state information in the distribution network environment, handling long-term dependencies in sequential data. Its self-attention mechanism better captures the relationships and long-term dependencies between different nodes in the distribution network, improving the model's understanding of states at multiple time steps. An embedding module generates embedding representations containing global information, and a selective embedding module focuses on the embedding information of specific nodes, improving training efficiency and feature extraction capabilities through selective learning focus. While MAAC's attention mechanism is concentrated in the evaluation network and has limited information fusion capabilities, the Transformer can enhance the action network, improving policy optimization efficiency. MAAC incorporating the Transformer can handle more complex tasks and significantly improve the performance of multi-agent systems.

[0133] This invention avoids modeling the underlying physical model through model-free algorithm processing and adopts a framework of "offline centralized training and online distributed execution". In the offline learning stage, the optimal policy is mined from a large number of samples and stored in the policy network. During online decision-making, the optimal decision is quickly given based on the real-time status of the system, thus eliminating the online optimization process. This gives it good real-time performance and computational performance. In the online execution stage, an action safety verification mechanism is introduced based on the human-machine hybrid augmented intelligence paradigm. When the original action of the agent is evaluated to have the risk of exceeding the limit, it intervenes and prevents it, strictly ensuring that the output action of the agent is within the safety constraint range.

[0134] This invention effectively reduces system network losses and voltage deviations, improves system safety and stability, and supports the low-carbon, economical, safe, and efficient operation of the system. Furthermore, it designs a fully innovative reinforcement learning framework, comprehensively enhancing the practicality, generalization, and security of reinforcement learning in real-world active and reactive power co-optimization scenarios in power distribution networks. This framework encompasses three stages: pre-training data augmentation, the T-MAAC method during training, and the hybrid reinforcement intelligence method during execution. Attached Figure Description

[0135] Figure 1 This is a flowchart illustrating the steps involved in expanding the photocharge training dataset in an embodiment of the present invention.

[0136] Figure 2 This is a schematic diagram of the structure of the T-MAAC offline centralized training model in an embodiment of the present invention.

[0137] Figure 3 This is a flowchart illustrating the offline training and online execution steps in an embodiment of the present invention.

[0138] Figure 4 This is a flowchart illustrating the steps of a reactive power optimization method for a distribution network based on data augmentation and hybrid augmented intelligence, as described in an embodiment of the present invention. Detailed Implementation

[0139] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific implementation of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0140] Example:

[0141] A reactive power optimization method for distribution networks based on data augmentation and hybrid augmented intelligence includes the following steps:

[0142] S1. Establish a diffusion model to expand the photocharge training dataset, such as... Figure 1 As shown, it includes the following steps:

[0143] S1.1 Data Acquisition and Preprocessing:

[0144] Collect power distribution network operation data over a period of time, including photovoltaic active power output, load active power, and load reactive power; standardize the collected data to ensure that the data can be used for training on the same scale;

[0145] S1.2 Forward diffusion process:

[0146] Noise is gradually added to the prepared data to simulate different environmental or system disturbances; the noise addition process at each step can be represented by multiple time steps to gradually blur the data; a set of data is randomly sampled from the preprocessed photovoltaic active power output data sample, load active power data sample, and load reactive power data sample as the initial sample. , , , The initial values ​​for photovoltaic active power output, load active power, and load reactive power are given, and the number of diffusion steps is set. and noise parameters At every step Add noise to the sample step by step , Samples were obtained after being subjected to different levels of noise. , , and The first The photovoltaic active power, load active power, and load reactive power of the step;

[0147] The noise addition process is shown in the following formula:

[0148] ;

[0149] in, For the first Samples after step diffusion hour , This is the initial sample; For the first The standard Gaussian noise in the step diffusion is resampled at each step to ensure the randomness and diversity of each step; For the first The noise parameters of the step diffusion change with time step. Linearly increasing, with a growth rate of k ;

[0150] S1.3, Reverse Generation Process:

[0151] Through a reverse denoising process, these noisy samples are gradually restored to more realistic and diverse samples; a denoising neural network model is trained, with the noisy data samples as input. That is, the samples generated in the forward diffusion process in step S1.2, the output of which is the predicted noise. The training objective is to minimize the denoising error, that is, to minimize the difference between the predicted noise and the actual noise at each time step. During training, the network learns how to recover the original samples from the noisy samples by minimizing this objective and predicting the added noise as much as possible. This enables the denoising neural network model to accurately recover the original samples from the noisy data at each step. The clear photovoltaic power output and load power data are recovered, and the optimization objective function of the training process is optimized. as follows:

[0152] ;

[0153] in, For the first step in the forward diffusion process Standard Gaussian noise in step diffusion, The first prediction of the denoising neural network model The noise component added in the step, Indicates different initial samples and noise The expectation of a random sample is and The average of the squares of the differences are taken as the Euclidean distance (L2 norm) to fully account for the diversity of the samples;

[0154] After training, the trained denoising neural network model is used to analyze the first... Samples after step diffusion Gradual noise reduction is used to generate diverse photovoltaic power output and load power data, as shown in the following formula:

[0155] ;

[0156] Among them, the Samples after step diffusion As initial noise sample ; For the trained denoising neural network model;

[0157] S1.4, Generate training dataset:

[0158] The diverse photovoltaic power output and load power data generated in step S1.3 are added to the original dataset to construct a training dataset that includes diverse combinations of photovoltaic active power output, load active power, and load reactive power data.

[0159] S2. Establish a Markov decision model for the coordinated optimization of active and reactive power in the distribution network. Coordinated optimization is achieved by optimizing the reactive power output of distributed photovoltaic power and the active power of distributed energy storage in the distribution network, as detailed below:

[0160] A distributed control framework is adopted, with the distribution network comprising multiple sub-regions and multiple nodes within each sub-region. Each node represents the location for setting up distributed photovoltaic (PV) or distributed energy storage within that sub-region. The local controllers of each sub-region act as intelligent agents, controlling the distributed PV and distributed energy storage within their respective sub-regions. The distribution network serves as a reinforcement learning environment, and the Markov Decision Model (POMDP) ​​for the distribution system is defined as follows: ,in For the number of agents, Discount factor;

[0161] For the node feature matrix and adjacency matrix The state set constituted, wherein the characteristic matrix Represents the characteristic information of all nodes. , , , , , and These represent the node's active power, reactive power, photovoltaic active power output, photovoltaic reactive power output, state of charge of distributed energy storage, and node voltage amplitude, respectively; node set. Represents all nodes in the distribution network topology. Indicates the first 1 node This represents the total number of nodes in the distribution network. include Voltage amplitude of each node; adjacency matrix Adjacency matrix represents the connectivity relationships in the topology. have OK Columns, adjacency matrix The first in Line number Column elements Indicates the first Nodes With the Nodes The connection relationship, when the first Nodes With the Nodes When there is a connection between them, the adjacency matrix The Middle Line number Column elements Otherwise, it is 0;

[0162] Observation set It represents the set of observation information of all agents. Each agent can only observe the state variables of the sub-region where the agent is located. The observation set includes the observations of all agents. The observations of each agent include the observations of all nodes in the sub-region where the agent is located.

[0163] Action Set It represents the set of actions of all intelligent agents, including the reactive power output of photovoltaic inverters and the charging and discharging power of distributed energy storage set up by all nodes in the sub-region where each intelligent agent is located;

[0164] T This invention is a state transition function that represents the probability of an agent transitioning from its current state to the next state after performing an action. It belongs to a model-free reinforcement learning method, which does not require a definite state transition probability. Furthermore, random state transitions can effectively cope with multiple uncertainties such as source load in the distribution network, and are suitable for the active and reactive power co-optimization problem of the distribution network.

[0165] Let the reward function represent the reward obtained by the agent after performing an action; based on the active and reactive power coordinated optimization model of the distribution network, the reward function is... as follows:

[0166] ;

[0167] ;

[0168] in, and These are the node voltage offsets. and network loss The additional penalty coefficient, in one embodiment, network loss The results can be directly obtained by calling power flow calculation packages such as pandapower or matpower; To take a positive function, For the first Nodes voltage amplitude, and These are the upper and lower safety limits for node voltage, respectively.

[0169] At each time step In this process, each agent makes the optimal action based on observations through an action network, and then performs the action according to the reward function. Each agent receives an immediate reward; the goal of each agent is to learn the optimal policy to maximize the cumulative reward; after all agents have performed the optimal action, the POMDP (Power Distribution System) will transition to the next state.

[0170] S3. Construct a T-MAAC offline centralized training model based on the Markov decision model, input the photocharge training dataset to train the T-MAAC offline centralized training model offline, and obtain the trained T-MAAC offline centralized training model.

[0171] In the T-MAAC offline centralized training model, such as Figure 2 As shown, the transformer module is embedded in the action network architecture of the agent, and the attention weight mechanism is embedded in the agent evaluation network. The action network and the evaluation network work together, and the agents are in a cooperative relationship. A homogeneous agent parameter sharing mechanism is adopted, that is, the action network and evaluation network of each agent are the same, so as to improve the performance and efficiency of the algorithm.

[0172] In one embodiment, an agent action network and an evaluation network are constructed based on existing technologies (LOWE R, WU Yi, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C] / / Proceedings of the 31st International Conference on Neural Information Processing Systems. California, USA: Curran Associates Inc., 2017: 6382-6393).

[0173] In the transformer module, the original state features are mapped to the input space to generate the embedded representations of each agent:

[0174] ;

[0175] ;

[0176] in, For the first Embedded information of each intelligent agent; For the first The first subregion where the agent is located Observations of each node The projection, the projection layer This is a linear transformation that converts the original observations into an embedding vector; For the first The number of nodes in the sub-region where each agent is located. ; The original input matrix;

[0177] Based on the original input matrix This maps the embedded information into a space suitable for computational attention:

[0178] ;

[0179] in, It is the index of the transformer layer in the transformer module; , , They represent the first The Query matrix, Key matrix, and Value matrix of the transformer layer; , , These represent the learnable weight matrices used to calculate the Query matrix, Key matrix, and Value matrix, respectively. For the first Input matrix of the transformer layer;

[0180] A masked self-attention mechanism is used to establish the correlation between nodes within a region; the formula for calculating masked self-attention is as follows:

[0181] ;

[0182]

[0183] in, For the first The mask self-attention calculation value of the transformer layer; The function representing the self-attention mechanism computation with a masked matrix; For activation functions; This is the transpose of the Key matrix. This is the scaling factor, and its value is the dimension of the Query matrix; This is a mask matrix where each element is either 0 or 1. Adjacency matrix This ensures that attention weights are calculated only between topologically connected nodes, thereby avoiding interference from irrelevant nodes, making it more adaptable to the physical characteristics of the distribution network, and improving computing performance and efficiency.

[0184] Layer normalization and linear transformation are applied to the output of each transformer layer to update the embedded features, iteratively constructing higher-level embedded features, as follows:

[0185] ;

[0186] in, For the first Embedded features output by the transformer layer;

[0187] After multi-layer masking self-attention computation, an aggregation embedding module is used to aggregate the embedding features of all nodes:

[0188] ;

[0189] ;

[0190] ;

[0191] ;

[0192] in, , , These are the Query, Key, and Value matrices of the aggregation embedding module, respectively. , , These are the learnable parameters of the Query, Key, and Value matrices of the aggregation embedding module, respectively. The global features are weighted representations of the global features obtained using the standard attention computation mechanism; The global embedding features are obtained through further processing via residual connections and layer normalization; The local neighborhood features extracted through MaskAttention are the first... The embedded features output by the transformer layer fuse local and global features, ensuring that local features are not lost during the global weighting process, thus improving the stability and representational ability of the T-MAAC offline centralized training model.

[0193] The SelectEmbedding module is used to select all nodes that are configured for distributed photovoltaic or distributed energy storage for embedding, thereby obtaining global information from the perspective of the agent corresponding to the sub-region where the node is located:

[0194] ;

[0195] in, Indicates from the first Global information extracted from the perspective of an individual agent. For the first The first subregion where the agent is located One node;

[0196] Using a gated recurrent unit (GRU) module to Mapped to the first Action value of an agent :

[0197] .

[0198] Agent evaluation network The formula for calculating the value is as follows:

[0199] ;

[0200] in, For the first The current observation action of the agent Below value, To evaluate network parameters, To evaluate the multilayer perceptron in the network, To evaluate a single-layer perceptron in a network, For the first Observations of an intelligent agent For other intelligent agents to the first The weighted sum of the values ​​of each agent is as follows:

[0201] ;

[0202] in, For the first The first agent on the first Attention weights of each agent; For the first Embedded functions of an agent For activation function, For shared matrices;

[0203] ;

[0204] in, , The first is calculated by comparing their similarity. The first agent on the first Attention weights for each agent.

[0205] The algorithm simulates the actual operating environment of a power distribution network through power flow calculation; it constructs an experience pool using an experience replay mechanism to store samples obtained from the interaction between the agent and the environment, thereby improving training stability and learning efficiency; it replicates the action network to construct the target action network and replicates the evaluation network to construct the target evaluation network, thereby further improving the stability of the algorithm.

[0206] like Figure 3 As shown, offline training specifically includes the following steps:

[0207] S3.1 Randomly initialize the action network parameters of the T-MAAC offline centralized training model. and target action network parameters Evaluate network parameters and target evaluation network parameters ;

[0208] S3.2 Initialize the distribution network environment status Read the input photocharge training dataset and obtain the initial observations for each agent. Set the number of training steps to "step" and the number of training rounds to "episode".

[0209] S3.3 Generate the agent's actions based on the agent's action network. ;

[0210] S3.4, The agent executes actions Then, the reward value of the agent is returned. ;

[0211] S3.5, All agents execute actions. Afterwards, the power distribution network environment transitions to the next state. The agent acquires the observations at the next moment. ;

[0212] S3.6, Take the sample Store in experience pool In, among them, , For the first The initial observations of each agent, , For the first The action value of each agent. , For the first The reward value for each agent. , For the first The next observation value of each agent;

[0213] S3.7 Repeat steps S3.3 to S3.6 until the experience pool is reached. The sample size in the sample reaches the set threshold. In subsequent training rounds, in the experience pool The sample size has reached its limit. Then, the experience pool is updated on a rolling basis using a FIFO (First In, First Out) strategy. ;

[0214] S3.8, The agent in the experience pool Randomly select a small batch of data, with the sample size being the batch size;

[0215] S3.9 Update the evaluation network according to the following loss minimization formula:

[0216] ;

[0217] in, For the first The update function of the evaluation network for each agent; In the experience pool The expected benefit of mid-sampling is the benefit of the current observation action pair. Next Evaluation network of individual agents value With the The objective function value of each agent square of the difference The average value is used to obtain the result. For the first The objective function values ​​for each agent are as follows:

[0218] ;

[0219] in, As a discount factor, in one embodiment, Take 0.99, To observe the action pair at the next moment Next A target evaluation network for individual agents value, To balance the parameters of maximum entropy and reward, in one embodiment, Take 0.01, No. Target Action Network of Individual Agents In the current observation Select action The probability of; for and The weighted average of the differences represents the first... Target Action Network of Individual Agents The weighted average of all possible actions observed at the next moment;

[0220] S3.10. Update the action network according to the gradient descent formula:

[0221] ;

[0222] ;

[0223] in, For the first Action network of individual agents The gradient of the update function; Let be the policy gradient part, representing the th... Action network of individual agents In the current observation Select action The gradient of the probability; This serves as a baseline to reduce the variance of the policy gradient. Indicates except the first Joint actions of other intelligent agents besides the individual intelligent agent;

[0224] S3.11. Update the target action network parameters and target evaluation network parameters for all agents:

[0225] ;

[0226] ;

[0227] in, In one embodiment, to update the weights, Take 0.995;

[0228] S3.12. Repeat steps S3.8 to S3.11 until all training steps are completed in this round.

[0229] S3.13. Repeat steps S3.2 to S3.12 until all rounds of training are completed, and obtain the trained T-MAAC offline centralized training model.

[0230] S4. Deploy the trained T-MAAC offline centralized training model to the computer system platform of the actual distribution network, generate optimization strategies online in real time, and strictly ensure the safety of output actions by introducing a hybrid augmented intelligence paradigm action safety verification module. Execute the optimization strategy to achieve reactive power optimization of the distribution network, such as... Figure 3 As shown, the specific process is as follows:

[0231] S4.1 Load the action network of the agent in the T-MAAC offline centralized training model trained in step S3, and set the simulation duration. ;

[0232] S4.2 In one embodiment, the optical load data in the system is read in real time through the computer system platform of the actual distribution network to obtain the observation value of the intelligent agent. ;

[0233] S4.3. Generate the current time step in real time based on the trained T-MAAC offline centralized training model. Action value of the agent ;

[0234] S4.4 Determine the action performed by the intelligent agent. The later Check whether the voltage levels of all nodes where photovoltaic inverters are located in the sub-region where the intelligent agent is located are within the safety constraints. If the constraints are met, execute the action. If the conditions are not met, the action will be corrected based on the voltage exceeding the limit, and then the action will be executed again. If the first Nodes with photovoltaic inverters are set up in the sub-region where each intelligent agent is located. The higher the voltage level, the more the PV action correction formula is shown below:

[0235] ;

[0236] in, and They are time points According to the Actions of an agent The reactive power output of the photovoltaic inverters in the corresponding sub-regions before and after PV operation correction. For a moment No. The voltage level of all photovoltaic inverter nodes in the sub-region corresponding to each intelligent agent. This is the upper limit of the node voltage. For a moment No. The voltage reactive power sensitivity of the node where the photovoltaic inverter is located in the sub-region corresponding to each intelligent agent represents the sensitivity of reactive power regulation to the voltage.

[0237] The node voltage safety constraints are as follows:

[0238] ;

[0239] in, This is the lower limit of the node voltage.

[0240] S4.5, Determine the action to be performed. The later If the state of charge (SOC) of all distributed energy storage systems (DESS) in the sub-region corresponding to each agent is within the constraints, and if not, the DESS action is corrected according to the limit exceedance; if the first agent... Each intelligent agent has a distributed energy storage (DESS) node set up in its sub-region. If the state of charge (SOC) exceeds the upper limit, the correction formula is as follows:

[0241] ;

[0242] in, For a moment According to the Actions of an agent The charging and discharging power of the distributed energy storage DESS in the corresponding sub-region after DESS action correction. This represents the upper limit of the DESS state of charge. For a moment No. The state of charge of distributed energy storage (DESS) in the sub-region where each agent is located. For DESS charging efficiency. This is the upper limit of DESS capacity;

[0243] The DESS safety constraints are as follows:

[0244] ;

[0245] in, and These are the upper and lower limits of the DESS state of charge, respectively.

[0246] The energy constraints for DESS are as follows:

[0247] ;

[0248] ;

[0249] ;

[0250] in, For DESS at time Energy levels; The discharge efficiency of DESS. and These are the upper and lower limits of the DESS capacity;

[0251] S4.6 Repeat steps S4.2 to S4.5 for all agents to obtain the results including time intervals. The corresponding time of all agents' actions Optimization strategy ;

[0252] S4.7 Repeat steps S4.2 to S4.6 until the entire online execution time is completed. The results include online execution time. Online execution time corresponding to the actions of all agents within the system Optimization strategy .

[0253] The preferred embodiments disclosed above are merely illustrative of the invention and its core concepts. Those skilled in the art will recognize that variations may occur in specific application scenarios and implementation based on the ideas of this invention; therefore, this specification should not be construed as limiting the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A power distribution network reactive power optimization method based on data enhancement and hybrid enhancement intelligence, characterized in that, Comprise the following steps: S1, establish diffusion model expansion light load training data set; Specifically, it comprises: S1.1, data acquisition and pretreatment: Collect the operation data of the distribution network in a period of time, including photovoltaic active power, load active power and load reactive power; The collected data is standardized; S1.2, forward diffusion process: Randomly sample a set of data from the pre-processed photovoltaic active power output data sample, load active power data sample and load reactive power data sample as initial samples , , , are the initial values of photovoltaic active power output, load active power and load reactive power respectively, set the number of diffusion steps and noise parameters , add noise to the samples step by step at each step , , , obtain samples affected by different noises , , and are the photovoltaic active power, load active power and load reactive power of the first step respectively S1.3, reverse generation process: training a denoising neural network model, the input of the denoising neural network model being a data sample with noise i.e. the sample generated by the forward diffusion process in step S1.2, the output being predicted noise the goal of the training being to minimize the denoising error, i.e. to minimize the difference between the predicted noise by the model at each time step and the true noise; S1.4, generate training data set: Add the diversified photovoltaic output and load power data generated in step S1.3 to the original data set, thereby constructing a training data set comprising diversified photovoltaic active power, load active power and load reactive power data combination; S2, establish Markov decision model of active and reactive power collaborative optimization of distribution network; The Markov decision model of active and reactive power collaborative optimization of distribution network is established, and the collaborative optimization is realized by optimizing the reactive power of distributed photovoltaic and the active power of distributed energy storage, as follows: Adopting a partitioned decentralized control framework, the power distribution network includes a plurality of sub-regions, and each sub-region includes a plurality of nodes, each node being a location where distributed photovoltaic or distributed energy storage is arranged in the sub-region, and a local controller of each sub-region is an intelligent agent, the distributed photovoltaic and the distributed energy storage in the sub-region are controlled through the local controller, the power distribution network is a reinforcement learning environment, and a Markov decision model POMDP is defined as wherein is the number of intelligent agents, is a discount factor. For the node feature matrix and adjacency matrix The state set constituted, wherein the characteristic matrix Represents the feature information of all nodes. , , , , , and These represent the node's active power, reactive power, photovoltaic active power output, photovoltaic reactive power output, state of charge of distributed energy storage, and node voltage amplitude, respectively; node set. Represents all nodes in the distribution network topology. Indicates the first 1 node This represents the total number of nodes in the distribution network. include Voltage amplitude of each node; adjacency matrix Adjacency matrix represents the topological connectivity relationships. have OK Columns, adjacency matrix The first in Line number Column elements Indicates the first Nodes With the Nodes The connection relationship, when the first Nodes With the Nodes When there is a connection between them, the adjacency matrix The Middle Line number Column elements Otherwise, it is 0; observation set represents the observation information set of all agents, each agent can only observe the state quantity of the sub-region where the agent is located, the observation set includes the observation quantities of all agents, and the observation quantity of each agent includes the observation quantities of all nodes in the sub-region where the agent is located; Action set denotes the action set of all agents, including the reactive power output of the photovoltaic inverter and the charging and discharging power of the distributed energy storage set by all nodes in the sub-area where each agent is located; T a state transition function representing a probability of the agent transitioning from a current state to a next state after performing an action; a reward function for representing a reward obtained by the intelligent agent after performing an action; according to the active and reactive power collaborative optimization model of the power distribution network, the reward function is as follows: ； ； wherein, and are additional penalty coefficients for nodal voltage deviation and network loss respectively; is a positive function, is the voltage amplitude of the th node , and and are the upper and lower security limits of the nodal voltage respectively. At each time step each agent makes an optimal action according to the observation through the action network, and then obtains an immediate reward according to the reward function The goal of each agent is to learn an optimal policy to maximize the cumulative reward; after all agents perform the optimal action, the POMDP will be transferred to the next state; S3, based on the Markov decision model, construct T-MAAC offline centralized training model, input light load training data set to train T-MAAC offline centralized training model offline, and get trained T-MAAC offline centralized training model; In the T-MAAC offline centralized training model, the transformer module is embedded in the action network architecture of the agent, and the attention weight mechanism is embedded in the evaluation network of the agent, the action network and the evaluation network are mutually cooperative, each agent is in cooperative relationship, and the isomorphic agent parameter sharing mechanism is adopted, that is, the action network and the evaluation network of each agent are the same; In the transformer module, the original state feature is mapped to the input space to generate the embedding representation of each agent: ； ； wherein, is the embedding information of the th agent; is the embedding information of the th agent; is the projection of the observation of the th node in the sub-region where the th agent is located, the projection layer is a linear transformation that converts the original observation into an embedding vector; is the number of nodes in the sub-region where the th agent is located, is the original input matrix; According to the original input matrix mapping the embedded information into a space suitable for computing attention: ； wherein, is an index of the number of transformer layers in the transformer module; , , respectively represent the Query matrix, Key matrix, Value matrix of the layer transformer layer; , , respectively represent the learnable weight matrix for calculating the Query matrix, Key matrix, Value matrix; is the input matrix of the layer transformer layer; The correlation between nodes in the region is established by using the masked self-attention mechanism; The calculation formula of the masked self-attention is as follows: ； wherein, is the first transformer layer, is the mask self-attention computation value of the transformer layer, represents the self-attention mechanism computation function with a mask matrix, is an activation function, is the transpose of the Key matrix, is a scaling factor, whose value is the dimension of the Query matrix, is a mask matrix, where the elements are 0 or 1, and the mask matrix is an adjacency matrix ; In the output of each layer of transformer layer, the embedding feature is updated by applying layer normalization and linear transformation, and high-level embedding features are iteratively constructed, as follows: ； wherein, is the first embedding features output by the layer transformer layer; After the multi-layer masked self-attention calculation, an aggregation embedding module is used to aggregate the embedding features of all nodes: ； ； ； ； in, , , These are the Query, Key, and Value matrices of the aggregation embedding module, respectively. , , These are the learnable parameters of the Query, Key, and Value matrices of the aggregation embedding module, respectively. The global features are weighted representations of the global features obtained using the standard attention computation mechanism; The global embedding features are obtained through further processing via residual connections and layer normalization; The local neighborhood features extracted through MaskAttention are the first... The embedded features output by the transformer layer fuse local and global features, ensuring that local features are not lost during the global weighting process, thus improving the stability and representational ability of the T-MAAC offline centralized training model. The select embedding module SelectEmbedding is used to select the node embedding of all distributed photovoltaic or distributed energy storage nodes to obtain the global information from the perspective of the agent corresponding to the subregion where the node is located: ； wherein, represents global information extracted from the perspective of the i-th agent, is the j-th node of the sub-region in which the i-th agent is located, is the j-th node of the sub-region in which the i-th agent is located,​​ using a gated recurrent unit module to map the state s to a value function v(s) for the first agent to a value function v(s) for the first agent to a value function v(s) for the first agent : ； S4, use the trained T-MAAC offline centralized training model to generate optimization strategy online in real time, and execute the optimization strategy to realize distribution network reactive power optimization.

2. The power distribution network reactive power optimization method based on data enhancement and hybrid enhancement intelligence of claim 1, wherein, the agent evaluation network The value calculation formula is as follows: ； wherein, is the value of the current observation of the th agent under the action is the evaluation network parameter, is a multi-layer perceptron in the evaluation network, is a single-layer perceptron in the evaluation network, is the observation of the th agent, is the weighted sum of the values of the th agent by other agents, as follows:​​ （19）； wherein, is the th attention weight for the th agent; is the th agent's embedding function, is an activation function; ； wherein, , , the attention weight of the first agent to the first agent is calculated by comparing their similarities.

3. The power distribution network reactive power optimization method based on data enhancement and hybrid enhancement intelligence of claim 2, wherein, In step S3, the operation environment of the actual distribution network is simulated by power flow calculation; The experience pool is constructed by using the experience replay mechanism, the samples obtained by the interaction between the agent and the environment are stored, the training stability and learning efficiency are improved, the target action network is constructed by copying the action network, and the target evaluation network is constructed by copying the evaluation network.

4. The power distribution network reactive power optimization method based on data enhancement and hybrid enhancement intelligence of claim 3, wherein, In step S3, the offline training specifically comprises the following steps: S3.1, randomly initializing the action network parameters of the T-MAAC offline centralized training model and the target action network parameters , the evaluation network parameters and the target evaluation network parameters ; S3.2, initialize power distribution network environment state , read the input light load training data set, obtain the initial observation of each agent , the number of training steps is set to step, and the number of training rounds is set to episode; S3.3, generating an action of the agent according to the action network of the agent ; S3.4, the agent performs an action After, the reward value of the agent is returned ; S3.5, all agents perform actions After, the power distribution grid environment transitions to the next state , the agent obtains the observation at the next time step ; S3.6, store the sample to the experience pool wherein, , is an initial observation value for the th agent, , is an action value for the th agent, , is a reward value for the th agent, , is an observation value for the th agent at the next time step; S3.7, repeat steps S3.3 to S3.6 until the sample capacity in the experience pool reaches a set threshold ; in subsequent training rounds, after the sample capacity in the experience pool reaches an upper limit , the experience pool is updated in a rolling manner using a FIFO (First In, First Out) strategy ; S3.8, the agent randomly samples a small batch of data from the experience pool with a sampling size of a batch size. S3.9, update the evaluation network according to the following minimum loss formula: ； in, For the first The update function of the evaluation network for each agent; In the experience pool The expected benefit of mid-sampling is the benefit of the current observation action pair. Next Evaluation network of individual agents value With the The objective function value of each agent square of the difference The average value is used to obtain the result. For the first The objective function values ​​for each agent are as follows: ； in, As a discount factor, To observe the action pair at the next moment Next A target evaluation network for individual agents value, To balance the parameters of maximum entropy and reward, No. Target Action Network of Individual Agents In the current observation Select action The probability of; for and The weighted average of the differences represents the first... Target Action Network of Individual Agents The weighted average of all possible actions observed at the next moment; S3.10, update the action network according to the gradient descent formula: ； ； wherein, is the update function for the action network of the th agent ; and is the gradient of the update function for the action network of the th agent ; and is the gradient of the probability of selecting an action given the current observation ; and is a baseline used to reduce the variance of the policy gradient, is the joint action of other agents except the th agent S3.11, update the target action network parameters and the target evaluation network parameters of all agents; ； ； wherein, to update the weights; S3.12, repeat steps S3.8 to S3.11 until the current round of training is completed for all training steps step; S3.13, repeat steps S3.2 to S3.12 until all rounds of training are completed, and a trained T-MAAC offline centralized training model is obtained.

5. The power distribution network reactive power optimization method based on data enhancement and hybrid enhancement intelligence of claim 4, wherein, In step S4, the trained T-MAAC offline centralized training model is used to generate an optimized strategy online, and a motion safety checking module based on a hybrid reinforcement intelligent paradigm is introduced to strictly ensure the safety of the output motion, and the specific process is as follows: S4.1, load the action network of the agent in the T-MAAC offline centralized training model trained in step S3, and set the online execution duration ; S4.2, real-time reading of the power distribution network light load data to obtain the current time Observations of the agent ​​ S4.3, real-time generation of the current moment according to the trained T-MAAC offline centralized training model action value of the agent ; S4.4 Determine the action performed by the intelligent agent. The later Check whether the voltage levels of all nodes where photovoltaic inverters are located in the sub-region where the intelligent agent is located are within the safety constraints. If the constraints are met, execute the action. If the conditions are not met, the action will be corrected based on the voltage exceeding the limit, and then the action will be executed again. If the first Nodes with photovoltaic inverters are set up in the sub-region where each intelligent agent is located. The higher the voltage level, the more the PV action correction formula is shown below: ； wherein, and are the time instants According to the action of the th agent , the reactive power output of the photovoltaic inverter PV in the corresponding sub-area before and after the PV action correction, are the time instants According to the action of the th agent, the voltage level of the node where all photovoltaic inverters in the corresponding sub-area are located, is the upper limit of the node voltage, are the time instants According to the action of the th agent, the voltage reactive sensitivity of the node where the photovoltaic inverter in the corresponding sub-area is located, indicating the sensitivity of the influence of reactive power adjustment on voltage. The node voltage safety constraint condition is as follows: ； wherein Vmin is the lower node voltage limit; S4.5, Determine the action to be performed. The later If the state of charge (SOC) of all distributed energy storage systems in the sub-region corresponding to each agent is within the constraints, and if not, the DESS action is corrected according to the over-limit behavior; if the first agent... Each intelligent agent has a distributed energy storage (DESS) node set up in its sub-region. If the state of charge (SOC) exceeds the upper limit, the correction formula is as follows: ； wherein, is the time According to the action of the charge and discharge power of the distributed energy storage system (DESS) in the corresponding sub-area after the DESS action correction, is the upper limit of the state of charge of the DESS, is the time The state of charge of the distributed energy storage system (DESS) in the sub-area where the is the charging efficiency of the DESS, is the upper limit of the DESS capacity; The DESS safety constraint condition is as follows: ； wherein, and are the upper and lower limits of the DESS state of charge, respectively; The DESS energy constraint condition is as follows: ； ； ； wherein, is the energy level of the DESS at time ; is the discharge efficiency of the DESS, and are upper and lower limits of the DESS capacity; S4.6, repeat steps S4.2 to S4.5 for all agents to obtain an optimized policy including time instants corresponding to actions of all agents ;​ S4.7, repeat steps S4.2 to S4.6 until the entire online execution duration is completed , obtaining an optimized policy for the corresponding online execution duration of all agents within the online execution duration .​

Images