Power distribution network reactive power optimization method based on data enhancement and hybrid enhancement intelligence

Abstract

The invention discloses a power distribution network reactive power optimization method based on data enhancement and hybrid enhancement intelligence. The method comprises the following steps: establishing a diffusion model expansion optical charge training data set; establishing a Markov decision model for active and reactive collaborative optimization of the power distribution network; constructing a T-MAAC off-line centralized training model based on a Markov decision model, and inputting the optical load training data set to carry out off-line training on the T-MAAC off-line centralized training model to obtain a trained T-MAAC off-line centralized training model; and deploying the trained T-MAAC off-line centralized training model on a platform, generating an optimization strategy on line in real time, and executing the optimization strategy to realize reactive power optimization of the power distribution network. The method can provide a real-time voltage control strategy, is adaptive to the uncertainty of the source load, and has good real-time performance and control economy.

Description

Technical Field

[0001] The invention belongs to the field of power system operation optimization control and computer technology, and specifically relates to a distribution network active and reactive coordinated optimization method based on reinforcement learning. Background Art

[0002] After the dual-carbon strategy and the whole-county rooftop photovoltaic policy were proposed, the penetration rate of distributed rooftop photovoltaic (PV) in the distribution network continued to increase, posing a huge challenge to the operation and control of the distribution system. The access of a large number of PVs has changed the power flow distribution of the traditional distribution network, resulting in power flow reversal, which has a great impact on the voltage quality and reliability of the power system. Since PV output is volatile, random and intermittent, as the penetration rate of PV continues to rise, the voltage quality problem deserves special attention among many technical challenges. Voltage quality is one of the important indicators of power quality in the power system. Among many power quality problems, the harm caused by excessive voltage fluctuations is the most widespread: it not only directly affects the performance of electrical equipment, but also brings difficulties to the stable and safe operation of the system. With the rapid development of distributed energy and electric vehicles, the distribution network, as the main support for the access of multiple sources and loads such as distributed energy and electric vehicle charging piles, is also facing huge challenges in economical, safe and stable operation. Reactive power optimization and voltage control are key technologies for optimizing the operation of distribution networks. They optimize the reactive power flow of the entire power grid by adjusting the reactive compensation equipment in the distribution network, and play an important role in optimizing voltage distribution, alleviating voltage over-limit, reducing line losses and improving power quality. In the current context of high proportion of renewable energy access and participation of diversified adjustable equipment, the accuracy and real-time performance of reactive voltage optimization control technology are the key to ensuring the access and consumption of renewable energy and maintaining the economic and safe operation of distribution networks.

[0003] Traditional distribution network voltage control is mainly based on voltage reactive control, which mainly adjusts the voltage of the distribution network by adjusting the tap position of the on-load tap changer (OLTC) and the number of capacitor groups. However, the response speed of OLTC and capacitor groups is slow and the operation and maintenance cost is high. They are not suitable for frequent operation and are difficult to deal with voltage fluctuations caused by large-scale DG power fluctuations. In addition, due to the large resistance and reactance ratio (R / X) of the distribution line, simple reactive control is difficult to solve the distribution network voltage fluctuation problem caused by DG and load active power fluctuations.

[0004] Traditional mathematical optimization methods, intelligent optimization algorithms and stochastic programming methods cannot effectively cope with highly uncertain environments, and have problems such as large computational complexity, easy to fall into local optimality, dependence on accurate parameters and models, and difficulty in achieving online control.

[0005] The power system contains a huge amount of data, which provides data support for the implementation of data-driven artificial intelligence methods. The core of data-driven technology is to analyze massive historical data through machine learning methods, extract high-density value information, and provide decision-making assistance for the planning and operation of distribution networks. Deep reinforcement learning (DRL) combines the perception advantages of deep learning and the decision-making advantages of reinforcement learning. It is a kind of sequential decision-making algorithm and has a high degree of structural fit with the reactive power optimization problem of distribution networks. DRL learns universal strategies with generalization by interacting with complex environments, without adjusting analytical models and re-solving specific decisions.

[0006] However, DRL has a high demand for data sample size and requires a large number of environmental interaction samples. Once the sample size cannot meet the requirements, the application effect will be greatly reduced. At the same time, the neural network of the deep reinforcement learning method is limited in its ability to extract features, and because the control decision is directly given by the neural network, the mechanism is unclear and has a certain black box nature. DRL uses historical data to train the agent offline, and deploys the agent online after training. During operation, the system may encounter extreme situations that were not encountered in the agent training stage. At this time, the feasibility and safety of the solution given by the agent are difficult to guarantee, limiting its application in actual power engineering.

[0007] In the prior art, such as the prior art solutions disclosed in Chinese patent documents CN113363998A, CN113363997A, CN111799808A, CN111884213A, and the paper (Liao Wenlong, Yu Yun, Wang Yusen, et al. Reactive power optimization of distribution network based on graph convolutional network [J]. Power System Technology, 2021, 45(6): 2150-2160.), there are the following disadvantages: 1) Traditional mathematical optimization methods, intelligent optimization algorithms and stochastic programming methods cannot effectively cope with highly uncertain environments, and have problems such as large computational complexity, easy to fall into local optimality, dependence on accurate parameters and models, and difficulty in achieving online control; 2) Supervised learning-based methods require a certain number of optimization methods to find the optimal decision as a "label" during the offline training phase. This process usually takes a long time and has high computational costs. 3) Reinforcement learning methods such as DDPG and DQN regard the control device as a single agent, which is difficult to expand to larger systems. As the number of control devices increases, the dimension of the action space explodes, making it difficult to effectively provide a better strategy. MADDPG, a typical representative of multi-agent reinforcement learning algorithms, has problems such as overestimation of q-values ​​and training instability. Although algorithms represented by multi-agent twin delayed deepdeterministic policy gradient (MATD3) have suppressed the over-estimation problem to a certain extent, when the number of agents increases, the input information of the network will also increase accordingly. When expanded to large-scale systems with more agents, the algorithm is inefficient and performs poorly.

[0008] 4) Reinforcement learning as a data-driven method has a large sample demand, high data collection cost and low sampling efficiency. The coordinated optimization of active and reactive power in distribution networks usually faces complex operating scenarios, such as load fluctuations and drastic changes in photovoltaic output. The quality of training data sets in existing reinforcement learning methods is not high, and the data distribution is uneven, making it difficult to cover various source-load output conditions. Although historical data can provide some data sources, the structure and scale of the power system are constantly changing, resulting in a small amount of effective data and low sample efficiency, making the reinforcement learning model less adaptable in a changing environment. Traditional data generation methods based on generative adversarial networks are prone to mode collapse during training, leading to training failure.

[0009] 5) The representation ability of the neural network architecture of existing reinforcement learning methods is limited. When faced with large-scale distribution networks, it is usually difficult to effectively optimize due to the huge state space and long-term dependencies. In addition, distribution networks often face changes in environment and load in actual applications, and traditional reinforcement learning methods often lack generalization capabilities when dealing with diverse and rapidly changing environments.

[0010] 6) Existing reinforcement learning methods have difficulty in handling complex agent action constraints. When formulating reward functions, they all guide the agent to find a feasible solution by adding a penalty term, which cannot guarantee the feasibility of the solution. At the same time, since the control decision is directly given by the neural network and the mechanism is unclear, existing reinforcement learning methods generally adopt the "offline training and online execution" framework. After the offline training is completed, the agent is deployed online. During the operation of the system, extreme situations that were not encountered in the agent training stage may occur. At this time, the security of the solution given by the agent is difficult to guarantee. Summary of the invention

[0011] In view of the above problems, the present invention proposes a method for active and reactive collaborative optimization of distribution network based on data enhancement and hybrid enhanced intelligence. First, based on the generative diffusion model, the future scheduling scenario data is expanded, and the historical operation data is aggregated to form a high-quality training data set; then, according to the active collaborative optimization model of the distribution network, a partially observable Markov decision process (POMDP) ​​is designed, and the model is solved by the Transformer-based Multi-agent actor-attention-critic (T-MAAC) algorithm embedded in Transformer; finally, using the trained model, an action safety verification module is introduced based on the hybrid enhanced intelligence paradigm in the online execution stage to strictly ensure the safety of the output action. The diffusion model improves the quality and diversity of the data, and the Transformer model helps to improve the DRL representation ability, greatly improving the stability and learning efficiency of the algorithm. Through the hybrid intelligent framework of offline training-online execution, the active and reactive collaborative optimization control of photovoltaic inverters and distributed energy storage in the distribution network is realized. The proposed method can provide a real-time voltage control strategy, adapt to source-load uncertainty, and has good real-time performance and control economy.

[0012] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0013] A method for reactive power optimization of a distribution network based on data enhancement and hybrid enhanced intelligence comprises the following steps: S1, establish a diffusion model to expand the light charge training data set; S2. Establish a Markov decision model for coordinated optimization of active and reactive power in distribution networks; S3. Construct a T-MAAC offline centralized training model based on the Markov decision model, input the optical load training data set to perform offline training on the T-MAAC offline centralized training model, and obtain a trained T-MAAC offline centralized training model; S4. Utilize the trained T-MAAC offline centralized training model to generate optimization strategies online in real time, and execute the optimization strategies to achieve reactive power optimization of the distribution network.

[0014] Furthermore, step S1 includes the following steps: S1.1. Data collection and preprocessing: Collect distribution network operation data over a period of time, including photovoltaic active output, load active power and load reactive power; standardize the collected data to ensure that the data is trained at the same scale; S1.2, forward diffusion process: Noise is gradually added to the prepared data to simulate different environments or system disturbances; the noise addition process of each step can be represented by multiple time steps to gradually blur the data; a group of data is randomly sampled from the pre-processed photovoltaic active output data samples, load active power data samples, and load reactive power data samples as the initial samples , , , are the initial values ​​of photovoltaic active output, load active power and load reactive power, respectively, and the diffusion step number is set and noise parameters , at each step Gradually add noise to the sample , , get samples affected by different noises , , and Respectively The PV active power, load active power and load reactive power of the same step; The noise adding process is as follows: ; in, For the After the diffusion step, hour , is the initial sample; For the Standard Gaussian noise in step diffusion, resampled at each step to ensure randomness and diversity at each step; For the The noise parameter of the step diffusion, with the time step Linear increase, the growth rate is k ; S1.3, Reverse generation process: Through the reverse denoising process, these noisy samples are gradually restored to more realistic and diverse samples; the denoising neural network model is trained, and the input of the denoising neural network model is the noisy data samples , that is, the samples generated by the forward diffusion process in step S1.2, and the output is the predicted noise The training goal is to minimize the denoising error, that is, to minimize the difference between the predicted noise and the actual noise of the model at each time step; during the training process, the network predicts the added noise as much as possible by minimizing this goal, and learns how to restore the original sample from the noisy sample, so that the denoising neural network model can accurately recover the original sample from the noisy data at each step. Restore clear photovoltaic output and load power data, and optimize the objective function of the training process as follows: ; in, In the forward diffusion process Standard Gaussian noise in step diffusion, The denoising neural network model predicts The noise part added in step 1, Represents different initial samples and noise The expectation of a random sample of and The square of the difference Euclidean distance (L2 norm) is averaged to fully consider the diversity of samples; After the training is completed, the denoising neural network model is trained to Sample after diffusion Step by step, denoising is performed to generate diversified photovoltaic output and load power data. The formula is as follows: ; Among them, Sample after diffusion As the initial noise sample ; is the trained denoising neural network model; S1.4. Generate training data set: The diversified photovoltaic output and load power data generated in step S1.3 are added to the original data set, thereby constructing a training data set including a diversified combination of photovoltaic active output, load active power and load reactive power data.

[0015] Furthermore, in step S2, a Markov decision model for active and reactive coordinated optimization of the distribution network is established, and coordinated optimization is achieved by optimizing the reactive output of distributed photovoltaics in the distribution network and the active power of distributed energy storage, as follows: The distributed control framework is adopted. The distribution network includes multiple sub-areas, each of which includes multiple nodes. Each node is the location where distributed photovoltaic or distributed energy storage is set in the sub-area. The local controller of each sub-area is the intelligent agent. The distributed photovoltaic and distributed energy storage in the sub-area are controlled by the local controller. The distribution network is a reinforcement learning environment. The Markov decision model (distribution system) POMDP is defined as ,in is the number of agents, is the discount factor; The node feature matrix and the adjacency matrix The state set is composed of, where the characteristic matrix Represents the characteristic information of all nodes, , , , , , and Represents the active load, reactive load, photovoltaic active output, photovoltaic reactive output, state of charge of distributed energy storage and node voltage amplitude of the node; node set Represents all nodes of the distribution network topology, Indicates nodes, is the total number of nodes in the distribution network, include The voltage amplitude of each node; the adjacency matrix Characterize the topological connection relationship, adjacency matrix have OK Column, adjacency matrix The Line Elements of a column Indicates Nodes With Nodes The connection relationship, when the Nodes With Nodes When there is a connection relationship between them, the adjacency matrix Middle Line Elements of a column , otherwise 0; Observation Set Represents the set of observation information of all agents. Each agent can only observe the state quantity of the sub-region where the agent is located. The observation set includes the observation quantities of all agents. The observation quantity of each agent includes the observation quantities of all nodes in the sub-region where the agent is located. Action Set Represents the action set of all agents, including the reactive output of the photovoltaic inverters set by all nodes in the sub-area where each agent is located and the charging and discharging power of the distributed energy storage; TA state transition function is a function that represents the probability of an intelligent agent transferring from a current state to a next state after performing an action. The present invention belongs to a reinforcement learning method that is not based on a model, does not require a certain state transition probability, and random state transitions can effectively cope with multiple uncertainties such as source and load in the distribution network, which is in line with the problem of active and reactive collaborative optimization of the distribution network. is a reward function that represents the reward obtained by the agent after performing an action; according to the active and reactive power collaborative optimization model of the distribution network, the reward function as follows: ; ; in, and The node voltage offsets are and network loss Additional penalty coefficient of To take a positive function, For the Nodes The voltage amplitude, and are the safety upper and lower limits of the node voltage respectively.

[0016] Further, in step S2, at each time step In the example, each agent makes the best action based on the observation through the action network, and then according to the reward function Get instant rewards; the goal of each agent is to learn the optimal strategy to maximize the cumulative reward; after all agents have performed the optimal actions, the POMDP (distribution system) will move to the next state.

[0017] Furthermore, in step S3, in the T-MAAC offline centralized training model, the transformer module is embedded in the action network architecture of the agent, and the attention weight mechanism is embedded in the evaluation network of the agent. The action network and the evaluation network work together, and each agent is in a cooperative relationship. A homogeneous agent parameter sharing mechanism is adopted, that is, the action network and evaluation network of each agent are the same, so as to improve the performance and efficiency of the algorithm.

[0018] Furthermore, in the transformer module, the original state features are mapped to the input space to generate the embedded representation of each agent: ; ; in, For the Embedding information of each agent; For the The first sub-region where the agent is located The number of observations per node Projection, projection layer is a linear transformation that converts the original observation into an embedded vector; For the The number of nodes in the sub-region where the agent is located, ; is the original input matrix; According to the original input matrix , mapping the embedded information into a space suitable for calculating attention: ; in, is the index of the number of transformer layers in the transformer module; , , Respectively represent The query matrix, key matrix, and value matrix of the transformer layer; , , Represent the learnable weight matrices for calculating the Query matrix, Key matrix, and Value matrix respectively; For the The input matrix of the transformer layer; The masked self-attention mechanism is used to establish the correlation between nodes in the region; the masked self-attention calculation formula is as follows: ;

[0019] in, For the The masked self-attention calculation value of the transformer layer; Represents the calculation function of the self-attention mechanism with a mask matrix; is the activation function; is the transpose of the Key matrix, is the scaling factor, whose value is the dimension of the Query matrix; is a mask matrix, where the elements are either 0 or 1. is the adjacency matrix , ensure that attention weights are calculated only between topologically connected nodes, thereby avoiding interference from irrelevant nodes, so as to better adapt to the physical characteristics of the distribution network and improve computing performance and efficiency; Layer Normalization and Linear transformation are applied to the output of each transformer layer to update the embedded features and iteratively build high-level embedded features, as follows: ; in, For the The embedded features of the transformer layer output; After multi-layer mask self-attention calculation, an aggregation embedding module is used to aggregate the embedding features of all nodes: ; ; ; ; in, , , are the Query, Key, and Value matrices of the aggregation embedding module, respectively. , , are the learnable parameters of the Query, Key, and Value matrices of the aggregation embedding module respectively; The weighted representation of the global features obtained using the standard attention calculation mechanism Attention is the global feature; is the global embedding feature obtained by further processing through residual connection and layer normalization; is the local neighborhood feature extracted by MaskAttention, i.e. The embedded features output by the transformer layer fuse the local neighborhood features with the global features. The local neighborhood features will not be lost due to the global weighting process, which improves the stability and representation ability of the T-MAAC offline centralized training model. Use the SelectEmbedding module to select all node embeddings that have distributed photovoltaics or distributed energy storage set up, and obtain global information from the perspective of the agent corresponding to the sub-area where the node is located: ; in, Indicates that from The global information extracted from the perspective of each agent, For the The first sub-region where the agent is located nodes; Use the gated recurrent unit (GRU) module to Mapping to The action value of an agent : .

[0020] Furthermore, the agent evaluation network The value is calculated as follows: ; in, For the The current observation action pair of the agent Next value, To evaluate the network parameters, To evaluate the multi-layer perceptron in the network, To evaluate the single-layer perceptron in the network, For the The observations of an agent, For other agents to The weighted sum of the values ​​of the agents is as follows: ; in, For the The agent is The attention weight of each agent; For the The embedded functions of an agent, is the activation function, is a shared matrix; ; in, , , by comparing their similarity to calculate the The agent is The attention weights of each agent.

[0021] Furthermore, in step S3, the operating environment of the actual distribution network is simulated through power flow calculation; an experience pool is constructed by adopting the experience replay mechanism to store samples obtained by the interaction between the intelligent agent and the environment to improve the training stability and learning efficiency, the action network is copied to construct the target action network, and the evaluation network is copied to construct the target evaluation network to further improve the stability of the algorithm.

[0022] Furthermore, in step S3, offline training specifically includes the following steps: S3.1. Randomly initialize the action network parameters of the T-MAAC offline centralized training model and target action network parameters , evaluate network parameters and target evaluation network parameters ; S3.2. Initialize the distribution network environment status , read the input light charge training data set and obtain the initial observation of each agent , the number of training steps is set to step, and the number of training rounds is set to episode.

[0023] S3.3. Generate the agent's actions based on the agent's action network ; S3.4. Agent Execution Actions After that, the reward value of the agent is returned ; S3.5. All agents perform actions After that, the distribution network environment transfers to the next state , the agent obtains the observation value at the next moment ; S3.6. Sample Store in experience pool Among them, , For the The initial observations of the agents, , For the The action value of an agent, , For the The reward value of an agent, , For the The observation value of each agent at the next moment; S3.7, repeat steps S3.3 to S3.6 until the experience pool The sample size in reaches the set threshold No need to wait until the experience pool Fill it up, speed up the training process and improve efficiency; in subsequent training rounds, The sample size in has reached the upper limit After that, the FIFO (First In, First Out) strategy is used to roll over and update the experience pool. ; S3.8, Agent in the Experience Pool Randomly extract small batches of data from the batch size. S3.9. Update the evaluation network according to the following loss minimization formula: ; in, For the The update function of the evaluation network of each agent; For the experience pool The expected return of sampling in is the current observation action pair Next The evaluation network of the agent value With The objective function value of the agent The square of the difference Taking the average value, we get; For the The objective function value of each agent is as follows: ; in, is the discount factor, Observe the action pair for the next moment Next The goal evaluation network of the agent value, To balance the parameters of maximum entropy and reward, No. Agent-based target action network In the current observation Next select action The probability of for and The weighted average of the differences between Agent-based target action network The weighted average of all possible actions observed at the next moment; S3.10. Update the action network according to the gradient descent formula: ; ; in, For the Action network of an agent The gradient of the update function of ; is the policy gradient part, indicating the Action network of an agent In the current observation Next select action The gradient of the probability of is the baseline, used to reduce the variance of the policy gradient, Indicates that except for The joint actions of other agents other than the agent; S3.11. Update the target action network parameters and target evaluation network parameters of all agents: ; ; in, To update the weights; S3.12, repeat steps S3.8 to S3.11 until all training steps step are completed in this round of training; S3.13. Repeat steps S3.2 to S3.12 until all rounds of training are completed to obtain a trained T-MAAC offline centralized training model.

[0024] Furthermore, in step S4, the trained T-MAAC offline centralized training model is used to intelligently generate an optimization strategy online, and the safety of the output action is strictly guaranteed by introducing the action safety verification module of the hybrid enhanced intelligence paradigm. The specific process is as follows: S4.1. Load the action network of the agent in the T-MAAC offline centralized training model trained in step S3 and set the simulation duration. ; S4.2. Read the light load data of the distribution network in real time to obtain the observation value of the intelligent agent ; S4.3. Generate the current moment in real time based on the trained T-MAAC offline centralized training model The action value of the agent ; S4.4. Determine the action performed by the agent After Whether the voltage levels of all nodes where photovoltaic inverters are set in the sub-area where the agent is located are within the safety constraint range, if the constraint is met, the action is executed If it is not satisfied, the action will be corrected according to the voltage exceeding the limit, and then the action will be executed ; If Set the node of photovoltaic inverter in the sub-area where the agent is located The voltage level is higher than the upper limit, and the PV action correction formula is as follows: ; in, and Separately for the moment According to Actions of an agent The reactive power output of the photovoltaic inverter PV in the corresponding sub-area before and after the PV action correction is performed. For the moment No. The voltage levels of all nodes where the photovoltaic inverters are located in the sub-area corresponding to the intelligent agent, is the upper limit of node voltage, For the moment No. The voltage reactive sensitivity of the node where the photovoltaic inverter is located in the sub-area corresponding to each intelligent agent indicates the sensitivity of reactive power regulation to voltage. The node voltage safety constraints are as follows: ; in, is the lower limit of node voltage; S4.5. Determine the execution action After Whether the state of charge (SOC) of all distributed energy storage systems (DESS) in the sub-area corresponding to the first agent is within the constraint range, if not, the DESS action is modified according to the limit exceeding; if the first agent The nodes of distributed energy storage DESS are set in the sub-area where each intelligent agent is located. The state of charge SOC exceeds the upper limit, and the correction formula is as follows: ; in, For the moment According to Actions of an agent The charging and discharging power of the distributed energy storage DESS in the corresponding sub-area after the DESS action correction is performed. is the upper limit of DESS state of charge, For the moment No. The charge state of the distributed energy storage DESS in the sub-area where the intelligent agent is located, is the charging efficiency of DESS, is the upper limit of DESS capacity; The DESS security constraints are as follows: ; in, and are the upper and lower limits of the DESS state of charge, respectively; The DESS energy constraints are as follows: ; ; ; in, For DESS at the moment energy level; is the discharge efficiency of DESS, and are the upper and lower limits of DESS capacity; S4.6. Repeat steps S4.2 to S4.5 for all agents to obtain the time The corresponding moments of all agents’ actions Optimization strategy ; S4.7, repeat steps S4.2 to S4.6 until the entire online execution time is completed , including online execution time The corresponding online execution time of all the actions of the agents in Optimization strategy .

[0025] Compared with the prior art, the advantages of the present invention are: In order to improve the data quality of the training set of the reinforcement learning method, the present invention proposes a data enhancement method based on the generative diffusion model. The diffusion model is used to simulate the process of adding noise and denoising to generate new samples, covering a wider range of photovoltaic output and load fluctuation scenarios, increasing the diversity of training samples, thereby improving the generalization and robustness of the algorithm.

[0026] In order to improve the generalization of MADRL for differentiated environments and the scalability for large systems, the present invention integrates the advantages of deep learning and reinforcement learning, and proposes a photovoltaic inverter intelligent collaborative voltage control method based on T-MAAC. Among them, the Transformer model is used to characterize the state information in the distribution network environment, process the long-term dependencies of sequence data, and use its self-attention mechanism to better capture the relationship and long-term dependencies between different nodes in the distribution network, thereby improving the model's understanding of multi-time-step states; an embedding representation containing global information is generated by an aggregate embedding module, and then the embedding module is selected to focus on the embedding information of a specific node, thereby improving the training efficiency and feature extraction capabilities through selective learning focus. The attention mechanism of MAAC is concentrated on the evaluation network, and the information fusion capability is limited, while the Transformer can enhance the action network and improve the efficiency of policy optimization. MAAC integrated with the Transformer can handle more complex tasks and greatly improve the performance of multi-agent systems.

[0027] The technology of the present invention avoids the modeling of the underlying physical model through model-free algorithm processing, and adopts the framework of "offline centralized training, online distributed execution". In the offline learning stage, the optimal strategy is mined through a large number of samples and stored in the strategy network. When making online decisions, the optimal decision is quickly given according to the real-time status of the system, thereby eliminating the process of online optimization, which makes it have better real-time and computing performance; and in the online execution stage, an action safety verification mechanism is introduced based on the human-machine hybrid enhanced intelligence paradigm. When it is assessed that the original action of the intelligent body has an out-of-limit risk, it intervenes to prevent it, and strictly ensures that the output action of the intelligent body is within the safety constraint range.

[0028] The method of the invention can effectively reduce system network loss and voltage deviation, improve the safety and stability of the system, and support the low-carbon economy, safe and efficient operation of the system. A full-chain innovative reinforcement learning framework is designed, from pre-training training set data enhancement, to the T-MAAC method in the training phase, to the hybrid enhanced intelligent method in the execution phase, to comprehensively improve the practicality, generalization and safety of reinforcement learning in the actual distribution network active and reactive collaborative optimization scenario from three stages. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flow chart of the steps of expanding the optical charge training data set in an embodiment of the present invention.

[0030] Figure 2 Schematic diagram of the structure of the T-MAAC offline centralized training model in an embodiment of the present invention.

[0031] Figure 3 Flow chart of offline training and online execution steps in an embodiment of the present invention.

[0032] Figure 4 The present invention is a flowchart of a method for reactive power optimization of a distribution network based on data enhancement and hybrid enhanced intelligence in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation of the present invention is described in detail below with reference to the accompanying drawings and examples.

[0034] Example: A method for reactive power optimization of a distribution network based on data enhancement and hybrid enhanced intelligence comprises the following steps: S1. Establish a diffusion model to expand the light charge training data set, such as Figure 1 As shown, the following steps are included: S1.1. Data collection and preprocessing: Collect distribution network operation data over a period of time, including photovoltaic active output, load active power and load reactive power; standardize the collected data to ensure that the data is trained at the same scale; S1.2, forward diffusion process: Noise is gradually added to the prepared data to simulate different environments or system disturbances; the noise addition process of each step can be represented by multiple time steps to gradually blur the data; a group of data is randomly sampled from the pre-processed photovoltaic active output data samples, load active power data samples, and load reactive power data samples as the initial samples , , , are the initial values ​​of photovoltaic active output, load active power and load reactive power, respectively, and the diffusion step number is set and noise parameters , at each step Gradually add noise to the sample , , get samples affected by different noises , , and Respectively The PV active power, load active power and load reactive power of the same step; The noise adding process is as follows: ; in, For the After the diffusion step, hour , is the initial sample; For the Standard Gaussian noise in step diffusion, resampled at each step to ensure randomness and diversity at each step; For the The noise parameter of the step diffusion, with the time step Linear increase, the growth rate is k ; S1.3, Reverse generation process: Through the reverse denoising process, these noisy samples are gradually restored to more realistic and diverse samples; the denoising neural network model is trained, and the input of the denoising neural network model is the noisy data samples , that is, the samples generated by the forward diffusion process in step S1.2, and the output is the predicted noise The training goal is to minimize the denoising error, that is, to minimize the difference between the predicted noise and the actual noise of the model at each time step; during the training process, the network predicts the added noise as much as possible by minimizing this goal, and learns how to restore the original sample from the noisy sample, so that the denoising neural network model can accurately recover the original sample from the noisy data at each step. Restore clear photovoltaic output and load power data, and optimize the objective function of the training process as follows: ; in, In the forward diffusion process Standard Gaussian noise in step diffusion, The denoising neural network model predicts The noise part added in step 1, Represents different initial samples and noise The expectation of a random sample of and The square of the difference Euclidean distance (L2 norm) is averaged to fully consider the diversity of samples; After the training is completed, the denoising neural network model is trained to Sample after diffusion Step by step, denoising is performed to generate diversified photovoltaic output and load power data. The formula is as follows: ; Among them, Sample after diffusion As the initial noise sample ; is the trained denoising neural network model; S1.4. Generate training data set: The diversified photovoltaic output and load power data generated in step S1.3 are added to the original data set, thereby constructing a training data set including a diversified combination of photovoltaic active output, load active power and load reactive power data.

[0035] S2. Establish a Markov decision model for the coordinated optimization of active and reactive power in the distribution network, and achieve coordinated optimization by optimizing the reactive power output of distributed photovoltaics in the distribution network and the active power of distributed energy storage. The details are as follows: The distributed control framework is adopted. The distribution network includes multiple sub-areas, each of which includes multiple nodes. Each node is the location where distributed photovoltaic or distributed energy storage is set in the sub-area. The local controller of each sub-area is the intelligent agent. The distributed photovoltaic and distributed energy storage in the sub-area are controlled by the local controller. The distribution network is a reinforcement learning environment. The Markov decision model (distribution system) POMDP is defined as ,in is the number of agents, is the discount factor; The node feature matrix and the adjacency matrix The state set is composed of, where the characteristic matrix Represents the characteristic information of all nodes, , , , , , and Represents the active load, reactive load, photovoltaic active output, photovoltaic reactive output, state of charge of distributed energy storage and node voltage amplitude of the node; node set Represents all nodes of the distribution network topology, Indicates nodes, is the total number of nodes in the distribution network, include The voltage amplitude of each node; the adjacency matrix Characterize the topological connection relationship, adjacency matrix have OK Column, adjacency matrix The Line Elements of a column Indicates Nodes With Nodes The connection relationship, when the Nodes With Nodes When there is a connection relationship between them, the adjacency matrix Middle Line Elements of a column , otherwise 0; Observation Set Represents the set of observation information of all agents. Each agent can only observe the state quantity of the sub-region where the agent is located. The observation set includes the observation quantities of all agents. The observation quantity of each agent includes the observation quantities of all nodes in the sub-region where the agent is located. Action Set Represents the action set of all agents, including the reactive output of the photovoltaic inverters set by all nodes in the sub-area where each agent is located and the charging and discharging power of the distributed energy storage; T A state transition function is a function that represents the probability of an intelligent agent transferring from a current state to a next state after performing an action. The present invention belongs to a reinforcement learning method that is not based on a model, does not require a certain state transition probability, and random state transitions can effectively cope with multiple uncertainties such as source and load in the distribution network, which is in line with the problem of active and reactive collaborative optimization of the distribution network. is a reward function that represents the reward obtained by the agent after performing an action; according to the active and reactive power collaborative optimization model of the distribution network, the reward function as follows: ; ; in, and The node voltage offsets are and network loss In one embodiment, the network loss Directly give the results by calling power system flow calculation packages such as pandapower or matpower; To take a positive function, For the Nodes The voltage amplitude, and are the safety upper and lower limits of the node voltage respectively.

[0036] At each time step In the example, each agent makes the best action based on the observation through the action network, and then according to the reward function Get instant rewards; the goal of each agent is to learn the optimal strategy to maximize the cumulative reward; after all agents have performed the optimal actions, the POMDP (distribution system) will move to the next state.

[0037] S3. Construct a T-MAAC offline centralized training model based on the Markov decision model, input the optical load training data set to perform offline training on the T-MAAC offline centralized training model, and obtain a trained T-MAAC offline centralized training model; In the T-MAAC offline centralized training model, Figure 2 As shown in the figure, the transformer module is embedded in the action network architecture of the agent, and the attention weight mechanism is embedded in the evaluation network of the agent. The action network and the evaluation network work together, and each agent is in a cooperative relationship. A homogeneous agent parameter sharing mechanism is adopted, that is, the action network and evaluation network of each agent are the same, so as to improve the performance and efficiency of the algorithm.

[0038] In one embodiment, an agent action network and an evaluation network are constructed according to the prior art (LOWE R, WU Yi, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C] / / Proceedings of the 31st International Conference on Neural Information Processing Systems. California, USA: Curran Associates Inc., 2017: 6382-6393.).

[0039] In the transformer module, the original state features are mapped to the input space to generate the embedded representation of each agent: ; ; in, For the Embedding information of each agent; For the The first sub-region where the agent is located The number of observations per node Projection, projection layer is a linear transformation that converts the original observation into an embedded vector; For the The number of nodes in the sub-region where the agent is located, ; is the original input matrix; According to the original input matrix , mapping the embedded information into a space suitable for calculating attention: ; in, is the index of the number of transformer layers in the transformer module; , , Respectively represent The query matrix, key matrix, and value matrix of the transformer layer; , , Represent the learnable weight matrices for calculating the Query matrix, Key matrix, and Value matrix respectively; For the The input matrix of the transformer layer; The masked self-attention mechanism is used to establish the correlation between nodes in the region; the masked self-attention calculation formula is as follows: ;

[0040] in, For the The masked self-attention calculation value of the transformer layer; Represents the calculation function of the self-attention mechanism with a mask matrix; is the activation function; is the transpose of the Key matrix, is the scaling factor, whose value is the dimension of the Query matrix; is a mask matrix, where the elements are either 0 or 1. is the adjacency matrix , ensure that attention weights are calculated only between topologically connected nodes, thereby avoiding interference from irrelevant nodes, so as to better adapt to the physical characteristics of the distribution network and improve computing performance and efficiency; Layer Normalization and Linear transformation are applied to the output of each transformer layer to update the embedded features and iteratively build high-level embedded features, as follows: ; in, For the The embedded features of the transformer layer output; After multi-layer mask self-attention calculation, an aggregation embedding module is used to aggregate the embedding features of all nodes: ; ; ; ; in, , , are the Query, Key, and Value matrices of the aggregation embedding module, respectively. , , are the learnable parameters of the Query, Key, and Value matrices of the aggregation embedding module respectively; The weighted representation of the global features obtained using the standard attention calculation mechanism Attention is the global feature; is the global embedding feature obtained by further processing through residual connection and layer normalization; is the local neighborhood feature extracted by MaskAttention, i.e. The embedded features output by the transformer layer fuse the local neighborhood features with the global features. The local neighborhood features will not be lost due to the global weighting process, which improves the stability and representation ability of the T-MAAC offline centralized training model. Use the SelectEmbedding module to select all node embeddings that have distributed photovoltaics or distributed energy storage set up, and obtain global information from the perspective of the agent corresponding to the sub-area where the node is located: ; in, Indicates that from The global information extracted from the perspective of each agent, For the The first sub-region where the agent is located nodes; Use the gated recurrent unit (GRU) module to Mapping to The action value of an agent : .

[0041] Agent Evaluation Network The value is calculated as follows: ; in, For the The current observation action pair of the agent Next value, To evaluate the network parameters, To evaluate the multi-layer perceptron in the network, To evaluate the single-layer perceptron in the network, For the The observations of an agent, For other agents to The weighted sum of the values ​​of the agents is as follows: ; in, For the The agent is The attention weight of each agent; For the The embedded functions of an agent, is the activation function, is a shared matrix; ; in, , , by comparing their similarity to calculate the The agent is The attention weights of each agent.

[0042] The operating environment of the actual distribution network is simulated through power flow calculation. The experience pool is constructed by adopting the experience replay mechanism to store samples obtained from the interaction between the intelligent agent and the environment to improve the training stability and learning efficiency. The action network is copied to construct the target action network, and the evaluation network is copied to construct the target evaluation network to further improve the stability of the algorithm.

[0043] like Figure 3 As shown, offline training specifically includes the following steps: S3.1. Randomly initialize the action network parameters of the T-MAAC offline centralized training model and target action network parameters , evaluate network parameters and target evaluation network parameters ; S3.2. Initialize the distribution network environment status , read the input light charge training data set and obtain the initial observation of each agent , the number of training steps is set to step, and the number of training rounds is set to episode.

[0044] S3.3. Generate the agent's actions based on the agent's action network ; S3.4. Agent Execution Actions After that, the reward value of the agent is returned ; S3.5. All agents perform actions After that, the distribution network environment transfers to the next state , the agent obtains the observation value at the next moment ; S3.6. Sample Store in experience pool Among them, , For the The initial observations of the agents, , For the The action value of an agent, , For the The reward value of an agent, , For the The observation value of each agent at the next moment; S3.7, repeat steps S3.3 to S3.6 until the experience pool The sample size in reaches the set threshold ; In subsequent training rounds, in the experience pool The sample size in has reached the upper limit After that, the FIFO (First In, First Out) strategy is used to roll over and update the experience pool. ; S3.8, Agent in the Experience Pool Randomly extract small batches of data from the batch size. S3.9. Update the evaluation network according to the following loss minimization formula: ; in, For the The update function of the evaluation network of each agent; For the experience pool The expected return of sampling in is the current observation action pair Next The evaluation network of the agent value With The objective function value of the agent The square of the difference Taking the average value, we get; For the The objective function value of each agent is as follows: ; in, is the discount factor. In one embodiment, Take 0.99, Observe the action pair for the next moment Next The goal evaluation network of the agent value, To balance the parameters of maximum entropy and reward, in one embodiment, Take 0.01, No. Agent-based target action network In the current observation Next select action The probability of for and The weighted average of the differences between Agent-based target action network The weighted average of all possible actions observed at the next moment; S3.10. Update the action network according to the gradient descent formula: ; ; in, For the Action network of an agent The gradient of the update function of ; is the policy gradient part, indicating the Action network of an agent In the current observation Next select action The gradient of the probability of is the baseline, used to reduce the variance of the policy gradient, Indicates that except for The joint actions of other agents other than the agent; S3.11. Update the target action network parameters and target evaluation network parameters of all agents: ; ; in, To update the weights, in one embodiment, Take 0.995; S3.12, repeat steps S3.8 to S3.11 until all training steps step are completed in this round of training; S3.13. Repeat steps S3.2 to S3.12 until all rounds of training are completed to obtain a trained T-MAAC offline centralized training model.

[0045] S4, deploy the trained T-MAAC offline centralized training model to the computer system platform of the actual distribution network, generate the optimization strategy online in real time, and strictly ensure the safety of the output action by introducing the action safety verification module of the hybrid enhanced intelligent paradigm, and execute the optimization strategy to achieve reactive power optimization of the distribution network, such as Figure 3 As shown, the specific process is as follows: S4.1. Load the action network of the agent in the T-MAAC offline centralized training model trained in step S3 and set the simulation duration. ; S4.2. In one embodiment, the computer system platform of the actual power distribution network connected to the system reads the light load data in real time to obtain the observation value of the intelligent agent. ; S4.3. Generate the current moment in real time based on the trained T-MAAC offline centralized training model The action value of the agent ; S4.4. Determine the action performed by the agent After Whether the voltage levels of all nodes where photovoltaic inverters are set in the sub-area where the agent is located are within the safety constraint range, if the constraint is met, the action is executed If it is not satisfied, the action will be corrected according to the voltage exceeding the limit, and then the action will be executed ; If Set the node of photovoltaic inverter in the sub-area where the agent is located The voltage level is higher than the upper limit, and the PV action correction formula is as follows: ; in, and Separately for the moment According to Actions of an agent The reactive power output of the photovoltaic inverter PV in the corresponding sub-area before and after the PV action correction is performed. For the moment No. The voltage levels of all nodes where the photovoltaic inverters are located in the sub-area corresponding to the intelligent agent, is the upper limit of node voltage, For the moment No. The voltage reactive sensitivity of the node where the photovoltaic inverter is located in the sub-area corresponding to each intelligent agent indicates the sensitivity of reactive power regulation to voltage. The node voltage safety constraints are as follows: ; in, is the lower limit of node voltage; S4.5. Determine the execution action After Whether the state of charge (SOC) of all distributed energy storage systems (DESS) in the sub-area corresponding to the first agent is within the constraint range, if not, the DESS action is modified according to the limit exceeding; if the first agent The nodes of distributed energy storage DESS are set in the sub-area where each intelligent agent is located. The state of charge SOC exceeds the upper limit, and the correction formula is as follows: ; in, For the moment According to Actions of an agent The charging and discharging power of the distributed energy storage DESS in the corresponding sub-area after the DESS action correction is performed. is the upper limit of DESS state of charge, For the moment No. The charge state of the distributed energy storage DESS in the sub-area where the intelligent agent is located, is the charging efficiency of DESS, is the upper limit of DESS capacity; The DESS security constraints are as follows: ; in, and are the upper and lower limits of the DESS state of charge, respectively; The DESS energy constraints are as follows: ; ; ; in, For DESS at the moment energy level; is the discharge efficiency of DESS, and are the upper and lower limits of DESS capacity; S4.6. Repeat steps S4.2 to S4.5 for all agents to obtain the time The corresponding moments of all agents’ actions Optimization strategy ; S4.7, repeat steps S4.2 to S4.6 until the entire online execution time is completed , including online execution time The corresponding online execution time of all the actions of the agents in Optimization strategy .

[0046] The preferred embodiments of the present application disclosed above are only used to help understand the present invention and its core ideas. For those skilled in the art, according to the ideas of the present invention, there will be changes in specific application scenarios and implementation operations, and this specification should not be interpreted as limiting the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence, characterized in that: The steps include: S1, establish a diffusion model to expand the light charge training data set; S2. Establish a Markov decision model for coordinated optimization of active and reactive power in distribution networks; S3. Construct a T-MAAC offline centralized training model based on the Markov decision model, input the optical load training data set to perform offline training on the T-MAAC offline centralized training model, and obtain a trained T-MAAC offline centralized training model; S4. Utilize the trained T-MAAC offline centralized training model to generate optimization strategies online in real time, and execute the optimization strategies to achieve reactive power optimization of the distribution network.

2. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 1, characterized in that: Step S1 includes the following steps: S1.

1. Data collection and preprocessing: Collect distribution network operation data over a period of time, including photovoltaic active output, load active power and load reactive power; standardize the collected data; S1.2, forward diffusion process: A group of data is randomly sampled from the preprocessed photovoltaic active output data samples, load active power data samples, and load reactive power data samples as initial samples. , , , are the initial values ​​of PV active output, load active output and load reactive power, respectively, and the number of diffusion steps is set and noise parameters , at each step Gradually add noise to the sample , , get samples affected by different noises , , and Respectively The PV active power, load active power and load reactive power of the same step; S1.3, Reverse generation process: Train a denoising neural network model whose input is a noisy data sample , that is, the samples generated by the forward diffusion process in step S1.2, and the output is the predicted noise ,The training goal is to minimize the denoising error, that is, to minimize the difference between the predicted noise and the true noise of the model at each time step; S1.

4. Generate training data set: The diversified photovoltaic output and load power data generated in step S1.3 are added to the original data set, thereby constructing a training data set including a diversified combination of photovoltaic active output, load active power and load reactive power data.

3. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 1, characterized in that: In step S2, a Markov decision model for coordinated optimization of active and reactive power in the distribution network is established, and coordinated optimization is achieved by optimizing the reactive power output of distributed photovoltaics in the distribution network and the active power of distributed energy storage, as follows: The partitioned distributed control framework is adopted. The distribution network includes multiple sub-areas, each of which includes multiple nodes. Each node is the location where distributed photovoltaic or distributed energy storage is set in the sub-area. The local controller of each sub-area is the intelligent agent. The distributed photovoltaic and distributed energy storage in the sub-area are controlled by the local controller. The distribution network is a reinforcement learning environment. The Markov decision model POMDP is defined as ,in is the number of agents, is the discount factor; The node feature matrix and the adjacency matrix The state set is composed of, where the characteristic matrix Represents the characteristic information of all nodes, , , , , , and Represents the active load, reactive load, photovoltaic active output, photovoltaic reactive output, state of charge of distributed energy storage and node voltage amplitude of the node; node set Represents all nodes of the distribution network topology, Indicates nodes, is the total number of nodes in the distribution network, include The voltage amplitude of each node; the adjacency matrix Characterize the topological connection relationship, adjacency matrix have OK Column, adjacency matrix The Line Elements of a column Indicates Nodes With Nodes The connection relationship, when the Nodes With Nodes When there is a connection relationship between them, the adjacency matrix Middle Line Elements of a column , otherwise 0; Observation Set Represents the set of observation information of all agents. Each agent can only observe the state quantity of the sub-region where the agent is located. The observation set includes the observation quantities of all agents. The observation quantity of each agent includes the observation quantities of all nodes in the sub-region where the agent is located. Action Set Represents the action set of all intelligent agents, including the reactive output of photovoltaic inverters set by all nodes in the sub-area where each intelligent agent is located and the charging and discharging power of distributed energy storage; T is the state transition function that represents the probability of the agent transferring from the current state to the next state after performing the action; is a reward function that represents the reward obtained by the agent after performing an action; according to the active and reactive power collaborative optimization model of the distribution network, the reward function as follows: ； ； in, and The node voltage offsets are and network loss Additional penalty coefficient of ; To take a positive function, For the Nodes The voltage amplitude, and are the safety upper and lower limits of the node voltage respectively.

4. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 3, characterized in that: In step S2, at each time step In the example, each agent makes the best action based on the observation through the action network, and then according to the reward function Get instant rewards; the goal of each agent is to learn the optimal strategy to maximize the cumulative reward; after all agents have performed the optimal action, POMDP will move to the next state.

5. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 4, characterized in that: In the T-MAAC offline centralized training model described in step S3, the transformer module is embedded in the action network architecture of the agent, and the attention weight mechanism is embedded in the evaluation network of the agent. The action network and the evaluation network work together, and each agent is in a cooperative relationship. A homogeneous agent parameter sharing mechanism is adopted, that is, the action network and evaluation network of each agent are the same.

6. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 5, characterized in that: In the transformer module, the original state features are mapped to the input space to generate the embedded representation of each agent: ； ； in, For the Embedding information of each agent; For the The first sub-region where the agent is located The number of observations per node Projection, projection layer is a linear transformation that converts the original observation into an embedded vector; For the The number of nodes in the sub-region where the agent is located, ; is the original input matrix; According to the original input matrix , mapping the embedded information into a space suitable for calculating attention: ； in, is the index of the number of transformer layers in the transformer module; , , Respectively represent The query matrix, key matrix, and value matrix of the transformer layer; , , Represent the learnable weight matrices for calculating the Query matrix, Key matrix, and Value matrix respectively; For the The input matrix of the transformer layer; The masked self-attention mechanism is used to establish the correlation between nodes in the region; the masked self-attention calculation formula is as follows: ； in, For the The masked self-attention calculation value of the transformer layer; Represents the calculation function of the self-attention mechanism with a mask matrix; is the activation function; is the transpose of the Key matrix, is the scaling factor, whose value is the dimension of the Query matrix; is a mask matrix, in which the elements are either 0 or 1. is the adjacency matrix ; Layer normalization and Linear transformation are applied to the output of each transformer layer to update the embedded features and iteratively build high-level embedded features as follows: ； in, For the The embedded features of the transformer layer output; After multi-layer mask self-attention calculation, an aggregation embedding module is used to aggregate the embedding features of all nodes: ； ； ； ； in, , , are the Query, Key, and Value matrices of the aggregation embedding module, respectively. , , are the learnable parameters of the Query, Key, and Value matrices of the aggregation embedding module respectively; The weighted representation of the global features obtained using the standard attention calculation mechanism Attention is the global feature; is the global embedding feature obtained by further processing through residual connection and layer normalization; is the local neighborhood feature extracted by MaskAttention, i.e. The embedded features output by the transformer layer fuse the local neighborhood features with the global features. The local neighborhood features will not be lost due to the global weighting process, which improves the stability and representation ability of the T-MAAC offline centralized training model. Use the SelectEmbedding module to select all node embeddings that have distributed photovoltaics or distributed energy storage set up, and obtain global information from the perspective of the agent corresponding to the sub-area where the node is located: ； in, Indicates that from The global information extracted from the perspective of each agent, For the The first sub-region where the agent is located nodes; Use the Gated Recurrent Unit module to Mapping to The action value of an agent : 。 7. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 6, characterized in that: Agent Evaluation Network The value is calculated as follows: ； in, For the The current observation action pair of the agent Next value, To evaluate the network parameters, To evaluate the multi-layer perceptron in the network, To evaluate the single-layer perceptron in the network, For the The observations of an agent, For other agents to The weighted sum of the values ​​of the agents is as follows: ； in, For the The agent is The attention weight of each agent; For the The embedded functions of an agent, is the activation function, is a shared matrix; ； in, , , by comparing their similarity to calculate the The agent is The attention weights of each agent.

8. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 7, characterized in that: In step S3, the operating environment of the actual distribution network is simulated through power flow calculation; the experience pool is constructed by adopting the experience replay mechanism to store samples obtained by the interaction between the intelligent agent and the environment to improve the training stability and learning efficiency; the action network is copied to construct the target action network; and the evaluation network is copied to construct the target evaluation network.

9. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 8, characterized in that: In step S3, offline training specifically includes the following steps: S3.

1. Randomly initialize the action network parameters of the T-MAAC offline centralized training model and target action network parameters , evaluate network parameters and target evaluation network parameters ; S3.

2. Initialize the distribution network environment status , read the input light charge training data set and obtain the initial observation of each agent , the number of training steps is set to step, and the number of training rounds is set to episode; S3.

3. Generate the agent's actions based on the agent's action network ; S3.

4. Agent Execution Actions After that, the reward value of the agent is returned ; S3.

5. All agents perform actions After that, the distribution network environment transfers to the next state , the agent obtains the observation value at the next moment ; S3.

6. Sample Store in experience pool Among them, , For the The initial observations of the agents, , For the The action value of an agent, , For the The reward value of an agent, , For the The observation value of each agent at the next moment; S3.7, repeat steps S3.3 to S3.6 until the experience pool The sample size in reaches the set threshold ; In subsequent training rounds, in the experience pool The sample size in has reached the upper limit After that, the FIFO (First In, First Out) strategy is used to roll over and update the experience pool. ; S3.8, Agent in the Experience Pool Randomly extract small batches of data from the batch size. S3.

9. Update the evaluation network according to the following loss minimization formula: ； in, For the The update function of the evaluation network of each agent; For the experience pool The expected return of sampling in is the current observation action pair Next The evaluation network of the agent value With The objective function value of the agent The square of the difference Taking the average value, we get; For the The objective function value of each agent is as follows: ； in, is the discount factor, Observe the action pair for the next moment Next The goal evaluation network of the agent value, To balance the parameters of maximum entropy and reward, No. Agent-based target action network In the current observation Next select action probability; for and The weighted average of the differences between Agent-based target action network The weighted average of all possible actions observed at the next moment; S3.

10. Update the action network according to the gradient descent formula: ； ； in, For the Action network of an agent The gradient of the update function of ; is the policy gradient part, indicating the Action network of an agent In the current observation Next select action The gradient of the probability of is the baseline, used to reduce the variance of the policy gradient, Indicates that except for The joint actions of other agents other than the agent; S3.

11. Update the target action network parameters and target evaluation network parameters of all agents: ； ； in, To update the weights; S3.12, repeat steps S3.8 to S3.11 until all training steps step are completed in this round of training; S3.

13. Repeat steps S3.2 to S3.12 until all rounds of training are completed to obtain a trained T-MAAC offline centralized training model.

10. A distribution network reactive power optimization method based on data enhancement and hybrid enhanced intelligence according to claim 9, characterized in that: In step S4, the trained T-MAAC offline centralized training model is used to intelligently generate the optimization strategy online, and the safety of the output action is strictly guaranteed by introducing the action safety verification module of the hybrid enhanced intelligence paradigm. The specific process is as follows: S4.

1. Load the action network of the agent in the T-MAAC offline centralized training model trained in step S3 and set the online execution time. ; S4.

2. Real-time reading of the light load data of the distribution network to obtain the current time Observations of the agent , ; S4.

3. Generate the current moment in real time based on the trained T-MAAC offline centralized training model The action value of the agent ; S4.

4. Determine the action performed by the agent After Whether the voltage levels of all nodes where photovoltaic inverters are set in the sub-area where the agent is located are within the safety constraint range, if the constraint is met, the action is executed If it is not satisfied, the action will be corrected according to the voltage exceeding the limit, and then the action will be executed ; If Set the node of photovoltaic inverter in the sub-area where the intelligent agent is located The voltage level is higher than the upper limit, and the PV action correction formula is as follows: ； in, and Separately for the moment According to Actions of an agent The reactive power output of the photovoltaic inverter PV in the corresponding sub-area before and after the PV action correction is performed. For the moment No. The voltage levels of all nodes where the photovoltaic inverters are located in the sub-area corresponding to the intelligent agent, is the upper limit of node voltage, For the moment No. The voltage reactive sensitivity of the node where the photovoltaic inverter is located in the sub-area corresponding to each intelligent agent indicates the sensitivity of reactive power regulation to voltage. The node voltage safety constraints are as follows: ； in, is the lower limit of node voltage; S4.

5. Determine the execution action After Whether the state of charge of all distributed energy storage in the sub-area corresponding to the first intelligent agent is within the constraint range, if not, the DESS action is modified according to the limit exceeding; if the first intelligent agent The nodes of distributed energy storage DESS are set in the sub-area where each intelligent agent is located. The state of charge SOC exceeds the upper limit, and the correction formula is as follows: ； in, For the moment According to Actions of an agent The charging and discharging power of the distributed energy storage DESS in the corresponding sub-area after the DESS action correction is performed. is the upper limit of DESS state of charge, For the moment No. The charge state of the distributed energy storage DESS in the sub-area where the intelligent agent is located, is the charging efficiency of DESS, is the upper limit of DESS capacity; The DESS security constraints are as follows: ； in, and are the upper and lower limits of the DESS state of charge, respectively; The DESS energy constraints are as follows: ； ； ； in, For DESS at the moment energy level; is the discharge efficiency of DESS, and are the upper and lower limits of DESS capacity; S4.

6. Repeat steps S4.2 to S4.5 for all agents to obtain the time The corresponding moments of all agents’ actions Optimization strategy ; S4.7, repeat steps S4.2 to S4.6 until the entire online execution time is completed , including online execution time The corresponding online execution time of all the actions of the agents in Optimization strategy .

Images