Power grid loss reduction method and system considering Lorenna series power flow approximation under data driving

Abstract

The invention relates to the technical field of loss reduction optimization of a power distribution network, in particular to a power grid loss reduction method and a power grid loss reduction system considering Lorenna series power flow approximation under data driving, and the method comprises the following steps: converting a line loss optimization task of the power distribution network into an intelligent agent interaction environment; the intelligent agent interaction environment is used for converting a line loss optimization task of the power distribution network into a Markov decision process under a data driving framework; and by adopting the load flow approximation method based on the Loran series, a nonlinear problem in the traditional load flow calculation is converted into a linear problem, so that the calculation complexity is greatly reduced. According to the method, the load flow calculation speed can be remarkably increased on the premise that the calculation precision is guaranteed, and power grid loss reduction optimization can meet the real-time requirement. Especially in a high-permeability distributed new energy access complex power grid, the method can quickly respond to the dynamic change of the power grid state, and effective support is provided for loss reduction optimization in actual operation.

Description

Technical Field

[0001] The present invention relates to the technical field of power distribution network loss reduction optimization, and in particular to a data-driven power distribution network loss reduction method and system considering Laurent series power flow approximation. Background Art

[0002] With the high penetration of renewable energy in the power grid, the operating characteristics of the traditional power grid have changed significantly. The intermittent and volatile nature of renewable energy generation has increased the uncertainty of the power grid's operating status, bringing new challenges to the safety, stability and economy of the power grid. In particular, the changes in power flow in the power grid caused by the access of distributed renewable energy have significantly increased the line loss of the power grid. How to reduce line loss by optimizing the power grid operation strategy while ensuring the quality of power supply has become an important research topic in the current field of distribution network dispatching and control.

[0003] At present, power grid loss reduction optimization mainly faces the following challenges:

[0004] 1) High computational complexity: After high-penetration distributed energy is connected, the power flow distribution of the distribution network is no longer stable, and the direction and size of power flow change frequently. Traditional power flow calculation methods cannot meet the needs of rapid solution of power grid status under high-frequency and real-time demand conditions.

[0005] 2) Increased uncertainty: Distributed renewable energy access introduces a large amount of randomness and uncertainty. Traditional deterministic optimization methods are difficult to cope with the dynamic decision-making needs in this complex environment.

[0006] For example, the invention patent with publication number CN116468159A discloses a reactive power optimization method based on double-delayed deep deterministic policy gradient, which uses two Critic networks to prevent the problem of over-estimation of Q value. At the same time, the algorithm can coordinate the continuous reactive power output of multiple PV and BES, and can control the charging and discharging of BES according to load demand. By optimizing the scheduling of smart inverters every hour, the method proposed by the present invention improves the voltage curve and reduces the active power loss of the distribution system. However, CN116468159A adopts the traditional power flow calculation method. Although it can achieve scheduling optimization, the complex power flow calculation process will limit the ability of real-time optimization. Moreover, the double-delayed deep deterministic policy gradient algorithm used by CN116468159A is a reinforcement learning model based on single-point value estimation. It is insufficient in processing complex distribution characteristics, which may lead to low robustness of the strategy. Moreover, the deterministic policy gradient algorithm mainly focuses on the deterministic optimization of the strategy, lacks sufficient constraints and incentives for exploration behavior, and is easy to fall into suboptimal strategies during training. Summary of the invention

[0007] Purpose of the invention: In order to solve the problems of high computational complexity and poor efficiency in the process of optimizing distribution network loss reduction, the present invention provides a data-driven power grid loss reduction method considering Laurent series power flow approximation. The present invention also discloses a data-driven power grid loss reduction system considering Laurent series power flow approximation.

[0008] Technical solution: In a first aspect, the present invention provides a data-driven power grid loss reduction method considering Laurent series power flow approximation, the method comprising:

[0009] The line loss optimization task of the distribution network is converted into an intelligent agent interaction environment, wherein the intelligent agent interaction environment is used to convert the line loss optimization task of the distribution network into a Markov decision process under a data-driven framework;

[0010] Construct a reinforcement learning model based on Distributive Soft Actor-Critic; the reinforcement learning model includes an Actor network and a Critic network, the Actor network is used to generate continuous actions according to the state space of the distribution network, and obtain the optimal strategy using the maximum entropy framework, and the Critic network is used to construct the probability distribution of the expected return using the mixed density network MDN, and approximate the target distribution of the expected return by minimizing the loss function;

[0011] The power flow in the intelligent agent interactive environment is calculated by using a linear power flow approximation method based on Laurent series, so as to obtain the optimized line loss of the entire system. The linear power flow approximation method based on Laurent series is to construct a corresponding linear equation group based on the relationship between the node current and the voltage of the distribution network, and iteratively solve the linear equation group to obtain the voltage of each load node after the power flow calculation and the optimized line loss;

[0012] The data samples are trained in a data-driven environment to achieve online loss reduction optimization of the distribution network, including:

[0013] Exploration phase: The agent collects a large number of empirical trajectory data samples to characterize the impact of grid status and strategy on line loss;

[0014] Training phase: Use the data samples collected in the exploration phase to train the Distributed Soft Actor-Critic based reinforcement learning model, so that it can gradually learn to extract information from the state, evaluate the strategy, and optimize the action selection;

[0015] Execution phase: The optimization action selection is sent to the distribution network environment. After receiving the power action, the distribution network performs power flow calculation based on the linear power flow approximation method of Laurent series to obtain the optimized effect of the action, namely the line loss and the corresponding node voltage.

[0016] Further, including:

[0017] The Markov decision process under the data-driven framework also includes: state space, action space, reward function and state transition probability distribution; wherein,

[0018] The state space is represented as follows: The state of the distribution network at each time t consists of the electrical characteristics of each node, including active power, reactive power, and voltage amplitude;

[0019] Action space: reactive output of PV inverters and active output of energy storage in the distribution network;

[0020] Reward function: It aims to guide the agent to execute the loss reduction strategy to optimize the operation of the power grid. The immediate reward r at the t decision step is t is the loss cost of the distribution network;

[0021] State transfer process: The agent observes the current state s at each time step t t , and select an action a according to the strategy π t After this action is applied to the distribution network, the system transfers to the next state s t+1 , and receive an instant reward t , the goal of the agent is to find a strategy π to maximize the expectation of long-term cumulative rewards through continuous interaction.

[0022] Further, including:

[0023] The optimal strategy in the Actor network is expressed as:

[0024]

[0025] in, represents the expectation, γ is the discount factor, r(s t ,a t ) is the immediate reward, H(π(·|s t )) represents the entropy term, which reflects the agent in state s t The randomness of the decision distribution under is used to maintain the exploratory nature of the strategy, and α is the coefficient of the entropy term.

[0026] H(π(·|s t )) and αlogπ(a′|s′) are different because they differ in their specific content and usage scenarios. The former reflects the overall randomness of the strategy and is used to describe the exploratory nature of the strategy; the latter is incorporated into the objective function as a regularization term, reflecting the weighting of the probability of specific actions. The two are essentially the same, but differ in form to adapt to different computing scenarios.

[0027] Further, including:

[0028] The construction process of the Critic network includes:

[0029] First, the goal of the Critic network is to approximate the expected return of the state-action pair, namely:

[0030]

[0031] Where Z(s,a) represents the current estimated distribution of state-action pair benefits;

[0032] Secondly, in order to handle the distribution estimation of returns, the Bellman equation is adjusted to adapt to the calculation of distributed critics. Therefore, the target distribution is defined as:

[0033]

[0034] Where R(s,a) is the reward function, γ is the discount factor, Z(s′,a′) represents the reward distribution of the next state-action pair, and αlogπ(a′|s′) is the entropy regularization term, which aims to pass the reward distribution;

[0035] Finally, the difference between the current distribution and the target distribution is minimized by solving the distance between the distributions. The Wasserstein distance W1 is used to measure the distance between the two random variables. The formula is as follows:

[0036]

[0037] In the formula, and are the cumulative distribution functions of distribution μ1 and μ2, corresponding to minimizing the current estimated distribution Z(s,a) of the Critic network and the target distribution represents a distributed Bellman operator, which updates the estimate of the payoff distribution of a state-action pair.

[0038] Therefore, the loss function of the Critic network is defined as:

[0039]

[0040] Among them, ω is the parameter of the Critic network. By minimizing the loss function, the target distribution of returns can be approached.

[0041] Further, including:

[0042] According to the relationship between the node current and voltage of the distribution network, the Laurent series is used to construct the corresponding linear equation group, which specifically includes:

[0043] The relationship between the voltage and current at the constant voltage node and the load node is expressed according to Kirchhoff's law;

[0044] The Laurent series expansion is performed on the inverse of the load node voltage to approximately describe the nonlinear relationship between voltage and current;

[0045] Use the ZIP load model to model the load node and express the complex power injection vector and current injection vector of the load node;

[0046] The linear equations are constructed by combining the nonlinear relationship between voltage and current, the complex power injection vector and the current injection vector of the load node.

[0047] Further, including:

[0048] The complex power injection vector of the load node is expressed as:

[0049] S d =(diag(α P )+diag(α I V d )+diag(α Z |V d | 2 ))S n ;

[0050] Among them, α P , α I and α Z are the load factors of the ZIP load model, S d is the complex power injection vector of the load node. The complex power is the complex form of the active power and reactive power of each load node. diag is the diagonal matrix operator, which converts the vector or matrix in the brackets into a diagonal matrix. V d is the voltage vector of the load node, S n is the rated complex power of the load node;

[0051] The current injection vector of the load node is expressed as:

[0052]

[0053] in, is the conjugate of the load node voltage vector, is the conjugate vector of the rated complex power of the load node.

[0054] Further, including:

[0055] The linear equation system is expressed as:

[0056]

[0057] In the formula, α P , α I and α Z are the load factors of the ZIP model, diag is the diagonal matrix operator, is the conjugate square inverse of the reference voltage, is the conjugate vector of the node rated complex power, the symbol ⊙ represents the Hadamard product, Y dd represents the admittance submatrix between two load nodes, Y ds represents the admittance submatrix between the load node and the constant voltage node, V s is the voltage at the constant voltage node, is the conjugate square inverse of the reference voltage, Indicates the conjugate reciprocal of the reference voltage.

[0058] Further, including:

[0059] The iterative solution of the linear equation group to obtain the voltage of each load node after the power flow calculation and the optimized line loss includes:

[0060] The linear equations are iteratively solved to obtain the voltage V of each node after the power flow calculation. d , expressed as:

[0061]

[0062] According to the voltage V of each load node d Calculate the line loss of the entire system:

[0063] P Loss =∑ m:n→m Re((V nm )·(Y nm ·V nm ) * );

[0064] Here, P Loss is the line loss of the entire distribution network system, V nm is the voltage drop of line nm, V nm =V n -V m , Y nm represents the admittance of the line, and Re(·) represents the real part of the complex number.

[0065] Further, including:

[0066] The training phase specifically includes:

[0067] Batch data sampling: Randomly extract small batches of data samples (s t ,a t ,rt ,s t+1 ) to reduce the correlation between samples and improve the stability of training;

[0068] Critic network update: Calculate target distribution based on the adjusted Bellman equation And use the Wasserstein distance W1 to minimize the gap between the target distribution and the current estimated distribution Z(s,a), thereby optimizing the Critic network;

[0069] Actor network update: By maximizing the expected benefit objective function with entropy regularization, the Actor network is updated so that it can balance exploration and exploitation when selecting actions, thus improving the effectiveness of the strategy.

[0070] Parameter adjustment: Dynamically update the coefficient α of the entropy term to ensure a balance between exploration and exploitation.

[0071] On the other hand, the present invention also provides a data-driven power grid loss reduction system considering Laurent series power flow approximation, the system comprising:

[0072] A task conversion module, used to convert the line loss optimization task of the distribution network into an intelligent agent interaction environment, wherein the intelligent agent interaction environment is used to convert the line loss optimization task of the distribution network into a Markov decision process under a data-driven framework;

[0073] A reinforcement learning model construction module is used to construct a reinforcement learning model based on Distributed Soft Actor-Critic; the reinforcement learning model includes an Actor network and a Critic network, the Actor network is used to generate continuous actions according to the state space of the distribution network, and obtain the optimal strategy using the maximum entropy framework, and the Critic network is used to construct the probability distribution of expected returns using a mixed density network MDN, and approximate the target distribution of expected returns by minimizing the loss function;

[0074] A power flow calculation module, used for calculating the power flow in the intelligent agent interaction environment by using a linear power flow approximation method based on Laurent series, and then obtaining the optimized line loss of the entire system, wherein the linear power flow approximation method based on Laurent series is to construct a corresponding linear equation group based on the relationship between the node current and the voltage of the distribution network, and iteratively solve the linear equation group to obtain the voltage of each load node after the power flow calculation and the optimized line loss;

[0075] The loss reduction optimization module is used to train data samples in a data-driven environment and ultimately achieve online loss reduction optimization of the distribution network. Specifically, it includes:

[0076] Exploration phase: The agent collects a large number of empirical trajectory data samples to characterize the impact of grid status and strategy on line loss;

[0077] Training phase: Use the data samples collected in the exploration phase to train the Distributed Soft Actor-Critic based reinforcement learning model, so that it can gradually learn to extract information from the state, evaluate the strategy, and optimize the action selection;

[0078] Execution phase: The optimization action selection is sent to the distribution network environment. After receiving the power action, the distribution network performs power flow calculation based on the linear power flow approximation method of Laurent series to obtain the optimized effect of the action, namely the line loss and the corresponding node voltage.

[0079] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0080] The present invention adopts a power flow approximation method based on Laurent series to transform the nonlinear problem in traditional power flow calculation into a linear problem, greatly reducing the computational complexity. This method can significantly improve the speed of power flow calculation while ensuring the calculation accuracy, so that the power grid loss reduction optimization can meet the real-time requirements. Especially in complex power grids with high penetration of distributed new energy access, this method can quickly respond to the dynamic changes of the power grid state, providing effective support for loss reduction optimization in actual operation.

[0081] The present invention combines the Distributed Soft Actor-Critic (DSAC) algorithm to build a reinforcement learning model, introduces a distributed Bellman operator and a mixed density network to estimate the distribution of benefits and quantify uncertainty, so that the intelligent agent can more accurately evaluate the effect of each action. This distributed reinforcement learning method shows good adaptability and stability in the uncertain environment brought by high penetration of new energy, can achieve more robust loss reduction control, effectively reduce line losses in power grid operation, and improve the overall optimization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A flow chart of a method for reducing power losses in a data-driven grid considering Laurent series power flow approximation according to an embodiment of the present invention;

[0083] Figure 2 This is a flow chart of the online loss reduction optimization method for the distribution network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] like Figure 1 As shown, the present invention provides a data-driven power grid loss reduction method considering Laurent series power flow approximation, the method comprising the following steps:

[0086] Step 1: Convert the power grid line loss optimization task into a Markov decision process under a data-driven framework, including state space, action space, reward function, state transition probability distribution and reinforcement learning agent interaction environment;

[0087] The goal of line loss optimization in power grids is to reduce power losses of lines through active and reactive power compensation devices in the power grid. Therefore, the decision-making center of the power grid is regarded as an intelligent entity to dispatch photovoltaic (PV) inverters and energy storage (BSS) equipment in the power grid with high penetration of new energy to optimize network losses.

[0088] State space: The state of the distribution network is composed of the electrical characteristics of each node at each moment, including active power P(t), reactive power Q(t) and voltage amplitude V(t).

[0089] Action space: The action space includes the reactive output of the PV inverter and the active output of the energy storage in the grid. Assuming that the grid contains m PVs and n BSSs, the action vector is defined as: t = {Q PV (t),P BSS (t)}.

[0090] Reward function: It aims to guide the intelligent agent to execute the loss reduction strategy to optimize the operation of the power grid. The reward function of the present invention is the loss cost of the power grid, which is in the form of: r t =-C p P Loss (t).

[0091] State transfer process: The agent observes the current state s at each time step t t , and select an action a according to the strategy π t After this action is applied to the distribution network, the system transfers to the next state s t+1 , and receive an instant reward t The goal of the agent is to find a strategy π through continuous interaction to maximize the expected long-term cumulative reward. Among them, γ is the discount factor.

[0092] Agent Interaction Environment: The agent interaction environment is responsible for converting the power grid loss reduction task into a data-driven Markov decision process (MDP), including state space, action space, reward function, state transition dynamics, and agent interaction process based on Laurent series power flow calculation. This interactive environment not only helps the agent obtain the real-time state of the power grid, but also uses power flow calculation methods to evaluate the effect of each action, thereby achieving online learning and loss reduction optimization.

[0093] Step 2: Propose a linear power flow approximation method based on Laurent series to improve the power flow calculation process in the intelligent agent interaction environment, so as to improve the calculation efficiency of power grid line loss;

[0094] In order to improve the calculation efficiency of power grid line loss, this step proposes a linear power flow approximation method based on Laurent series to improve the power flow calculation process in the interactive environment of reinforcement learning agents. The action of the reinforcement learning agent is power, so the power of the distribution network will change after the action. After the power changes, the power flow calculation must be performed to obtain the changed power flow calculation results. Consider the node set of the power grid and construct its admittance matrix Y to describe the electrical connection between the network nodes. Based on the characteristics of the single-phase power grid, the Laurent series is used to approximate the inverse function of the node voltage to simplify the nonlinear relationship in the power flow calculation.

[0095] First, a model describing the relationship between node current and voltage is constructed. Consider a power grid structure containing constant voltage nodes and load nodes. According to Kirchhoff's law, the relationship between node current and voltage can be expressed as:

[0096]

[0097] In the above formula, I s and I d are the current injections into the constant voltage node and the load node, respectively, V s and V d is the voltage of the corresponding node, Y ss , Y sd , Y ds and Y dd They represent the admittance sub-matrices between constant voltage nodes, between constant voltage nodes and load nodes, between load nodes and constant voltage nodes, and between load nodes respectively.

[0098] In order to improve the calculation efficiency, the Laurent series expansion is performed on the inverse of the load node voltage to approximate the nonlinear relationship between voltage and current. The Laurent series expansion and the approximation of the inverse of the node voltage can be expressed as follows: In the loop r1<|z-z0|<r2, the Laurent series expansion is used to expand the inverse of the node voltage (V i ) -1 Specifically, assuming that the node voltage Vi At a given point V 0 If the change in the vicinity is small, a first-order approximation can be performed to obtain the following approximate expression:

[0099]

[0100] In the formula, f(V i ) represents the voltage V at node i i The inverse function of the node voltage is a nonlinear approximation, V 0 is a reference voltage used as an approximate center in the Laurent series expansion.

[0101] Then, the ZIP load model is introduced. In order to more accurately reflect the load characteristics, the load node is modeled using the ZIP load model, and the node load is defined as a combination of constant power, constant current and constant impedance. d It is expressed as:

[0102] S d =(diag(α P )+diag(α I V d )+diag(α Z |V d | 2 ))S n

[0103] In the formula, α P , α I and α Z are the load factor of the ZIP model, S d is the complex power injection vector of the load node. The complex power is the complex form of the active power and reactive power of each node. diag is the diagonal matrix operator, which converts the vector or matrix in the brackets into a diagonal matrix. V d is the voltage vector of the load node, S n is the rated complex power of the node.

[0104] Using the ZIP model, the current injection into the load node can be more accurately modeled as:

[0105]

[0106] In the formula, I d represents the current injection vector of the load node, is the conjugate of the load node voltage vector, is the conjugate vector of the node's rated complex power. This model ensures that the intelligent agent can effectively consider different load characteristics when calculating the relationship between node current and voltage, thereby improving the accuracy of grid state estimation.

[0107] Finally, combined with the above approximate model, the relationship between node current and voltage can be constructed as a linear equation group, which significantly reduces the complexity of power flow calculation. The form of the linear equation group is:

[0108]

[0109] in:

[0110]

[0111] In the formula, α P , α I and α Z are the load factors of the ZIP model, diag is the diagonal matrix operator, is the conjugate square inverse of the reference voltage, is the conjugate vector of the node rated complex power, and the symbol ⊙ represents the Hadamard product (element-by-element multiplication).

[0112] By iteratively solving the above linear equations, the voltage V of each node after the power flow calculation can be obtained. d , calculated as:

[0113]

[0114] Furthermore, the present invention calculates the line loss of the entire system according to the voltage of each node:

[0115] P Loss =Σ m:n→m Re((V nm )·(Y nm ·V nm ) * )

[0116] Here, P Loss is the line loss of the entire power grid system, V nm is the voltage drop of line nm, V nm =V n -V m , Y nm represents the admittance of the line, and Re(·) represents the real part of the complex number.

[0117] Step 3: Build a reinforcement learning model based on Distributed Soft Actor-Critic, design a distributed Bellman operator and a critic network based on a mixed density network to improve the decision-making effect of the agent in the task of reducing power grid losses;

[0118] In this step, a reinforcement learning model based on Distributed Soft Actor-Critic (DSAC) is constructed to improve the decision-making effect of the agent in the task of reducing the loss of the distribution network. By designing a unique actor network and critic network and taking advantage of distributed reinforcement learning, it can effectively deal with the uncertainty in the environment and improve the stability and robustness of the strategy.

[0119] The DSAC algorithm mainly includes the following key networks:

[0120] Actor network π(s): The Actor network is used for strategy generation. Its goal is to generate continuous actions a from a given state s to maximize the cumulative return of the agent. In DSAC, the Actor network adopts the maximum entropy framework, and its goal is to maximize the following expected return function:

[0121]

[0122] In the formula, π * represents the best strategy, represents the expectation, γ is the discount factor, r(s t ,a t ) is the immediate reward, H(π(·|s t )) represents the entropy term, which is used to maintain the exploratory nature of the strategy, and α is the coefficient that controls the entropy regularization weight. The introduction of the entropy term maintains a balance between exploration and utilization, allowing the agent to adapt to complex loss reduction tasks. The Actor network extracts the temporal characteristics of the environment through a recurrent neural network to ensure that the strategy generation has dynamic adaptability.

[0123] Critic network Q(s,a): The Critic network is used to evaluate the value distribution of a given state-action pair. In traditional reinforcement learning methods, the Q value is a single point estimate and cannot characterize the benefit distribution of the strategy. However, DSAC uses distributed reinforcement learning to introduce a mixture density network (MDN) to estimate the distribution of the Critic value. MDN can model distributions of different shapes, including irregular distributions such as long-tail distributions, allowing the Critic network to more accurately reflect uncertainty.

[0124] The Actor network π(s) in step 3 can generate an action a based on the current state s, and the definition of action a is a t = {Q PV (t),P BSS (t)};

[0125] It is the active power of photovoltaics and the reactive power of energy storage. When this power is sent to the distribution network, the power flow of the distribution network will change, so it is necessary to calculate the power flow. According to the power flow calculation results (the calculation results can obtain the network loss and voltage conditions to calculate the reward function of reinforcement learning), the effect of the action is calculated. The complex power composed of active and reactive power is S in step 2. n The purpose of power flow calculation is to calculate the voltage V d , after obtaining the voltage of each node, the network loss can be calculated.

[0126] The goal of the critic network is to approximate the expected return of a state-action pair, namely:

[0127]

[0128] Where Z(s,a) represents the revenue distribution of the state-action pair. MDN generates the mean, variance, and mixing coefficient of multiple mixed distributions to construct the probability distribution of the revenue, providing a more flexible and accurate value assessment.

[0129] In DSAC, in order to handle the distribution estimation of returns, the Bellman equation needs to be adjusted to adapt to the calculation of distributed critics. This adjusted Bellman equation is called the distributed Bellman operator, which is defined as:

[0130]

[0131] In the formula, R(s,a) is the reward function, γ is the discount factor, Z(s′,a′) represents the reward distribution of the next state-action pair, and αlogπ(a′|s′) is the entropy regularization term, which aims to encourage the agent to maintain randomness when exploring strategies through the reward distribution. t )) and αlogπ(a′|s′) are different because they differ in their specific content and usage scenarios. The former reflects the overall randomness of the strategy and is used to describe the exploratory nature of the strategy; the latter is incorporated into the objective function as a regularization term, reflecting the weighting of the probability of specific actions. The two are essentially the same, but differ in form to adapt to different computing scenarios. R(s,a) is the reward function, r t is an immediate and specific reward value, used to describe the benefit of the t decision step; R(s,a) is the benefit expanded to the distribution level, which combines the long-term goal and the entropy regularization term to enhance the stability of the strategy and the distribution estimation ability.

[0132] In distributed reinforcement learning, the goal is to minimize the difference between the current distribution and the target distribution by solving the distance between the distributions. DSAC uses the Wasserstein distance W1 to measure the distance between two random variables. The formula is as follows:

[0133]

[0134] In the formula, and are the cumulative distribution functions of distribution μ1 and μ2 respectively, by minimizing the estimated distribution Z(s,a) of the Critic network and the target distribution The Wasserstein distance between them can get a more accurate value assessment.

[0135] Therefore, the loss function of the Critic network is defined as:

[0136]

[0137] Among them, ω is the parameter of the Critic network. By minimizing the loss function, the target distribution of returns can be approached, thereby providing a more stable and reliable value assessment.

[0138] Step 4: Based on the Laurent series power flow approximation method in step 2 and the Distributed SoftActor-Critic reinforcement learning model in step 3, train a data-driven online loss reduction agent, such as Figure 2 shown.

[0139] This step describes the training process of the DSAC agent in a data-driven environment to achieve online power grid loss reduction optimization. The training process is divided into three main stages: exploration stage, training stage and execution stage. Through these three stages, the decision-making ability of the agent is gradually optimized to achieve the goal of reducing power grid line losses.

[0140] Step 4.1: Exploration phase. In the exploration phase, the agent mainly interacts with the environment through random actions to collect enough data samples to characterize the impact of grid status and strategy on line loss. The specific steps include:

[0141] Random action generation: The agent selects action a in a random manner t , covering various parts of the action space and ensuring the diversity of data distribution.

[0142] Power flow calculation based on Laurent series: At each time step t, the selected action a t The data is transmitted to the power grid, and the power flow and line loss are calculated using the Laurent series power flow approximation method in step 2. The efficiency of this approximation method ensures that the exploration phase can maintain rapid calculation when a large number of action samples are taken. The agent makes an action and sends it to the environment, calculates the power flow, and provides feedback on the quality of the action.

[0143] Data storage: records the state of each interaction t 、Action a t、Instant Rewards t and the next state s t+1 , and store it in the experience replay pool to provide sample data for subsequent training stages.

[0144] Step 4.2: Training phase. In the training phase, the Actor network and Critic network of the DSAC agent are trained using the data collected in the exploration phase, so that it gradually learns to extract information from the state, evaluate strategies, and optimize action selection. The specific steps include:

[0145] Batch data sampling: Randomly extract small batches of data samples (s t ,a t ,r t ,s t+1 ) to reduce the correlation between samples and improve the stability of training.

[0146] Critic network update: Calculate the target distribution according to the distributed Bellman equation in step 3 And the Wasserstein distance W1 is used to minimize the gap between the target distribution and the current estimated distribution Z(s,a), thereby optimizing the Critic network.

[0147] Actor network update: By maximizing the entropy regularized expected return objective function, the Actor network is updated so that it can balance exploration and exploitation when selecting actions, improving the effectiveness of the strategy.

[0148] Temperature parameter adjustment: Dynamically update the entropy regularization parameter α to ensure a balance between exploration and exploitation.

[0149] Step 4.3: Execution phase. In the execution phase, the trained Actor network is deployed to the actual environment to perform the task of reducing power grid losses. The specific process is as follows:

[0150] Policy network preservation: Save the trained Actor network parameters to form the final policy network of the agent.

[0151] Online decision-making and loss reduction optimization: The intelligent agent makes decisions based on the current state s in actual power grid operation. t Choose the best action a t , and use power flow calculation methods to verify and optimize power grid line losses to ensure the loss reduction effect of the system under different working conditions.

[0152] In summary, first of all, the idea of ​​the reinforcement learning method is that the agent and the environment interact to generate a large number of experience trajectories, that is, the exploration phase, and then the agent learns the best strategy from a large number of experience trajectories, that is, the training phase. Then, during the interaction process in the exploration phase, the DSAC agent in step 3 makes an action through the Actor network, that is, the action space defined in step 1 is sent to the distribution network environment. After receiving this action on power, the distribution network will perform a power flow calculation to obtain the optimized effect of this action, that is, calculate the network loss and voltage. This power flow calculation process uses the power flow calculation method based on the Laurent series in step 2.

[0153] Therefore, step 1 defines the line loss solution framework based on reinforcement learning, namely MDP, including actions, states, rewards, and distribution network interaction environment. Step 2 improves the power flow calculation process in the interactive environment. Step 3 designs a specific reinforcement learning algorithm to solve the framework of step 1. Step 4 describes the specific three-stage solution process.

[0154] In another aspect of this embodiment, a data-driven power grid loss reduction system considering Laurent series power flow approximation is provided, the system comprising:

[0155] A task conversion module, used to convert the line loss optimization task of the distribution network into an intelligent agent interaction environment, wherein the intelligent agent interaction environment is used to convert the line loss optimization task of the distribution network into a Markov decision process under a data-driven framework;

[0156] A reinforcement learning model construction module is used to construct a reinforcement learning model based on Distributed Soft Actor-Critic; the reinforcement learning model includes an Actor network and a Critic network, the Actor network is used to generate continuous actions according to the state space of the distribution network, and obtain the optimal strategy using the maximum entropy framework, and the Critic network is used to construct the probability distribution of expected returns using a mixed density network MDN, and approximate the target distribution of expected returns by minimizing the loss function;

[0157] A power flow calculation module, used for calculating the power flow in the intelligent agent interaction environment by using a linear power flow approximation method based on Laurent series, and then obtaining the optimized line loss of the entire system, wherein the linear power flow approximation method based on Laurent series is to construct a corresponding linear equation group based on the relationship between the node current and the voltage of the distribution network, and iteratively solve the linear equation group to obtain the voltage of each load node after the power flow calculation and the optimized line loss;

[0158] The loss reduction optimization module is used to train data samples in a data-driven environment and ultimately achieve online loss reduction optimization of the distribution network. Specifically, it includes:

[0159] Exploration phase: The agent collects a large number of empirical trajectory data samples to characterize the impact of grid status and strategy on line loss;

[0160] Training phase: Use the data samples collected in the exploration phase to train the reinforcement learning model based on Distributive Soft Actor-Critic, so that it can gradually learn to extract information from the state, evaluate strategies, and optimize action selection; Execution phase: Send the optimized action selection to the distribution network environment. After receiving this power action, the distribution network performs power flow calculation based on the linear power flow approximation method of Laurent series, thereby obtaining the optimized effect of this action, that is, the line loss and the corresponding node voltage.

[0161] Other technical features corresponding to the data-driven power grid loss reduction system considering Laurent series power flow approximation described in the embodiment of the present invention are similar to the data-driven power grid loss reduction method considering Laurent series power flow approximation, and will not be repeated here.

[0162] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0164] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data-driven power grid loss reduction method considering Laurent series power flow approximation, characterized in that: The method includes: The line loss optimization task of the distribution network is converted into an intelligent agent interaction environment, wherein the intelligent agent interaction environment is used to convert the line loss optimization task of the distribution network into a Markov decision process under a data-driven framework; Construct a reinforcement learning model based on Distributive Soft Actor-Critic; the reinforcement learning model includes an Actor network and a Critic network, the Actor network is used to generate continuous actions according to the state space of the distribution network, and obtain the optimal strategy using the maximum entropy framework, and the Critic network is used to construct the probability distribution of the expected return using the mixed density network MDN, and approximate the target distribution of the expected return by minimizing the loss function; The power flow in the intelligent agent interactive environment is calculated by using a linear power flow approximation method based on Laurent series, so as to obtain the optimized line loss of the entire system. The linear power flow approximation method based on Laurent series is to construct a corresponding linear equation group based on the relationship between the node current and the voltage of the distribution network, and iteratively solve the linear equation group to obtain the voltage of each load node after the power flow calculation and the optimized line loss; The data samples are trained in a data-driven environment to achieve online loss reduction optimization of the distribution network, including: Exploration phase: The agent collects a large number of empirical trajectory data samples to characterize the impact of grid status and strategy on line loss; Training phase: Use the data samples collected in the exploration phase to train the Distributed Soft Actor-Critic based reinforcement learning model, so that it can gradually learn to extract information from the state, evaluate the strategy, and optimize the action selection; Execution phase: The optimization action selection is sent to the distribution network environment. After receiving the power action, the distribution network performs power flow calculation based on the linear power flow approximation method of Laurent series to obtain the optimized effect of the action, namely the line loss and the corresponding node voltage.

2. The data-driven power grid loss reduction method considering Laurent series power flow approximation according to claim 1 is characterized in that: The Markov decision process under the data-driven framework also includes: state space, action space, reward function and state transition probability distribution; wherein, The state space is represented as follows: The state of the distribution network at each time t consists of the electrical characteristics of each node, including active power, reactive power, and voltage amplitude; Action space: reactive output of PV inverters and active output of energy storage in the distribution network; Reward function: It aims to guide the agent to execute the loss reduction strategy to optimize the operation of the power grid. The immediate reward r at the t decision step is t is the loss cost of the distribution network; State transfer process: The agent observes the current state s at each time step t t , and select an action a according to the strategy π t After this action is applied to the distribution network, the system transfers to the next state s t+1 , and receive an instant reward t , the goal of the agent is to find a strategy π to maximize the expectation of long-term cumulative rewards through continuous interaction.

3. The data-driven power grid loss reduction method considering Laurent series power flow approximation according to claim 2 is characterized in that: The optimal strategy in the Actor network is expressed as: in, represents the expectation, γ is the discount factor, r(s t ,a t ) is the immediate reward, H(π(·|s t )) represents the entropy term, which reflects the agent in state s t The randomness of the decision distribution under is used to maintain the exploratory nature of the strategy, and α is the coefficient of the entropy term.

4. The data-driven power grid loss reduction method considering Laurent series power flow approximation according to claim 3 is characterized in that: The construction process of the Critic network includes: First, the goal of the Critic network is to approximate the expected return of the state-action pair, namely: Where Z(s,a) represents the current estimated distribution of state-action pair benefits; Secondly, in order to handle the distribution estimation of returns, the Bellman equation is adjusted to adapt to the calculation of distributed critics. Therefore, the target distribution is defined as: Where R(s,a) is the reward function, γ is the discount factor, Z(s′,a′) represents the reward distribution of the next state-action pair, and αlogπ(a′|s′) is the entropy regularization term, which aims to encourage the agent to maintain randomness when exploring strategies through reward distribution. Finally, the difference between the current distribution and the target distribution is minimized by solving the distance between the distributions. The Wasserstein distance W1 is used to measure the distance between the two random variables. The formula is as follows: In the formula, and are the cumulative distribution functions of distribution μ1 and μ2, corresponding to minimizing the current estimated distribution Z(s,a) of the Critic network and the target distribution represents the distributed Bellman operator, which is used to update the estimate of the payoff distribution of state-action pairs; Therefore, the loss function of the Critic network is defined as: Among them, ω is the parameter of the Critic network. By minimizing the loss function, the target distribution of returns can be approached.

5. The data-driven power grid loss reduction method considering Laurent series power flow approximation according to claim 4 is characterized in that: According to the relationship between the node current and voltage of the distribution network, the Laurent series is used to construct the corresponding linear equation group, which specifically includes: The relationship between the voltage and current at the constant voltage node and the load node is expressed according to Kirchhoff's law; The Laurent series expansion is performed on the inverse of the load node voltage to approximately describe the nonlinear relationship between voltage and current; Use the ZIP load model to model the load node and express the complex power injection vector and current injection vector of the load node; The linear equations are constructed by combining the nonlinear relationship between voltage and current, the complex power injection vector and the current injection vector of the load node.

6. The data-driven power grid loss reduction method considering Laurent series power flow approximation according to claim 5, characterized in that: The complex power injection vector of the load node is expressed as: S d =(diag(α P )+diag(α I V d )+diag(α Z |V d | 2 ))S n ; Among them, α P , α I and α Z are the load factors of the ZIP load model, S d is the complex power injection vector of the load node. The complex power is the complex form of the active power and reactive power of each load node. diag is the diagonal matrix operator, which converts the vector or matrix in the brackets into a diagonal matrix. V d is the voltage vector of the load node, S n is the rated complex power of the load node; The current injection vector of the load node is expressed as: in, is the conjugate of the load node voltage vector, is the conjugate vector of the rated complex power of the load node.

7. The data-driven power grid loss reduction method considering Laurent series power flow approximation according to claim 6, characterized in that: The linear equation system is expressed as: In the formula, α P , α I and α Z are the load factors of the ZIP model, diag is the diagonal matrix operator, V 0*-2 is the conjugate square inverse of the reference voltage, is the conjugate vector of the node rated complex power, the symbol ⊙ represents the Hadamard product, Y dd represents the admittance submatrix between two load nodes, Y ds represents the admittance submatrix between the load node and the constant voltage node, V s is the voltage of the constant voltage node, V 0*-2 is the conjugate square inverse of the reference voltage, V 0*-1 Indicates the conjugate reciprocal of the reference voltage.

8. The data-driven power grid loss reduction method considering Laurent series power flow approximation according to claim 7, characterized in that: The iterative solution of the linear equation group to obtain the voltage of each load node after the power flow calculation and the optimized line loss includes: The linear equations are iteratively solved to obtain the voltage V of each node after the power flow calculation. d , expressed as: According to the voltage V of each load node d Calculate the line loss of the entire system: P Loss =∑ m:n→m Re((V nm )·(Y nm ·V nm ) * ); Here, P Loss is the line loss of the entire distribution network system, V nm is the voltage drop of line nm, V nm =V n -V m , Y nm represents the admittance of the line, and Re(·) represents the real part of the complex number.

9. The data-driven power grid loss reduction method considering Laurent series power flow approximation according to claim 8, characterized in that: The training phase specifically includes: Batch data sampling: Randomly extract small batches of data samples (s t ,a t ,r t ,s t+1 ) to reduce the correlation between samples and improve the stability of training; Critic network update: Calculate target distribution based on the adjusted Bellman equation And use the Wasserstein distance W1 to minimize the gap between the target distribution and the current estimated distribution Z(s,a), thereby optimizing the Critic network; Actor network update: By maximizing the expected benefit objective function with entropy regularization, the Actor network is updated so that it can balance exploration and exploitation when selecting actions, thus improving the effectiveness of the strategy. Parameter adjustment: Dynamically update the coefficient α of the entropy term to ensure a balance between exploration and exploitation.

10. A data-driven power grid loss reduction system considering Laurent series power flow approximation, characterized in that: The system includes: A task conversion module, used to convert the line loss optimization task of the distribution network into an intelligent agent interaction environment, wherein the intelligent agent interaction environment is used to convert the line loss optimization task of the distribution network into a Markov decision process under a data-driven framework; A reinforcement learning model construction module is used to construct a reinforcement learning model based on Distributed Soft Actor-Critic; the reinforcement learning model includes an Actor network and a Critic network, the Actor network is used to generate continuous actions according to the state space of the distribution network, and obtain the optimal strategy using the maximum entropy framework, and the Critic network is used to construct the probability distribution of expected returns using a mixed density network MDN, and approximate the target distribution of expected returns by minimizing the loss function; A power flow calculation module, used for calculating the power flow in the intelligent agent interaction environment by using a linear power flow approximation method based on Laurent series, and then obtaining the optimized line loss of the entire system, wherein the linear power flow approximation method based on Laurent series is to construct a corresponding linear equation group based on the relationship between the node current and the voltage of the distribution network, and iteratively solve the linear equation group to obtain the voltage of each load node after the power flow calculation and the optimized line loss; The loss reduction optimization module is used to train data samples in a data-driven environment and ultimately achieve online loss reduction optimization of the distribution network. Specifically, it includes: Exploration phase: The agent collects a large number of empirical trajectory data samples to characterize the impact of grid status and strategy on line loss; Training phase: Use the data samples collected in the exploration phase to train the Distributed Soft Actor-Critic based reinforcement learning model, so that it can gradually learn to extract information from the state, evaluate the strategy, and optimize the action selection; Execution phase: The optimization action selection is sent to the distribution network environment. After receiving the power action, the distribution network performs power flow calculation based on the linear power flow approximation method of Laurent series to obtain the optimized effect of the action, namely the line loss and the corresponding node voltage.

Images