Power distribution network optimal dispatching method and system based on multi-target deep reinforcement learning

Abstract

The application discloses a power distribution network optimal scheduling method and system based on multi-target deep reinforcement learning, relates to the technical field of optimal operation of power distribution networks, and comprises the following steps: a power distribution network operation model and a probability power flow model considering the uncertainty of photovoltaic output and load demand are established, probability power flow calculation under uncertainty is completed, and the probability density function of power distribution network node voltage and line flow is obtained through analysis; a utility function based on preference is introduced, a voltage out-of-limit and line overload risk index considering severity weight is established, and a risk-economic collaborative power distribution network multi-target optimal operation problem is constructed; the decision-making process of the problem is modeled as a multi-target Markov decision-making process, a multi-target deep reinforcement learning algorithm based on decomposition is adopted to learn and train the decision-making process of a reinforcement learning intelligent agent, a Pareto strategy set is obtained, an optimal operation strategy is screened out, the power distribution network is optimized and controlled, and collaborative optimization of power distribution network system operation risk and cost is realized.

Description

Technical Field

[0001] This invention relates to the field of distribution network optimization operation technology, and in particular to a distribution network optimization scheduling method and system based on multi-objective deep reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the increasing demand for grid-connected renewable energy, especially solar power, there is a need to accelerate the transformation of traditional distribution networks into active distribution networks with bidirectional power flow. However, a high proportion of photovoltaic grid connection poses technical challenges to the safe operation of the system, particularly overvoltage and line reverse overload issues. In actual operation, photovoltaic inverters and energy storage systems in the distribution network can provide rapid reactive / active power support, but their frequent participation in regulation often increases operating costs and may compress arbitrage profits from energy storage. Therefore, balancing the safety and economy of system operation has become an important direction in current research on distribution network optimization and scheduling.

[0004] Traditional distribution network optimization and scheduling strategies are mostly based on deterministic information about the distribution network, without considering the risks brought by uncertainties such as distributed photovoltaic power and loads, and their increased complexity to grid operation and scheduling. Although some distribution network optimization and scheduling strategies that consider uncertainties on both the source and load sides have been proposed, existing strategies mainly focus on single-objective optimization based on operating costs and convert risks into penalty factors in the objective function to balance system security and economy. However, the setting of penalty factors is subjective and it is difficult to achieve a globally optimal trade-off between operating risks and costs. Existing strategies also use multi-objective optimization algorithms to handle the dual risk-economic objectives of distribution network scheduling, but these algorithms are prone to problems such as incomplete Pareto front coverage and insufficient solution diversity, and it is difficult to balance the convergence and distribution of solutions. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this invention provides a distribution network optimization scheduling method and system based on multi-objective deep reinforcement learning. By intelligently sensing the operational risks of the distribution network under uncertain conditions, and using a multi-objective deep reinforcement learning algorithm to train and generate diverse risk-economic trade-off strategies, the knee-point strategy is selected to simultaneously minimize operational risks and costs, thereby achieving synergistic optimization of the operational risks and costs of the distribution network system, and taking into account both the safety and economy of distribution network optimization scheduling.

[0006] In a first aspect, the present invention provides a distribution network optimization scheduling method based on multi-objective deep reinforcement learning.

[0007] A distribution network optimization scheduling method based on multi-objective deep reinforcement learning includes:

[0008] Establish a distribution network operation model and a probabilistic power flow model that consider the uncertainties of photovoltaic output and load demand, complete the probabilistic power flow calculation under uncertainty, and analyze the probability density functions of distribution network node voltage and line power flow.

[0009] Based on the probability density function, a preference-based utility function is introduced to establish voltage overrun and line overload risk indicators that take severity weights into account. Combined with the distribution network operation model, a risk-economic synergy multi-objective optimization operation problem of the distribution network is constructed.

[0010] The decision-making process of the multi-objective optimization operation problem of the distribution network is modeled as a multi-objective Markov decision process. A decomposition-based multi-objective deep reinforcement learning algorithm is used to learn and train the reinforcement learning agent to obtain the Pareto policy set.

[0011] The optimal operating strategy selected from the Pareto strategy set is used to optimize and control the distribution network.

[0012] Further technical solutions include the following: the distribution network operation model and probabilistic power flow model: photovoltaic power generation uncertainty model, photovoltaic inverter adjustable non-functional capacity model, energy storage system model, load uncertainty model, probabilistic power flow model based on semi-invariant method, and Gram-Charlier series expansion model.

[0013] The construction of the distribution network operation model and probabilistic power flow model includes:

[0014] Based on the Beta distribution characteristics of light intensity, a photovoltaic power generation uncertainty model is constructed to simulate the randomness of photovoltaic output power.

[0015] Based on the maximum active power output of photovoltaics, the adjustable reactive power of the inverter is calculated, and an adjustable reactive power model of the photovoltaic inverter is constructed.

[0016] Based on the constraints of energy storage charging and discharging power, rated power, state of charge, and charging and discharging efficiency in the distribution network, an energy storage system model is constructed.

[0017] A load uncertainty model is constructed by simulating the time-varying characteristics of distribution network load using the normal distribution method.

[0018] Based on the AC nonlinear power flow model of the distribution network, considering the uncertainties of photovoltaic and load, the correlation between node injected power disturbance and node voltage and line power flow changes is established. The node injected power disturbance is characterized by semi-invariants, and the semi-invariants of node voltage state and line power flow state are calculated. A probabilistic power flow model based on the semi-invariant method is constructed.

[0019] The Gram-Charlier series expansion is performed on the probabilistic power flow model based on the semi-invariant method. The semi-invariant calculation results are transformed into probability density functions of node voltage and line power flow, resulting in the Gram-Charlier series expansion model.

[0020] A further technical solution involves constructing a risk-economic synergistic multi-objective optimization operation problem for distribution networks. The process is as follows:

[0021] Based on the risk-preference utility function, and combined with the probability density function of distribution network node voltage and line power flow, voltage over-limit risk index and line overload risk index considering severity weight are constructed.

[0022] Based on voltage over-limit risk indicators and line overload risk indicators, a distribution network operation risk is constructed. With the minimization of distribution network operation risk and operation cost as the objective function, and combined with constraints, a risk-economic synergistic multi-objective optimization operation problem of the distribution network is constructed. Among them, the constraints are based on the distribution network operation model and include inverter reactive power output capacity constraints, energy storage operation constraints, and distribution network operation constraints.

[0023] A further technical solution is that the voltage over-limit risk index is obtained by calculating the weighted risk value of all node voltages exceeding the upper and lower limits. The voltage over-limit risk value is calculated based on the probability density function of the node voltage and the voltage over-limit severity function. The voltage over-limit severity function is represented by a risk-preference utility function of the over-limit ratio.

[0024] The line overload risk index is obtained by calculating the weighted risk value of all lines exceeding the power limit. The line overload risk value is calculated based on the probability density function of line power flow and the line overload severity function. The line overload severity function is represented by a risk-preference utility function of the overload ratio.

[0025] A further technical solution models the decision-making process of the multi-objective optimization operation problem of the distribution network as a multi-objective Markov decision process, which consists of a quintuple. It means that, among them:

[0026] It is a state set, consisting of the probability distribution parameters of photovoltaic and load at each node, the state of charge of energy storage, and the time-of-use electricity price; This is a set of actions, including reactive power output commands for photovoltaic inverters and charging / discharging power commands for energy storage systems. State transition probability, used to describe the probability of evolving to the next state given the current state and control action; The reward set is in vector form, constructed based on the objective function, defined as the inverse of the operating risk and operating cost, and then normalized. This is a discount factor used to balance current rewards and future long-term rewards.

[0027] A further technical solution employs a decomposition-based multi-objective deep reinforcement learning algorithm to learn and train the decision-making process of the reinforcement learning agent, obtaining a Pareto policy set, including:

[0028] Define a set of weight vectors that satisfy the normalization constraints, where each set of weight vectors corresponds to a single-objective subproblem;

[0029] By using a scalarization function to map vector rewards to scalar rewards, a multi-objective problem can be decomposed into a set of single-objective sub-problems.

[0030] A policy population consisting of K policies and an external Pareto archive are constructed. K reinforcement learning agents are set up for parallel training, each agent corresponding to a set of weight vectors. An off-policy deep reinforcement learning algorithm is used to train the agents in parallel. During the training process, the performance of the learned policies is periodically evaluated, and dominated policies are removed based on Pareto dominance. Non-dominated policies are stored in the external Pareto archive and continuously updated. The off-policy deep reinforcement learning algorithm can be any one of the SAC algorithm, DDPG algorithm, or TD3 algorithm.

[0031] After training, all non-dominated strategies are extracted from the external Pareto archive to obtain the Pareto policy set.

[0032] A further technical solution involves a reinforcement learning agent that selects the knee-point strategy from the Pareto strategy set as the optimal operating strategy to regulate the photovoltaic inverters and energy storage systems in the distribution network. The knee-point strategy is the inflection point of the Pareto front and represents the optimal trade-off between risk and cost.

[0033] Secondly, this invention provides a power distribution network optimization scheduling system based on multi-objective deep reinforcement learning.

[0034] A distribution network optimization scheduling system based on multi-objective deep reinforcement learning includes:

[0035] The distribution network model building module is used to establish a distribution network operation model and a probabilistic power flow model that considers the uncertainty of photovoltaic output and load demand, complete the probabilistic power flow calculation under uncertainty, and analyze the probability density functions of distribution network node voltage and line power flow.

[0036] The multi-objective optimization operation problem construction module is used to establish voltage over-limit and line overload risk indicators that take severity weights into account, based on the probability density function and by introducing a preference-based utility function. Combined with the distribution network operation model, it constructs a risk-economic synergistic multi-objective optimization operation problem for the distribution network.

[0037] The optimization strategy solution module is used to model the decision-making process of the multi-objective optimization operation problem of the distribution network as a multi-objective Markov decision process. It adopts a decomposition-based multi-objective deep reinforcement learning algorithm to learn and train the reinforcement learning agent to obtain the Pareto policy set.

[0038] The control module is used to optimize and control the distribution network based on the optimal operating strategy selected from the Pareto strategy set.

[0039] Thirdly, the present invention also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-described distribution network optimization scheduling method based on multi-objective deep reinforcement learning when executing the executable instructions stored in the memory.

[0040] Fourthly, the present invention also provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-described distribution network optimization scheduling method based on multi-objective deep reinforcement learning.

[0041] The above one or more technical solutions have the following beneficial effects:

[0042] This invention provides a distribution network optimization scheduling method and system based on multi-objective deep reinforcement learning, which can provide a balanced strategy that takes into account both safety and economy for real-time optimization operation of distribution networks under uncertain conditions. By establishing a multi-dimensional model considering the uncertainty of photovoltaic output and load demand, this invention transforms uncertainty into computable probabilistic data, solving the problem that existing technologies do not fully consider uncertainty risks. It can accurately quantify the risks of voltage exceedances and line overloads at distribution network nodes. By introducing a preference-based utility function and severity weights to construct risk indicators, it achieves accurate quantitative assessment of the risks of voltage exceedances and line overloads of different degrees caused by photovoltaic and load uncertainties in the distribution network, providing a scientific basis for the quantification of safety objectives. It can intelligently perceive the operational risks of the distribution network under uncertain conditions and achieve a trade-off between operational risks and operating costs. Using a decomposition-based multi-objective deep reinforcement learning algorithm, the risk-economic co-optimization problem is decomposed into multiple single-objective sub-problems for parallel solution. The resulting Pareto strategy set can provide diverse risk-economic trade-off schemes to meet the needs of different operating scenarios, especially the knee strategy, which can achieve the optimal balance between risk and cost. The entire process of this invention is clear and can be used interactively with actual distribution networks. The reinforcement learning agent can quickly output optimized control commands based on the real-time status of the distribution network. It is suitable for real-time optimized operation of distribution networks under uncertain conditions, realizing the coordinated optimization of the operation risk and cost of the distribution network system, and taking into account both the safety and economy of the optimized scheduling of the distribution network.

[0043] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0045] Figure 1 This is a flowchart illustrating the power distribution network optimization scheduling method based on multi-objective deep reinforcement learning in Embodiment 1 of the present invention.

[0046] Figure 2 This is a schematic diagram of the power distribution network structure in Embodiment 1 of the present invention;

[0047] Figure 3 This is a schematic diagram of the Pareto front obtained by training a decomposition-based multi-objective deep reinforcement learning algorithm in Embodiment 1 of the present invention.

[0048] Figure 4 This is a schematic diagram illustrating the risk of distribution network voltage exceeding limits under different preferred operating strategies in Embodiment 1 of the present invention.

[0049] Figure 5 This is a schematic diagram illustrating the overload risk of distribution network lines under different preferred operating strategies in Embodiment 1 of the present invention. Detailed Implementation

[0050] It should be noted that the following detailed descriptions are exemplary and are intended only to describe specific embodiments and to provide further explanation of the invention, and are not intended to limit the scope of exemplary embodiments of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0051] Example 1

[0052] This embodiment provides a distribution network optimization scheduling method based on multi-objective deep reinforcement learning, such as... Figure 1 As shown, the specific steps include:

[0053] Step S1: Establish a distribution network operation model and a probabilistic power flow model that consider the uncertainty of photovoltaic output and load demand, complete the probabilistic power flow calculation under uncertainty, and analyze the probability density functions of distribution network node voltage and line power flow.

[0054] Step S2: Based on the probability density function, introduce a preference-based utility function, establish voltage over-limit and line overload risk indicators that consider severity weights, and combine them with the distribution network operation model to construct a risk-economic synergistic multi-objective optimization operation problem for the distribution network.

[0055] Step S3: Model the decision-making process of the multi-objective optimization operation problem of the distribution network as a multi-objective Markov decision process, and use a decomposition-based multi-objective deep reinforcement learning algorithm to learn and train the reinforcement learning agent to obtain the Pareto policy set.

[0056] Step S4: Optimize and regulate the distribution network according to the optimal operating strategy selected from the Pareto strategy set.

[0057] The following content will provide a more detailed introduction to the power distribution network optimization scheduling method based on multi-objective deep reinforcement learning proposed in this embodiment.

[0058] In step S1, the established distribution network operation model and probabilistic power flow model include: a photovoltaic power generation uncertainty model, a photovoltaic inverter adjustable non-functional capacity model, an energy storage system model, a load uncertainty model, a probabilistic power flow model based on the semi-invariant method, and a Gram-Charlier series expansion model. By establishing these models, probabilistic power flow calculations under uncertainties in photovoltaic output and load demand can be achieved, and the probability density functions of distribution network node voltages and line power flows can be analytically obtained. The construction of the above models specifically includes:

[0059] Constructing an uncertainty model for photovoltaic (PV) power generation: PV output power is closely related to light intensity, and since light intensity is random, PV output power is also random. According to statistics, within a certain time period... Inside, the light intensity can be approximated by a Beta distribution, and its probability density function can be expressed as:

[0060] (1)

[0061] In the above formula, and ( ) are nodes Place The actual and maximum light intensity during the time period; , All of these are shape parameters of the Beta distribution.

[0062] Photovoltaic power output The relationship between light intensity and illumination intensity is shown in the following formula:

[0063] (2)

[0064] In the above formula, For nodes Place Photovoltaic output power during the time period; For the irradiated area, This refers to the photoelectric conversion efficiency.

[0065] Furthermore, based on the Beta distribution characteristics of light intensity, the probability density function of light intensity is determined. Knowing this probability density function, the probability density function of photovoltaic output can be calculated using formula (2), which also exhibits a Beta distribution and can be expressed as:

[0066] (3)

[0067] In the above formula, This represents the maximum output power of the photovoltaic system.

[0068] Constructing an adjustable reactive power model for a photovoltaic inverter: Since photovoltaic output power exhibits randomness and uncertainty, the adjustable reactive power of the inverter is calculated based on the maximum active power output of the photovoltaic system (considering the uncertainty of photovoltaic power, also known as the maximum possible active power output of the photovoltaic system), as shown in the following formula:

[0069] (4)

[0070] In the above formula, For nodes The inverter has adjustable reactive power; The apparent power of the inverter; For photovoltaics The maximum active power output during the specified time period.

[0071] Constructing an energy storage system model: Based on the constraints of energy storage charging and discharging power, rated power, state of charge, and charging and discharging efficiency in the distribution network, it can be represented as follows:

[0072] (5)

[0073] (6)

[0074] (7)

[0075] In the above formula, For nodes The charging and discharging power of the energy storage system; This refers to the rated power of the energy storage system. The state of charge of the energy storage system; This refers to the rated capacity of the energy storage system. , These are the charging and discharging efficiencies of the energy storage system, respectively.

[0076] Constructing a load uncertainty model: Distribution network loads exhibit time-varying and uncertain characteristics. The widely used normal distribution model is employed to simulate the time-varying patterns and uncertainties of the load, which can be expressed as:

[0077] (8)

[0078] (9)

[0079] In the above formula, For nodes Active power of load; For nodes Reactive power of load; , and , These are the mean and standard deviation of the active and reactive power of the load, respectively.

[0080] Constructing a probabilistic power flow model based on the semi-invariant method: Based on the existing AC nonlinear power flow model of the power system, considering the uncertainties of photovoltaic power generation and load (i.e., introducing the photovoltaic power generation uncertainty model and load uncertainty model constructed above), for any time... By utilizing the voltage sensitivity relationship, linearized equations for node voltage, line power flow state variables, and node injected power are established, which can be expressed as:

[0081] (10)

[0082] In the above formula, , These represent the baseline expected values ​​of the injected active power and reactive power at the nodes, respectively. , These represent the reference expected value vectors for the active power and reactive power transmitted through the line, respectively. , These represent the random perturbation vectors of active and reactive power injected into the nodes, respectively. , These represent the random disturbance vectors representing the active and reactive power transmitted through the line, respectively. , These represent the reference expected value vectors for node voltage and phase angle, respectively; , These represent the vectors of changes in node voltage and phase angle, respectively. , The power flow equations represent the node-injected power and the line-transmitted power, respectively. This represents the Jacobian matrix after iterative convergence in deterministic power flow calculation; Represents the sensitivity matrix, by Calculated.

[0083] Next, the semi-invariants of each order of the nodal injected power random variable are calculated using the central moment theory. These semi-invariants are then substituted into formula (10) as the random disturbance values ​​of the nodal injected power. Convolution is then performed to calculate the semi-invariants of each order of the nodal voltage state variable and the line power flow state variable, which can be expressed as:

[0084] (11)

[0085] In the above formula, Indicates the first Order-semi-invariants; Representing matrix elements Power of 1.

[0086] Constructing the Gram-Charlier series expansion model: Using the semi-invariants of the node voltage and line power flow state variables obtained through the above-mentioned probabilistic power flow calculation method based on semi-invariants, the probability density functions corresponding to the node voltage and line power flow can be obtained through Gram-Charlier series expansion:

[0087] (12)

[0088] In the above formula, The probability density function is the standard normal distribution. express of The result of the derivative of the order; For random variables of Semi-invariant of order.

[0089] The above model enables probabilistic power flow calculation under uncertainties in photovoltaic output and load demand, and analytically obtains the probability density functions of distribution network node voltage and line power flow.

[0090] In step S2, a preference-based utility function is introduced to construct a risk preference-based utility function for voltage overrun and line overload. Combined with the probability density functions of distribution network node voltage and line power flow obtained in the previous step, a voltage overrun risk index and a line overload risk index considering severity weights are constructed.

[0091] Specifically, the voltage over-limit risk index is obtained by calculating the weighted risk value of all node voltages exceeding the upper and lower limits, and can be expressed as:

[0092] (13)

[0093] In the above formula, and These represent the number of nodes that exceed the upper and lower voltage limits, respectively. and These are the upper and lower limits of the voltage, respectively. and The first Nodes Voltage at any moment The corresponding probability density function and over-limit severity function are used to characterize the voltage over-limit risk value based on the voltage probability density function and voltage over-limit severity function of this node.

[0094] The voltage exceedance severity function is represented by a risk-preference utility function based on the exceedance ratio, which can be expressed as:

[0095] (14)

[0096] The line overload risk index is obtained by calculating the weighted risk value of all lines exceeding their power limits, and can be expressed as:

[0097] (15)

[0098] In the above formula, and The first Line The probability density function and overload severity function of the active power transmitted by the line at any given time are used to characterize the overload risk value of the line based on the probability density function of the power flow and the overload severity function of the line. The number of lines whose active power exceeds the upper limit; This represents the upper limit of active power transmitted through the line.

[0099] The line overload severity function is represented by a risk-preference utility function based on the proportion of overload. It can be expressed as:

[0100] (16)

[0101] Furthermore, based on voltage over-limit risk indicators and line overload risk indicators, distribution network operation risks are constructed. With the minimization of distribution network operation risks and operating costs as objective functions, and combined with constraints, a risk-economic synergistic multi-objective optimization operation problem for distribution networks is constructed.

[0102] In this embodiment, the objective function of the risk-economic synergy multi-objective optimization operation problem of the distribution network is:

[0103] (17)

[0104] in, and Representing operational risks and operational costs respectively, as follows:

[0105] (18)

[0106] (19)

[0107] In the above formula, Indicates the current time period; For the entire operating cycle; This represents the node number; the system has a total of [number missing]. One node; This indicates an indicator of voltage over-limit risk. The indicators represent the line overload risk indicators; the specific calculation methods for each risk indicator are shown in (13)-(16); This indicates the reactive power cost of the inverter; This indicates the reactive power output of the inverter; This indicates the cost of energy storage charging and discharging operations; Indicates the charging and discharging power of the energy storage system. Indicates charging status. Indicates the discharge state; It is a time-of-use electricity price.

[0108] In addition, the constraints of the risk-economic synergistic multi-objective optimization operation problem of distribution networks include each time period. The decision variables must simultaneously satisfy network and equipment operation constraints, specifically including: inverter reactive power output capacity constraints, given by formula (4); energy storage operation constraints, described by formulas (5)-(7); and distribution network operation constraints, such as nodes without photovoltaic or energy storage being constrained by formulas (20) and (21), which can be expressed as:

[0109] (20)

[0110] (twenty one)

[0111] in, This represents the set of nodes configured with photovoltaics. This represents the set of nodes configured for energy storage.

[0112] Meanwhile, in order to assess the risk targets, a probabilistic power flow method based on semi-invariants is adopted. The relationship between node-injected disturbances and state variables such as voltage and line power flow is established through formula (10), semi-invariant transmission is achieved using formula (11), and the Gram-Charlier series expansion of formula (12) is used to approximate the probability density function, thereby calculating the risk indicators of voltage over-limit and line overload.

[0113] In step S3, the risk-economic collaborative multi-objective optimization decision-making process of the distribution network is modeled as a multi-objective Markov decision process. A decomposition-based multi-objective deep reinforcement learning algorithm is used to learn and train the reinforcement learning agent to obtain a Pareto policy set. The multi-objective Markov decision process can be represented by the following quintuple:

[0114] (twenty two)

[0115] In the above formula, For a set of states; For action sets; The state transition probability; The reward set is in vector form; This is the discount factor. Compared to a single-objective Markov decision process, a multi-objective Markov decision process replaces the scalar reward function with... 3D vector reward function .

[0116] In the context of this embodiment, the power distribution network (i.e., the environment) structure meets the system operation constraints. The specific definitions of each element in the multi-objective Markov decision process are as follows:

[0117] ① State space: in any During a given time period, the state space consists of the probability distribution parameters of photovoltaic and load at each node, the state of charge (SOC) of energy storage, and the time-of-use electricity price, and can be represented as:

[0118] (twenty three)

[0119] ② Action Space: The actions of the intelligent agent include reactive power output commands from the photovoltaic inverter and charging / discharging power commands from the energy storage system. Furthermore, to satisfy the constraints, i.e., formulas (4)-(7), a scaling factor is introduced. and The reactive power output of the inverter and the charging and discharging power of the energy storage system can be expressed as:

[0120] (twenty four)

[0121] (25)

[0122] in, and This can be further expressed as:

[0123] (26)

[0124] (27)

[0125] It is important to note that scaling the output can also avoid problems such as vanishing or exploding gradients, preventing these issues from hindering the learning process during training.

[0126] ③ State transition probability: The state transition probability describes the probability that the system will evolve to the next state given the current state and control actions. It is defined as follows:

[0127] (28)

[0128] in, and Representing time steps Random variables of state and action, , and This indicates its corresponding implementation value.

[0129] ④ Vector Reward Function: Combining the objective function, i.e., formula (17), a multi-objective instantaneous reward vector is constructed to minimize the operational risks and costs of the distribution network. Since this objective is the opposite of the reward mechanism in reinforcement learning, the vector reward function is defined as the inverse of the objective function and, after standardization, can be expressed as:

[0130] (29)

[0131] in, and It is the standardized objective function, expressed as:

[0132] (30)

[0133] In the above formula, and It serves as a benchmark for operational risk and cost. A standardized reward function can mitigate the problem of orders-of-magnitude differences between different objectives, thereby improving the efficiency of finding the Pareto frontier.

[0134] ⑤ Discount Factor It is used to balance current rewards and future long-term rewards.

[0135] Based on the above definition, the objective of this multi-objective Markov decision process is to find a strategy that maximizes the cumulative vector reward. The goal is:

[0136] (31)

[0137] In the context of multiple objectives, if and only if It is no worse than another strategy across all target dimensions and is better than it in at least one dimension. of At that time, it is called a strategy Pareto Domination Strategy The goal of this embodiment is to learn a set of Pareto optimal operating strategies, providing multiple trade-offs between operating risk and operating cost.

[0138] This embodiment employs a decomposition-based multi-objective deep reinforcement learning algorithm to solve the problem and obtain a Pareto policy set. This algorithm maps vector rewards to scalar rewards using a scalarization function, decomposing a multi-objective problem into a set of single-objective sub-problems for parallel solution. Specifically, a set of weight vectors is defined. Furthermore, the weights satisfy the normalization constraint, and each set of weight vectors corresponds to a single-objective subproblem, which is represented by the following scalarization method:

[0139] (33)

[0140] In the above formula, Index for sub-problems / strategies.

[0141] Building upon this, in decomposition-based multi-objective deep reinforcement learning algorithms, a system is constructed and maintained consisting of... A strategy population consisting of 1 strategy and external Pareto archives In order to achieve multiple trade-off solutions simultaneously, set The system employs several parallel-trained reinforcement learning agents, each corresponding to a pre-defined set of weight vectors, to approximate different regions of the Pareto front in parallel during a single training session. Any off-policy deep reinforcement learning algorithm can be used during training, such as SAC (Soft Actor-Critic), DDPG (Deep Deterministic Policy Gradient), TD3 (Twin Delayed Deep Deterministic Policy Gradient), etc. For specific off-policy deep reinforcement learning algorithms, please refer to existing training algorithms; details will not be elaborated here.

[0142] Furthermore, during training, all learned preference-specific policies are periodically evaluated in the original target space to obtain a cumulative performance vector. Based on this performance vector and the Pareto dominance relation, dominated solutions are eliminated, and the remaining non-dominated strategies are stored in an external archive. This archive is continuously updated throughout the training process.

[0143] After the above training was completed, through... Non-dominated policies are extracted to obtain the final Pareto policy set.

[0144] The risk-economic collaborative optimization operation model of distribution network based on multi-objective deep reinforcement learning, constructed through the above methods and steps, can be used interactively with the actual distribution network. By analyzing the state variables transmitted to the distribution network in real time, it can perceive the operational risks of the distribution network under uncertain conditions and realize a distribution network optimization operation strategy that takes into account both system operation risks and operating costs.

[0145] In step S4, the reinforcement learning agent optimizes and controls the distribution network based on the training-obtained distribution network optimization operation strategy. Specifically, the reinforcement learning agent filters or selects the knee strategy from the Pareto strategy set obtained in step S3 as the optimal operation strategy. This knee strategy is... Figure 3 The inflection point of the Pareto front, indicated by the middle arrow, represents the optimal trade-off strategy for risk-cost equilibrium. Based on this optimal operating strategy, the photovoltaic inverters and energy storage systems in the distribution network are regulated to perceive the operating risks of the distribution network under uncertain conditions and achieve optimized operation of the distribution network that simultaneously considers system operating risks and operating costs.

[0146] Furthermore, to verify the effectiveness and superiority of the distribution network optimization scheduling method based on multi-objective deep reinforcement learning provided in this embodiment, this embodiment also... Figure 2 The power distribution network shown is subjected to operation optimization simulation. Figure 2 The distribution network shown has 34 nodes, with 6 distributed photovoltaic (PV) systems connected (nodes 6, 9, 16, 17, 26, and 29) and 4 energy storage systems connected (nodes 3, 8, 16, and 23). The aforementioned risk-economic collaborative optimization operation / scheduling method for distribution networks based on multi-objective deep reinforcement learning is used to solve the problem. Considering that the form and coverage of the Pareto policy set are affected by the off-policy training algorithm, three methods—SAC, DDPG, and TD3—are selected to train the reinforcement learning agent, resulting in the following... Figure 3 The Pareto front is shown. (By...) Figure 3 It is evident that the Pareto front exhibits a clear risk-economic trade-off: when the weights favor risk control, the strategy tends to more aggressively utilize inverter reactive power regulation and energy storage charging and discharging to mitigate risk, thereby increasing regulation costs and compressing energy storage arbitrage opportunities, resulting in a corresponding decrease in economic indicators; conversely, when the weights favor economic efficiency, control actions become more conservative, economic returns increase, but risk mitigation capabilities weaken. Among these, the knee-point strategy can simultaneously minimize both distribution network operational risks and operating costs.

[0147] Based on the Pareto policy set obtained above, the reinforcement learning agent selects the knee-point policy to optimize and control the power distribution network. Figure 4 andFigure 5 This paper demonstrates how reinforcement learning agents, under risk-preference strategies (considering only operational risk), economic-preference strategies (considering only operational costs), and knee-point strategies, address voltage exceedance and line overload risks after distribution network regulation. Figure 4 and Figure 5 It can be seen that the risk-preference strategy can significantly reduce system operation risk, demonstrating the strongest safety margin; the knee-point strategy has a risk level between the risk-preference and economic preference strategies, maintaining low risk while avoiding the economic sacrifice caused by over-adjustment; although the economic preference strategy obtains the highest energy storage arbitrage profit, energy storage still maintains a large discharge power during the gradual ramp-up of photovoltaic power, exacerbating local voltage rise and thus amplifying the risk of voltage exceedance. The method proposed in this embodiment can provide a balanced strategy that takes into account both safety and economy for real-time optimization operation of distribution networks under uncertain conditions.

[0148] Example 2

[0149] This embodiment provides a distribution network optimization scheduling system based on multi-objective deep reinforcement learning, the system comprising:

[0150] The distribution network model building module is used to establish a distribution network operation model and a probabilistic power flow model that considers the uncertainty of photovoltaic output and load demand, complete the probabilistic power flow calculation under uncertainty, and analyze the probability density functions of distribution network node voltage and line power flow.

[0151] The multi-objective optimization operation problem construction module is used to establish voltage over-limit and line overload risk indicators that take severity weights into account, based on the probability density function and by introducing a preference-based utility function. Combined with the distribution network operation model, it constructs a risk-economic synergistic multi-objective optimization operation problem for the distribution network.

[0152] The optimization strategy solution module is used to model the decision-making process of the multi-objective optimization operation problem of the distribution network as a multi-objective Markov decision process. It adopts a decomposition-based multi-objective deep reinforcement learning algorithm to learn and train the reinforcement learning agent to obtain the Pareto policy set.

[0153] The control module is used to optimize and control the distribution network based on the optimal operating strategy selected from the Pareto strategy set.

[0154] Example 3

[0155] This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the method provided in this embodiment.

[0156] Example 4

[0157] This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, will cause the processor to execute the method described above in this embodiment.

[0158] The steps and methods involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0159] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0160] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention has been described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A distribution network optimization scheduling method based on multi-objective deep reinforcement learning, characterized in that, include: Establish a distribution network operation model and a probabilistic power flow model that consider the uncertainties of photovoltaic output and load demand, complete the probabilistic power flow calculation under uncertainty, and analyze the probability density functions of distribution network node voltage and line power flow. Based on the probability density function, a preference-based utility function is introduced to establish voltage over-limit and line overload risk indicators that take severity weights into account. Combined with the distribution network operation model, a risk-economic synergy multi-objective optimization operation problem of the distribution network is constructed. The decision-making process of the multi-objective optimization operation problem of the distribution network is modeled as a multi-objective Markov decision process. A decomposition-based multi-objective deep reinforcement learning algorithm is used to learn and train the reinforcement learning agent to obtain the Pareto policy set. The distribution network is optimized and controlled based on the optimal operating strategy selected from the Pareto strategy set. Based on the risk-preference utility function, and combined with the probability density function of distribution network node voltage and line power flow, voltage over-limit risk index and line overload risk index considering severity weight are constructed. A multi-objective deep reinforcement learning algorithm based on decomposition is used to learn and train the decision-making process of the reinforcement learning agent, resulting in a Pareto policy set, including: Define a set of weight vectors that satisfy the normalization constraints, where each set of weight vectors corresponds to a single-objective subproblem; By using a scalarization function to map vector rewards to scalar rewards, a multi-objective problem can be decomposed into a set of single-objective sub-problems. A policy population consisting of K policies and an external Pareto archive are constructed. K reinforcement learning agents are set up for parallel training, each agent corresponding to a set of weight vectors. An off-policy deep reinforcement learning algorithm is used to train the agents in parallel. During the training process, the performance of the learned policies is periodically evaluated, and dominated policies are removed based on Pareto dominance. Non-dominated policies are stored in the external Pareto archive and continuously updated. The off-policy deep reinforcement learning algorithm can be any one of the SAC algorithm, DDPG algorithm, or TD3 algorithm. After training, all non-dominated strategies are extracted from the external Pareto archive to obtain the Pareto policy set.

2. The distribution network optimization scheduling method based on multi-objective deep reinforcement learning as described in claim 1, characterized in that, The distribution network operation model and probabilistic power flow model include: photovoltaic power generation uncertainty model, photovoltaic inverter adjustable non-functional capacity model, energy storage system model, load uncertainty model, probabilistic power flow model based on semi-invariant method, and Gram-Charlier series expansion model; The construction of the distribution network operation model and probabilistic power flow model includes: Based on the Beta distribution characteristics of light intensity, a photovoltaic power generation uncertainty model is constructed to simulate the randomness of photovoltaic output power. Based on the maximum active power output of photovoltaics, the adjustable reactive power of the inverter is calculated, and an adjustable reactive power model of the photovoltaic inverter is constructed. Based on the constraints of energy storage charging and discharging power, rated power, state of charge, and charging and discharging efficiency in the distribution network, an energy storage system model is constructed. A load uncertainty model is constructed by simulating the time-varying characteristics of distribution network load using the normal distribution method. Based on the AC nonlinear power flow model of the distribution network, considering the uncertainties of photovoltaic and load, the correlation between node injected power disturbance and node voltage and line power flow changes is established. The node injected power disturbance is characterized by semi-invariants, and the semi-invariants of node voltage state and line power flow state are calculated. A probabilistic power flow model based on the semi-invariant method is constructed. The Gram-Charlier series expansion is performed on the probabilistic power flow model based on the semi-invariant method. The semi-invariant calculation results are transformed into probability density functions of node voltage and line power flow, resulting in the Gram-Charlier series expansion model.

3. The distribution network optimization scheduling method based on multi-objective deep reinforcement learning as described in claim 1, characterized in that, The process of constructing a risk-economic synergistic multi-objective optimization operation problem for a power distribution network is as follows: Based on voltage over-limit risk indicators and line overload risk indicators, a distribution network operation risk is constructed. With the minimization of distribution network operation risk and operation cost as the objective function, and combined with constraints, a risk-economic synergistic multi-objective optimization operation problem of the distribution network is constructed. Among them, the constraints are based on the distribution network operation model and include inverter reactive power output capacity constraints, energy storage operation constraints, and distribution network operation constraints.

4. The distribution network optimization scheduling method based on multi-objective deep reinforcement learning as described in claim 3, characterized in that, The voltage over-limit risk index is obtained by calculating the weighted risk value of all node voltages exceeding the upper and lower limits. The voltage over-limit risk value is calculated based on the probability density function of the node voltage and the voltage over-limit severity function. The voltage over-limit severity function is represented by a risk-preference utility function of the over-limit ratio. The line overload risk index is obtained by calculating the weighted risk value of all lines exceeding the power limit. The line overload risk value is calculated based on the probability density function of line power flow and the line overload severity function. The line overload severity function is represented by a risk-preference utility function of the overload ratio.

5. The distribution network optimization scheduling method based on multi-objective deep reinforcement learning as described in claim 1, characterized in that, The decision-making process of the multi-objective optimization operation problem of the distribution network is modeled as a multi-objective Markov decision process, which consists of a quintuple. It means that, among them: It is a state set, consisting of the probability distribution parameters of photovoltaic and load at each node, the state of charge of energy storage, and the time-of-use electricity price; This is a set of actions, including reactive power output commands for photovoltaic inverters and charging / discharging power commands for energy storage systems. State transition probability, used to describe the probability of evolving to the next state given the current state and control action; The reward set is in vector form, constructed based on the objective function, defined as the inverse of the operating risk and operating cost, and then normalized. This is a discount factor used to balance current rewards and future long-term rewards.

6. The distribution network optimization scheduling method based on multi-objective deep reinforcement learning as described in claim 1, characterized in that, The reinforcement learning agent selects the knee-point strategy from the Pareto strategy set as the optimal operating strategy to regulate the photovoltaic inverters and energy storage systems in the distribution network. The knee-point strategy is the inflection point of the Pareto front and is the optimal trade-off strategy between risk and cost.

7. A distribution network optimization scheduling system based on multi-objective deep reinforcement learning, employing the method described in claim 1, characterized in that, include: The distribution network model building module is used to establish a distribution network operation model and a probabilistic power flow model that considers the uncertainty of photovoltaic output and load demand, complete the probabilistic power flow calculation under uncertainty, and analyze the probability density functions of distribution network node voltage and line power flow. The multi-objective optimization operation problem construction module is used to establish voltage over-limit and line overload risk indicators that take severity weights into account, based on the probability density function and by introducing a preference-based utility function. Combined with the distribution network operation model, it constructs a risk-economic synergistic multi-objective optimization operation problem for the distribution network. The optimization strategy solution module is used to model the decision-making process of the multi-objective optimization operation problem of the distribution network as a multi-objective Markov decision process. It adopts a decomposition-based multi-objective deep reinforcement learning algorithm to learn and train the reinforcement learning agent to obtain the Pareto policy set. The control module is used to optimize and control the distribution network based on the optimal operating strategy selected from the Pareto strategy set.

8. An electronic device, characterized in that, include: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the power distribution network optimization scheduling method based on multi-objective deep reinforcement learning as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device stores executable instructions that, when executed by a processor, implement the power distribution network optimization scheduling method based on multi-objective deep reinforcement learning as described in any one of claims 1-6.

Images