Power distribution network multi-target reactive power optimization method and system based on deep reinforcement learning, and medium

Abstract

The invention provides a power distribution network multi-target reactive power optimization method and system based on deep reinforcement learning, and a medium. The method comprises the following steps: S1, constructing a multi-target optimization model considering voltage deviation and comprehensive operation cost; s2, converting the optimization model in the field of the power distribution network into a multi-target Markov decision process; s3, improving a depth Q network algorithm to a multi-target condition, and proposing a multi-target depth Q network algorithm; s4, improving a soft actor commentator algorithm to a multi-target soft actor commentator algorithm, and providing a multi-target soft actor commentator algorithm; and S5, carrying out reward function normalization, and carrying out solution by applying improved multi-target deep reinforcement learning based on a multi-target deep Q network and a multi-target soft actor commentator. According to the method, the voltage stability of the high-proportion new energy power distribution network can be remarkably improved, and optimal scheduling is realized by coordinating various resources in the power distribution network.

Description

Technical Field

[0001] This invention relates to the field of voltage control technology for distribution networks, specifically a multi-objective reactive power optimization method, system, and medium for distribution networks based on deep reinforcement learning. Background Technology

[0002] With the accelerated pace of global energy structure transformation, the high proportion of renewable energy integration has become an inevitable trend in the development of modern power distribution networks. However, these energy sources are intermittent and unstable, especially with the integration of highly volatile emerging loads such as distributed photovoltaic (PV) power, which presents the distribution network with a series of challenges, including voltage exceeding limits and power fluctuations. For example, the output power of PV power generation is greatly affected by weather conditions, exhibiting significant diurnal variations, and even the fluctuations between sunny and rainy days can impact the stability of the distribution network. This volatility can lead to unstable grid voltage and even cause power system overload and equipment damage. Therefore, how to ensure stable grid operation through reasonable reactive power optimization has become a critical issue that urgently needs to be addressed in the power system field.

[0003] Currently, in the field of reactive power optimization research, traditional methods are mainly based on mathematical programming and heuristic algorithms. While mathematical programming methods can solve optimization problems to some extent, they are computationally inefficient when dealing with high-dimensional nonlinear problems and often exhibit significant limitations in dynamic scenarios. Heuristic algorithms, such as genetic algorithms and particle swarm optimization, can provide effective solutions to certain problems, but they lack adaptability to complex and highly dynamic power grid environments, making it difficult to obtain real-time and efficient optimization results in rapidly changing systems. Deep reinforcement learning (DRL) combines the advantages of neural networks and reinforcement learning, adapting to environmental fluctuations through interactive training, and is suitable for high-dimensional nonlinear optimization problems. However, most existing research focuses on single objectives and lacks systematic modeling and solutions for multi-objective collaborative optimization. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-objective reactive power optimization method, system and medium for distribution networks based on deep reinforcement learning. By constructing a multi-objective reactive power optimization model that takes into account both voltage stability and overall operating cost, and transforming it into a multi-objective Markov decision process, the improved multi-objective deep reinforcement learning based on multi-objective deep Q-network and multi-objective soft actor commentator is used to solve the problem, thereby improving the voltage stability of the distribution network, coordinating various resources in the distribution network, and achieving optimized scheduling.

[0005] A multi-objective reactive power optimization method for power distribution networks based on deep reinforcement learning includes the following steps:

[0006] S1: Construct a multi-objective optimization model that takes into account both voltage deviation and overall operating cost. The multi-objective optimization model outputs an objective function vector that includes the cumulative voltage deviation, system active power loss, and equipment operating cost.

[0007] S2: Based on the multi-objective optimization model in step S1, construct a multi-objective Markov decision process (MOMDP) ​​model. The MOMDP model uses the real-time operating state of the distribution network as the state space, the control strategies of photovoltaic (PV), static var generator (SVG), and capacitor bank (CB) as the action space, and designs the reward function with the objective function vector of step S1 as the core.

[0008] S3: For the discrete control variables in the action space in step S2, the deep Q network DQN algorithm is improved into a multi-objective deep Q network MODQN algorithm. The state-action-reward samples of the MOMDP model in step S2 are stored through the experience replay mechanism, and the discrete action decision is optimized based on the time-varying exploration rate.

[0009] S4: For the continuous control variables in the action space in step S2, the soft actor critic SAC algorithm is improved into the multi-objective soft actor critic MOSAC algorithm. Based on the state space and reward function of the MOMDP model in step S2, the policy entropy coefficient is dynamically adjusted through the maximum entropy theory to optimize the exploratory nature and robustness of continuous action decision-making.

[0010] S5: Normalize the reward function vector output in step S2 to eliminate dimensional differences, and then combine the MODQN algorithm in step S3 with the MOSAC algorithm in step S4 to solve the multi-objective deep reinforcement learning problem, and output coordinated control commands to achieve reactive power optimization of the distribution network.

[0011] Furthermore, in step S2, the MOMDP model is defined by a six-tuple (S, A, P, R, W, γ), where the state space S, action space A, and reward function R correspond to the input variables, decision variables, and objective function of the optimization model in step S1, respectively. The preference space W is used for weighted fusion of multi-objective rewards, and the state transition probability P represents the dynamic change of the distribution network state after the action is executed.

[0012] Furthermore, the MODQN algorithm in step S3 specifically includes: constructing a Q-value network input layer based on the MOMDP state space in step S2, using a discrete subset of actions in the action space as the output layer, and optimizing the network parameters by minimizing the loss function. The calculation of the loss function depends on the reward value and state transition data in the experience samples stored in step S2, and the discrete subset of actions includes the number of capacitor bank input groups.

[0013] Furthermore, the MOSAC algorithm in step S4 specifically includes: constructing a policy network based on the MOMDP state space and the continuous action subset (photovoltaic reactive power output, SVG compensation amount) in step S2; optimizing the network parameters by maximizing the weighted sum of cumulative reward and policy entropy; wherein the cumulative reward is calculated through the reward function in step S2; and the entropy coefficient is adjusted by dynamic gradient to balance exploration and utilization.

[0014] Furthermore, in step S5, the reward function normalization process specifically involves: based on the upper and lower limits of the reward function vector in step S2 (corresponding to the target values ​​when the preference weights are [1,0] and [0,1]), mapping the voltage deviation and the comprehensive operating cost reward value to the [0,1] interval through linear transformation, and using the normalization result as the input to the algorithms in steps S3 and S4.

[0015] Furthermore, the multi-objective deep reinforcement learning solution in step S5 specifically involves merging the discrete action decision output in step S3 with the continuous action decision output in step S4 into a complete control strategy, substituting it into the multi-objective optimization model in step S1 to verify its feasibility, and iteratively optimizing until the optimal reactive power control command covering the Pareto front is obtained.

[0016] Furthermore, the real-time operating status of the distribution network in step S2 includes photovoltaic active power output, load active power, and reactive power.

[0017] Furthermore, the discrete control variables in step S3 include the capacitor bank operation, and the continuous control variables in step S4 include photovoltaic and SVG output.

[0018] A multi-objective reactive power optimization system for power distribution networks based on deep reinforcement learning includes: a computer-readable storage medium and a processor;

[0019] The computer-readable storage medium is used to store executable instructions;

[0020] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the deep reinforcement learning-based multi-objective reactive power optimization method for power distribution networks.

[0021] A non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned multi-objective reactive power optimization method for power distribution networks based on deep reinforcement learning.

[0022] This invention offers the following advantages: By constructing a multi-objective optimization model that considers both voltage deviation and overall operating cost, and transforming it into a MOMDP model in the decision-making domain, it effectively overcomes the limitations of traditional methods in handling high-dimensional nonlinear problems. Furthermore, by employing the IMODRL algorithm based on MODQN and MOSAC to solve the multi-objective reactive power optimization model, it achieves hybrid control of discrete and continuous operation of distribution network resources, ensuring the safe and stable operation of the power grid. Attached Figure Description

[0023] Figure 1 This is a flowchart of a multi-objective reactive power optimization method for power distribution networks based on deep reinforcement learning, according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figure 1 This invention provides a multi-objective reactive power optimization method for power distribution networks based on deep reinforcement learning, comprising the following steps:

[0026] Step S1: Construct a multi-objective optimization model that takes into account both voltage deviation and overall operating cost;

[0027] Step S2: Transform the optimization model in the power distribution network field into a multi-objective Markov decision process (MOMDP).

[0028] Step S3: Improve the deep Q network (DQN) algorithm to the multi-objective case and propose the multi-objective deep Q network (MODQN) algorithm;

[0029] Step S4: Improve the soft actor critic (SAC) algorithm to the multi-objective case and propose the multi-objective soft actor critic (MOSAC) algorithm;

[0030] Step S5: Normalize the reward function and solve it using an improved multi-objective deep reinforcement learning method based on a multi-objective deep Q-network and a multi-objective soft actor critic.

[0031] Step S1 is as follows:

[0032] The multi-objective reactive power optimization scheduling model for power distribution networks proposed in this invention aims to utilize an improved multi-objective deep reinforcement learning (IMODRL) algorithm to provide reasonable reactive power control commands, and optimize the system's reactive power through the collaborative use of multiple resources, thereby ensuring the safe and stable operation of the power grid. The objective function f1 is shown in formula (1):

[0033] (1)

[0034] Where, N t It is the scheduling period, ΔU i It represents the voltage offset of all nodes in the system.

[0035] Considering the static var generator (SVG), capacitor bank (CB), and photovoltaic (PV) as controllable devices, the overall system operating cost mainly consists of the system's active power loss cost and the equipment's operating cost. The model's objective function f2 can be expressed as:

[0036] (2)

[0037] Where ρ1, ρ2, ρ3, and ρ4 represent the unit active power loss cost, PV unit power generation cost, SVG unit reactive power compensation cost, and CB operation cost, respectively, and N B Let N be the set of all nodes in the distribution network. SVG N CB and N PV These are the sets of nodes connected to SVG, CB, and PV, respectively. ij Q ij These represent the active and reactive power flowing through branch ij, respectively. i Let R be the voltage at node i. ij Let Q be the resistance between the lines at nodes i and j. SVG,q,t n CB,p,t and P PV,h,t Let q represent the reactive power emitted by the q-th SVG, the number of actions of the p-th CB, and the active power output of the h-th PV at time t, respectively, and Δt be the time step.

[0038] The constraints are as follows:

[0039] ① Node constraints

[0040] In a power distribution network, it is necessary to ensure the balance between supply and demand at each node, as shown below:

[0041] (3)

[0042] (4)

[0043] Among them, P i and Q i These represent the active and reactive power injected into node i, respectively, U j Let θ be the voltage at node j. ij G represents the voltage phase angle difference between node i and node j. ij and B ij Let be the conductance and susceptance of branch ij, respectively.

[0044] ② Voltage constraint

[0045] To prevent voltage overshoot at nodes, the distribution network should meet the constraints shown in equation (5).

[0046] (5)

[0047] Among them, U i It is the voltage at node i.

[0048] ③ Current constraints

[0049] (6)

[0050] (7)

[0051] in, These are the active and reactive power obtained from the upstream network at time t. These are the active and reactive power generated by the PV at node i at time t, respectively. These are the reactive power generated by SVG and CB at node i at time t, respectively. These are the active and reactive loads of node i at time t. It represents the active and reactive power losses of the network at time t.

[0052] ④PV constraint

[0053] (8)

[0054] (9)

[0055] Among them, P PV,t P is the active power of PV at time t.PV,max Q is the maximum access capacity of PV on node i. PV,i,t P PV,i,t These are the active and reactive power outputs of the PV at access node i at time t, S. PV,i It is the installed capacity of PV on node i.

[0056] ⑤SVG constraints

[0057] (10)

[0058] Among them, Q SVG,i,t Q is the reactive power emitted by the SVG of access node i at time t. SVG,i,max It is the maximum access capacity of the SVG on node i.

[0059] ⑥CB constraint

[0060] (11)

[0061] (12)

[0062] Where, x CB,i,t x is the number of CB groups at node i at time t. CB,i,max Q is the maximum number of CB groups at access node i. CB,i,t C is the reactive power of CB at access node i at time t. i It is the capacitance of a single capacitor bank at node i.

[0063] Step S2 is as follows:

[0064] MOMDP is an improved model based on the original Markov decision process (MDP). It extends MDP to handle multiple conflicting or synergistic objectives, and can be composed of a multi-objective vector with multiple objectives and multiple reward functions. It can be represented using a six-tuple as follows:

[0065] (13)

[0066] Where S is the system state space, and the system state at time t is S. t Let S be the action space, A be the reward function vector, and R be the scalar reward signal of each target, i.e., R = [R1 R2 ... R2 ... R3]. N ] T N is the number of objectives. P is the state transition probability function, representing the system environment in the current state S. t Below, the DRL agent executes action policy A. t Then proceed to the next state S t+1The probability distribution is given by W, which is the preference space consisting of a series of weight vectors ω. m The formula consists of m, which is the index of the preference vector in the preference space, and γ, which is a discount factor that ensures the later the reward is placed, the smaller its impact on the reward function, including the uncertainty of future rewards.

[0067] During the training of MOMDP with multiple agents, the decision model interacts with the network environment, learns strategies for different reward tasks, continuously adjusts its output actions based on the rewards obtained, and gradually learns the optimal decision strategy to cope with different environmental changes and task requirements.

[0068] This invention employs the advanced IMODRL method for multi-objective reactive power optimization scheduling of the distribution network. The constructed MOMDP model mainly includes a state space, an action space, and a reward function.

[0069] ①State Space

[0070] The system state space S represents the operating status of the distribution network, consisting of photovoltaic active power output, load active power, and reactive power, as shown below:

[0071] (14)

[0072] Among them, P L,t Q L,t These are the active and reactive power vectors of the load at time t, respectively. PV,t It is the photovoltaic active power vector at time t.

[0073] ② Action Space

[0074] The action space defines the set of actions that a DRL agent can execute, and consists of control policies for PV, SVG, and CB.

[0075] (15)

[0076] The corresponding control variable can be represented as:

[0077] (16)

[0078] (17)

[0079] (18)

[0080] Among them, A CB,i,t A SVG,i,t A PV,i,t Let be the control strategy vectors for each controllable device at time t. The above definition ensures that all control variables satisfy the constraints, reducing the number of infeasible solutions generated during the algorithm's exploration process and thus improving the algorithm's learning efficiency.

[0081] ③ Reward function

[0082] The reward function represents the feedback value obtained by the DRL agent after performing a specific action. The optimization problem of this invention can be expressed as:

[0083] (19)

[0084] Among them, R t f is the real-time reward at time t; 1,t f 2,t Let be the voltage deviation at time t and the overall operating cost, respectively. o is a vector composed of Boolean variables. If the action satisfies the system constraints, o = [0,0], otherwise o = [1,1]. r is the penalty vector for violating the constraints.

[0085] The MOMDP model constructed in this invention determines the current state S in each decision cycle. t Develop control strategy A t Next, the DRL agent interacts with the environment for training and learning, thereby obtaining the reward function R. t Training then proceeds to the next time step t+1, iterating the above process until the termination time T is reached. The model's objective is to maximize the cumulative reward, which is the expected value of the state-action relationship Q. π (s t ,a t ), represented as:

[0086] (20)

[0087] Where E is the mathematical expectation, s t It is a state vector, a t Is the DRL agent in state s t The action vector taken below, through f ω (R t )=ω T R t Generate different preference vectors ω m scalar reward, when ω m When fixed, the MOMDP simplifies to a standard MDP and can be solved using conventional DRL. Considering all possible preference weight vectors to solve the MOMDP model yields a series of corresponding rewards, resulting in a set of non-dominated solutions, known as the Pareto front. When it is impossible to enhance the expected reward of a particular objective through any policy without sacrificing the expected rewards of other objectives, the resulting policy π... * This is the Pareto optimal policy. Therefore, the core problem in solving the proposed multi-objective optimization scheduling model for optical storage is to obtain a policy network that can cover the Pareto front in the entire preference weight space as much as possible.

[0088] Step S3 is as follows:

[0089] The MODQN algorithm makes three key improvements over the traditional DQN algorithm: 1) a multi-objective Q-value network architecture design; 2) a reward fusion mechanism based on preference weights; and 3) a time-varying exploration rate optimization strategy. These improvements enable the algorithm to effectively handle the Pareto optimal decision problem for discrete capacitor banks.

[0090] As a deep reinforcement learning algorithm, DQN combines traditional Q-learning with deep neural networks to solve reinforcement learning problems with high-dimensional discrete state spaces. By approximating the Q-value function using a neural network, the algorithm can learn and make decisions in complex environments. Furthermore, DQN employs an experience replay technique to reduce the correlation between training data, enhancing stability, improving learning efficiency, and accelerating the convergence process. Its main idea is to store the agent's experiential information during training in an experience pool and randomly select a batch of experiences from the pool for training. The input is the current action policy A of the DRL agent. t A sample data {S} can be obtained t A t , R t , S t+1 , ω m The data is stored in the experience pool. During training, the agent samples and updates the Q-network according to a random policy. The goal of DQN is to optimize the loss function L. Q (θ) is optimized and expressed as:

[0091] (twenty one)

[0092] Where Q(s) t ,a t ;θ) is the Q value estimated based on the network parameter θ, r t +γ×maxQ(s t+1 ,a t+1 ;θ`) represents the Q value estimated based on the previous network parameters θ`.

[0093] When dealing with discrete decision variables, the DQN agent selects the CB action strategy based on probability.

[0094] (twenty two)

[0095] (twenty three)

[0096] Among them, A CB,t It is CB's action decision, A CB,t ∈[0,4], N0 is the number of objective functions, ω m,k It is the preference weight of the k-th objective, εt σ is the time-varying exploration rate, ε0 is the exploration rate decay coefficient, and ε0 is the initial exploration rate. Setting a time-varying exploration rate can automatically reduce random exploration in the later stages of training, accelerating convergence. (Neural Network Q) k The parameters are optimized by minimizing the loss function, and the Q parameters of the two neural networks are updated according to the following formula.

[0097] (twenty four)

[0098] (25)

[0099] Among them, S CB,t A CB,t ~D, D is the experience pool, L Qk (ω m ) is the loss function, N d It refers to the number of dimensions, where d is the dimension of the action, and y is the dimension of the action. l It is the target Q value, R` CB,t It is the normalized reward function vector, Q` k It is a target Q-network.

[0100] Step S4 is as follows:

[0101] The MOSAC algorithm improves upon the traditional SAC algorithm primarily in three dimensions: 1) dynamic weighting of multi-objective policy entropy; 2) value function reconstruction based on preference space; and 3) constraint embedding design of continuous action space. These improvements significantly enhance the multi-objective coordination capability of continuous control variables such as PV / SVG.

[0102] The SAC algorithm is a reinforcement learning algorithm based on maximum entropy theory. Its main idea is to maximize the entropy of the policy to achieve higher exploratory power and robustness. Unlike reinforcement learning where the sole objective is to maximize the agent's cumulative reward, the SAC algorithm considers both the reward function and entropy in each output action to balance maximum reward with policy randomness. This approach improves the agent's exploratory ability and robustness. The optimization objective of SAC can be expressed as:

[0103] (26)

[0104] (27)

[0105] (28)

[0106] Where, r sf (s t ,a tH(π(·|s)) is the reward value trained by the soft policy. To maximize the reward and improve learning efficiency and policy performance, soft policy evaluation and soft policy improvement are carried out alternately, forming the main stages of the iterative training process. α is the entropy coefficient, used to control the randomness of the optimal policy, and is automatically adjusted according to the policy during training to increase the exploration level of the region. t H0 = -dim(A) represents the entropy value, indicating the negative dimension of the action space. π(·) is the general policy. t (·) represents the strategy at time t, ρ π It is state s t and action a t The joint distribution of λ, where λ is the learning rate, ∇ α This is the gradient function, and J(α) is the loss function.

[0107] When a set of sample data {S} is randomly drawn from the experience pool t A t , R t , S t+1 , ω m After that, the parameters of the policy network are updated according to the following formula:

[0108] (29)

[0109] (30)

[0110] Where J(θ) is the loss function for evaluating the network, y is the predicted Q value, and R` t Here, PV and SVG are normalized reward function vectors for continuous action devices, Q' is the target evaluation network, β is the weight parameter of the entropy term, H is the entropy under policy π, and ∇ θ τ is the gradient of the loss function, and τ is the soft update coefficient.

[0111] Step S5 is as follows:

[0112] Because reward values ​​exist with different dimensions, the proposed MODQN and MOSAC algorithms employ a normalization method to process the reward function vector, ensuring the additivity of the reward values. The specific formulas are as follows:

[0113] (31)

[0114] Among them, R max,1,t R max,2,t R min,1,t R min,2,t These are the upper and lower limits of the two objective functions at time t, respectively, with corresponding preference weights ω. m =[1,0] and ω m The target value when =[0,1]. R 1,t R2,t These are the reward values ​​of the two objective functions at time t, which are solved using conventional DRL.

[0115] Another aspect of the present invention provides a multi-objective reactive power optimization system for power distribution networks based on deep reinforcement learning, comprising: a computer-readable storage medium and a processor;

[0116] The computer-readable storage medium is used to store executable instructions;

[0117] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the deep reinforcement learning-based multi-objective reactive power optimization method for power distribution networks.

[0118] In another aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned multi-objective reactive power optimization method for power distribution networks based on deep reinforcement learning.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A power distribution network multi-objective reactive power optimization method based on deep reinforcement learning, characterized in that, The method comprises the following steps: S1: constructing a multi-objective optimization model considering voltage deviation and comprehensive operation cost, the multi-objective optimization model outputs a target function vector containing voltage offset cumulative value, system active loss and device action cost; S2: based on the multi-objective optimization model of step S1, a multi-objective Markov decision process (MOMDP) model is constructed, the MOMDP model takes the real-time operation state of the distribution network as the state space, takes the control strategy of the photovoltaic (PV), static var generator (SVG) and capacitor bank (CB) as the action space, and takes the target function vector of step S1 as the core to design the reward function; S3: for the discrete control variables in the action space of step S2, the deep Q network (DQN) algorithm is improved into a multi-objective deep Q network (MODQN) algorithm, the state-action-reward samples of the MOMDP model in step S2 are stored through the experience replay mechanism, and the time-varying exploration rate is optimized for the discrete action decision; S4: for the continuous control variables in the action space of step S2, the soft actor critic (SAC) algorithm is improved into a multi-objective soft actor critic (MOSAC) algorithm, based on the state space and the reward function of the MOMDP model of step S2, the strategy entropy coefficient is dynamically adjusted through the maximum entropy theory, and the exploration and robustness of the continuous action decision are optimized; S5: the reward function vector output by step S2 is normalized to eliminate the dimensional difference, and then the multi-objective deep reinforcement learning is solved by combining the MODQN algorithm of step S3 and the MOSAC algorithm of step S4, and the coordinated control instruction is output to realize the reactive power optimization of the distribution network.

2. The method of claim 1, wherein, In step S2, the MOMDP model is defined by six tuples (S, A, P, R, W, γ), wherein the state space S, the action space A and the reward function R correspond to the input variable, the decision variable and the target function of the optimization model in step S1 respectively, the preference space W is used for weighted fusion of multi-objective reward, and the state transition probability P represents the dynamic change of the distribution network state after the action is executed.

3. The method of claim 1, wherein, The MODQN algorithm of step S3 specifically comprises: constructing a Q value network input layer based on the MOMDP state space of step S2, taking a discrete action subset in the action space as an output layer, optimizing network parameters by minimizing a loss function, the calculation of the loss function depends on the reward value and the state transition data in the experience samples stored in step S2, and the discrete action subset includes the number of capacitor banks put into groups.

4. The method of claim 1, wherein, The MOSAC algorithm of step S4 specifically comprises: constructing a strategy network based on the MOMDP state space and the continuous action subset (photovoltaic reactive power output and SVG compensation amount) of step S2, optimizing network parameters by maximizing the weighted sum of cumulative reward and strategy entropy, the cumulative reward is calculated through the reward function of step S2, and the entropy coefficient is dynamically adjusted through the gradient to balance exploration and utilization.

5. The method of claim 1, wherein, In step S5, the reward function normalization processing is specifically: based on the upper and lower limits of the reward function vector in step S2 (corresponding to the target values when the preference weight is [1, 0] and [0, 1]), the voltage deviation and the comprehensive operation cost reward value are mapped to the [0, 1] interval through linear transformation, and the normalization result is taken as the input of the algorithms in steps S3 and S4.

6. The method of claim 1, wherein, The multi-objective deep reinforcement learning solving in step S5 is specifically: the discrete action decision output in step S3 and the continuous action decision output in step S4 are combined into a complete control strategy, which is substituted into the multi-objective optimization model in step S1 to verify the feasibility, and the iteration optimization is performed until the optimal reactive power control instruction covering the Pareto frontier is obtained.

7. The method of claim 1, wherein, The real-time operation state of the distribution network in step S2 includes photovoltaic active output, load active and reactive power.

8. The method of claim 1, wherein, The discrete control variable in step S3 includes capacitor group action, and the continuous control variable in step S4 includes photovoltaic and SVG output.

9. A deep reinforcement learning based multi-objective reactive power optimization system for distribution network, comprising: A computer readable storage medium and a processor; The computer readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer readable storage medium, and execute the multi-objective reactive power optimization method for the distribution network based on deep reinforcement learning in any one of claims 1-8.

10. A non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the multi-objective reactive power optimization method for the distribution network based on deep reinforcement learning in any one of claims 1-8.

Images