Distributed photovoltaic scheduling method based on multi-agent consensus optimization

Abstract

The invention relates to the technical field of distributed photovoltaic scheduling, in particular to a distributed photovoltaic scheduling method based on multi-agent consensus optimization, and the method comprises the steps: modeling a real-time reactive power optimization decision of a photovoltaic inverter into a networked multi-agent Markov decision process; according to the method, the minimum voltage offset and the minimum network loss are mapped into reward functions, a multi-agent soft actor-commentator method based on a maximum entropy frame is adopted, a consensus penalty term is added to constrain that local strategies and value function parameters of all agents tend to be consistent, and convergence and global collaboration of distributed learning are ensured. A constraint violation priority experience playback mechanism is introduced, sparse abnormal samples are efficiently utilized through an experience enhancement module and a priority calculation module, and the learning efficiency and safety of the model under key boundary conditions are improved. According to the method, the operation problems of voltage crossing, sharp increase of network loss and the like caused by bidirectional power flow, intermittency and volatility in a high-permeability photovoltaic access power distribution network scene are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed photovoltaic scheduling, in particular to a distributed photovoltaic scheduling method based on multi-agent consensus optimization. BACKGROUND

[0002] With the large-scale access of distributed photovoltaic energy in the distribution network, the energy utilization efficiency is significantly improved, but also new technical challenges. The traditional one-way power flow gradually evolves into a complex two-way flow, making the traditional protection mechanism and voltage regulation invalid, causing voltage out-of-limit, network loss surge and other problems. At the same time, photovoltaic equipment has the characteristics of intermittency and volatility, which leads to an increase in the difficulty of photovoltaic resource scheduling, further threatening the normal operation of the power system.

[0003] The existing optimization method mostly adopts a centralized optimization model, which highly depends on the accurate modeling of the power grid system, and the solving process is complex. When facing the access of distributed photovoltaic and other uncertain disturbance sources, its robustness and stress are insufficient. In addition, in the distributed framework, the model-free method learns the strategy based on local observation and local reward, lacks global information exchange or coordination mechanism, and the strategy update of a single agent will change the dynamics of the environment, causing the environment under the perspective of other agents to be non-stationary, which restricts the convergence speed and learning stability, resulting in poor global coordination performance. Finally, the distribution network has high safety requirements, but in actual operation, the abnormal samples that cause voltage out-of-limit are sparse, and the model is over-trained on normal samples, but when dealing with high-risk abnormal events, it shows insufficient robustness. SUMMARY

[0004] The present application aims to provide a distributed photovoltaic scheduling method based on multi-agent consensus optimization, which aims to solve the problems of voltage out-of-limit and network loss surge caused by two-way flow, intermittency and volatility in the scenario of high penetration rate of distributed photovoltaic access to the distribution network.

[0005] A distributed photovoltaic scheduling method based on multi-agent consensus optimization is applied to a partitioned distribution network containing distributed photovoltaic inverters, comprising the following steps:

[0006] Step S1: In a distribution network with nodes, including photovoltaic nodes, , model each inverter in each photovoltaic node as an independent agent, construct a target model, and establish a dual-objective optimization model considering the safety and economy of the distribution network, including the first target of minimizing voltage deviation and the second target of minimizing network loss;

[0007] Step S2: Map the reactive power decision process of the distributed photovoltaic inverter into a networked multi-agent Markov decision process, and construct a mapping relationship connecting the dual objectives and the reward function;

[0008] Step S3: Solve the multi-agent Markov decision process based on the multi-agent soft policy-comment algorithm to achieve policy learning and network parameter update with exploratory capabilities;

[0009] Step S4: Design a consensus mechanism for multi-agent optimization, introduce a random consensus penalty term to ensure the coordination and consistency of local policies and value function parameters of each agent in distributed learning;

[0010] Step S5: Employ the experience enhancement and priority calculation modules to construct a violation priority experience replay mechanism to improve the sampling efficiency of sparse and anomalous samples.

[0011] This invention models real-time reactive power optimization as a networked multi-agent Markov decision process and uses a multi-agent soft policy-comment algorithm to solve the multi-agent Markov decision process. By introducing a consensus mechanism and constraint violation priority experience replay, a distributed cooperative control strategy with high communication efficiency and high sample efficiency is realized. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating the overall architecture of the distributed photovoltaic scheduling based on multi-agent consensus optimization of this invention.

[0013] Figure 2 This is a flowchart of the distributed photovoltaic scheduling method based on multi-agent consensus optimization according to the present invention;

[0014] Figure 3 , Figure 4 The figure shows the simulation results of the method of the present invention under the first network specification.

[0015] Figure 5 , Figure 6 The figure shows the simulation results of the second network specification using the method of the present invention. Detailed Implementation

[0016] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described below are only for explaining the present invention and do not limit the scope of protection of the present invention.

[0017] The present invention will be further described in detail below through preferred embodiments:

[0018] As attached Figure 1 and attached Figure 2As shown: This embodiment discloses a distributed photovoltaic scheduling method based on multi-agent consensus optimization, including the following steps:

[0019] Step S1: In having In a distribution network of nodes, including One photovoltaic node, Each inverter in each photovoltaic node is modeled as an independent intelligent agent, and a target model is constructed. A dual-objective optimization model that takes into account both the safety and economy of the distribution network is established, including the first objective of minimizing voltage deviation and the second objective of minimizing network loss.

[0020] Step S2: Map the reactive power decision process of the distributed photovoltaic inverter into a networked multi-agent Markov decision process, and construct a mapping relationship connecting the dual objectives and the reward function;

[0021] Step S3: Solve the multi-agent Markov decision process based on the multi-agent soft policy-comment algorithm to achieve policy learning and network parameter update with exploratory capabilities;

[0022] Step S4: Design a consensus mechanism for multi-agent optimization, introduce a random consensus penalty term to ensure the coordination and consistency of local policies and value function parameters of each agent in distributed learning;

[0023] Step S5: Using the experience enhancement and priority calculation module, a violation priority experience replay mechanism is constructed to improve the sampling efficiency of sparse anomalous samples.

[0024] Furthermore, in step S1, the steps for constructing the dual-objective optimization model are as follows:

[0025] In having In a distribution network of nodes, including One photovoltaic node, Each inverter in each photovoltaic node is modeled as an independent intelligent agent. Based on real-time operating data, the intelligent agent monitors its own reactive power. Make decisions to respond in real time to voltage fluctuations at photovoltaic nodes.

[0026] The overall objective function aims to minimize both voltage offset and network loss, making it a dual-objective optimization problem that balances safety and economy.

[0027]

[0028] This includes minimizing voltage offset. Minimize network loss Dual objectives.

[0029] The physical constraints are as follows:

[0030] 1) Active and reactive power balance constraints

[0031]

[0032] in, and Photovoltaic nodes (i.e., intelligent agent) At that moment Net injected active power and net injected reactive power; For photovoltaic nodes The active power output of the photovoltaic inverter; For photovoltaic nodes Active power of the load at the location; Photovoltaic nodes and At any moment Voltage amplitude; G ij To connect photovoltaic nodes With photovoltaic nodes The branch conductance, B ij To connect photovoltaic nodes With photovoltaic nodes Branch circuit susceptance; For photovoltaic nodes and The voltage phase angle difference between them; To connect with photovoltaic nodes A set of connected neighboring photovoltaic nodes; For photovoltaic nodes The reactive power output of the photovoltaic inverter; For photovoltaic nodes The reactive load power at the location.

[0033] 2) Apparent power capacity constraint

[0034] Active power output of photovoltaic inverter and reactive power The combination must be limited to the inverter's rated apparent power capacity. To ensure the safe operation of the equipment, the specific expression is as follows:

[0035]

[0036] 3) Voltage safety constraints

[0037] For photovoltaic nodes in the distribution network At any moment Minimum and maximum permissible voltage amplitudes. Dispatch-up photovoltaic node voltage. It must always be maintained within the safe operating range specified by the system. Inside.

[0038]

[0039] Furthermore, in step S2, the steps for constructing the Markov decision process are as follows:

[0040] Reactive power decision-making for distributed photovoltaic (PV) inverters is defined as a networked multi-agent Markov decision process, where each PV inverter... It is an independent intelligent agent that operates in a shared environment.

[0041] 1) State space and local observation space

[0042] During the distributed execution phase, due to limitations in communication bandwidth and privacy protection, the intelligent agent... Decision Based on its local observations Local observation From the local observation area Obtain This is a subset of photovoltaic nodes with strong voltage correlation characteristics with the agent. In a zoned distribution network, this set includes the branch-end photovoltaic nodes physically connected to the agent. Local observation. Defined as an intelligent agent At any moment The collected and acquired local runtime data vectors are specifically represented as follows:

[0043]

[0044] in, For intelligent agents Local observation area The node voltage magnitude vector; For local observation area The phase angle vector; and Local observation areas The vector of active and reactive power of the load; For intelligent agents Active power output of the photovoltaic node; For intelligent agents The reactive power output of the photovoltaic node.

[0045] The operation of the entire network architecture is divided into two phases: a distributed execution phase and a centralized training phase.

[0046] During the centralized training phase, each agent's local observations Together they constitute the global state Define the state space as Then the global state vector Mapped in the state space, i.e. .

[0047] 2) Decision-making space

[0048] At any moment intelligent agent Based on local observations Make a decision Specifically, the decision-making of intelligent agents Define the decision space as follows: [Value of reactive power output command for its own photovoltaic inverter in the next time period]. Decision-making of each agent Mapped to its corresponding local decision space In, that is The decisions of all agents together constitute joint decision-making. ;

[0049] During the centralized training phase, each agent's local observations Together they form the global state vector Define the state space as Then the global state Mapped in the state space, i.e. reward function Define intelligent agents In global state Joint decision-making And cause the state to transition. The scalar feedback signal obtained subsequently maximizes the expected cumulative discount reward. It guides the updating and optimization of decisions made by various intelligent agents, among which This is the discount factor.

[0050] 3) Reward function

[0051] reward function Define intelligent agents In state Joint decision-making And cause the state to transition. The scalar feedback signal obtained subsequently. The reward function of agent i. The definition is as follows:

[0052]

[0053] in, This is the system reference voltage; For agents to perform joint decision-making Afterwards, its local observation area Internal photovoltaic node exist Voltage amplitude at any given moment; For network losses; These are weighting coefficients used to balance voltage regulation and network loss optimization objectives.

[0054] The above reward function By minimizing the voltage offset for the first objective The second objective is to minimize network loss. By applying negative weighting, the original problem of minimizing physical constraints is transformed into a problem of maximizing cumulative reward in multi-agent reinforcement learning. This mapping method ensures that while agents pursue high reward signals, they can simultaneously achieve coordinated optimization of distribution network voltage smoothing and operational loss reduction at the physical level. Based on the defined reward function, the expected cumulative discount reward is maximized. It guides the updating and optimization of policies for each intelligent agent, among which This is the discount factor.

[0055] Network losses of power grid The calculation is the sum of active power losses across all branches:

[0056]

[0057] in For the collection of distribution network branches, For photovoltaic nodes and photovoltaic nodes The resistance of the branch between, For photovoltaic nodes and photovoltaic nodes The square of the current in the branch between.

[0058] Furthermore, in step S3, the multi-agent soft policy-comment solution and update steps are as follows:

[0059] The constructed multi-agent Markov decision process needs to be solved using a neural network with exploratory capabilities. This invention employs a multi-agent soft policy-evaluation algorithm based on a maximum entropy reinforcement learning framework. Each agent contains a policy network and an evaluation network, and the updates and optimizations of these two networks follow the process described below.

[0060] 1) Optimization objective of soft policy networks

[0061] Soft policy network objective function It is a linear combination that maximizes the expected cumulative return and the policy entropy:

[0062]

[0063] in, Represents the mathematical expectation. Indicates the experience pool in the buffer. Random sampling time global state , Indicates joint decision-making Obeying the policy network Defined random policy distribution; Discount factor; For the reward function; The temperature coefficient of entropy is used to control the randomness of the strategy; For soft policy networks In global state Entropy;

[0064] 2) Soft decision-value function

[0065] Evaluating the use of soft decision-value functions in networks The update is performed, and the update equation is:

[0066]

[0067] in, To evaluate the network parameters, and Representing time respectively and The global state; Represents the state transition probability The next state generated Expectations; Discount factor; The soft state-value function for the next time step is calculated as follows:

[0068]

[0069] in, Indicates the state Make a decision The logarithmic probability, Represents the mathematical expectation. In policy network sampling Joint decision-making at any time .

[0070] 3) Q-network loss function

[0071] The multi-agent soft policy-comment algorithm employs a double-Q network mechanism and introduces an online soft Q function. and and the corresponding objective Q function and ;

[0072] Q network loss function The objective is to minimize the mean square error between the online evaluation network's predicted value and the target value, which is defined as follows:

[0073]

[0074] Among them, E Indicates in the experience buffer pool Randomly selected empirical tuples, representing respectively Global state at any given moment, joint decision-making, rewards, and Global state at all times; To evaluate the decision-making value of network predictions online; The calculation method for the target decision value is as follows:

[0075]

[0076] in, for Time-based reward function; Discount factor; For the first A target evaluation network, This means taking the minimum of the two target decision values, thereby suppressing the overestimation of target decision values ​​during the training process; The temperature coefficient of entropy; Indicating soft policy networks in Make decisions at all times The probability density.

[0077] 4) Policy Network Update

[0078] Soft policy networks minimize the loss function This drives the soft policy network parameters toward a balance between expected cumulative return and policy entropy.

[0079]

[0080] Based on the above loss function, gradient ascent is used to adjust the gradient corresponding to the objective function of the soft policy network. Update, the update method is as follows:

[0081]

[0082] in, For parameters of the soft policy network, For parameters The gradient operator; The objective function of the soft policy network; To make decisions under the current strategy The logarithmic probability; To evaluate the value of network-predicted target decisions online, this gradient update formula aims to improve the parameters through the gradient ascent algorithm. Update in the direction of maximizing expected cumulative reward and policy entropy.

[0083] Furthermore, in step S4, the consensus mechanism steps for multiple agents are as follows:

[0084] Within the maximum entropy reinforcement learning framework, to ensure that the local learning of distributed agents can achieve global collaboration, this invention introduces a consensus mechanism. The consensus mechanism aims to enable each agent to... Local parameterized copy Convergence, parameterized replicas Defined as:

[0085]

[0086] in, To evaluate network parameters; For the first The policy network parameters of each agent; the consensus optimization problem is formalized as:

[0087]

[0088] in, For the first The local objective function of an agent without considering consensus constraints; The constraint requires each agent to maintain its own copy of local parameters. During the optimization iteration process, the parameters tend to be consistent globally, where st represents the constraint that ensures the consistency of these replica parameters.

[0089] This invention employs a stochastic consensus strategy, integrating equality constraints into the local objective through Lagrange relaxation and a quadratic penalty term to achieve distributed optimization. In this process, the global communication graph is determined using the connection relationships of physical photovoltaic nodes in the distribution network, and the adjacency matrix of the communication graph is used to... degree matrix Perform mathematical representation.

[0090] Local consensus Lagrangian function of agent i The definition is as follows:

[0091]

[0092] in, For intelligent agents The local update loss function is calculated as follows:

[0093]

[0094] The loss function is the Q-network loss function. For the first The loss function of the policy network of each agent; For intelligent agents Consensus punishment multiplier; It is a global state and joint decision-making As input, by parameters The output value of the determined nonlinear mapping function; Communication graph degree matrix The Kroneck block; For the adjacency matrix of the communication graph Kronecker block; quadratic penalty term through quantization of local output Information aggregated with neighborhood weights The deviation between them is penalized during training iterations to drive the agents to reach a global consensus in the feature representation space. For intelligent agents The local observation area.

[0095] The calculation method for the Kronecker block is as follows:

[0096]

[0097]

[0098] in, It is a parameterized copy of the agent. Identity matrices of the same dimension Communication graph degree matrix The diagonal elements, The adjacency matrix is ​​the first Line number Column elements. Through the Kronecker product operation, the scalar form of the communication topology weights is mapped to a high-dimensional neural network parameter space, thereby realizing the network parameters of each agent's soft policy. With evaluation network parameters Strict consistency constraints in distributed iterative processes.

[0099] Furthermore, in step S5, the constraint violation priority experience replay mechanism steps are as follows:

[0100] In power systems, constraint violations that lead to voltage out-of-bounds or line overloads are sparse anomalous samples. To address the issues of sample sparsity and low training efficiency, this invention establishes a constraint violation priority experience replay mechanism, comprising two modules: experience enhancement and priority calculation.

[0101] 1) Experience Enhancement Module

[0102] By leveraging the data symmetry of homogeneous agents, additional empirical tuples are generated through permutation operations, thereby improving the sample acquisition rate of multi-agent systems. .

[0103] set up Let be the set of all feasible permutation matrices. Let be any element of the permutation matrix. When the agent obtains a fundamental empirical tuple from the environment... At that time, new empirically enhanced tuples are generated through matrix multiplication. Store in the experience replay pool :

[0104]

[0105] in, right The elements of the vector are subjected to coaxial permutations in parallel components. Leveraging the physical symmetry of homogeneous agents, the permutation matrix is... By applying the data items of each agent to the corresponding positions in the vector in a synchronous manner to each component of the tuple, enhanced samples can be generated without changing the physical logic of the system.

[0106] 2) Priority Calculation Module

[0107] To prioritize the playback of samples with constraint violation risk, this invention calculates sampling priority based on the sample's timing difference error and voltage violation amount. .sample Sampling priority Defined as:

[0108]

[0109] in, sample The timing difference error; A small constant used to avoid zero priority; The timing difference error weight controls the contribution of timing difference error to priority. To constrain violations of weights; For photovoltaic nodes At any moment The voltage amplitude; These are the minimum and maximum allowable voltage amplitudes for photovoltaic nodes; Use the ReLU function to calculate voltage violations; This refers to the set of all photovoltaic nodes in the distribution network.

[0110] Examples of hyperparameters involved in this invention are shown in Table 1.

[0111] Table 1

[0112]

[0113] Both the soft policy network and the evaluation network consist of three fully connected feedforward layers. The weights of the linear layers are randomly initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.1.

[0114] Online simulation was performed based on the method of this invention, and the selected network specifications are shown in Table 2.

[0115] Table 2

[0116]

[0117] The simulation results for the first network specification are attached. Figure 3 and attached Figure 4 As shown in the attached figure, the simulation results for the second network specification are as follows. Figure 5 and attached Figure 6 As shown in Table 3, the simulation data are for the first and second network specifications mentioned above.

[0118] Table 3

[0119]

[0120] Based on simulation data provided in the embodiments of the present invention, this method demonstrates technical advantages and beneficial effects in distributed photovoltaic (PV) scheduling. Under different network scales, the present invention maintains the average voltage of PV nodes within the allowable offset. Within the specified range. Meanwhile, the average voltage out-of-bounds rates for the first and second network specifications are 0.773% and 1.997%, respectively. Despite the increased system scale and complexity, the method of this invention effectively reduces the out-of-bounds rate, avoiding the risk of voltage out-of-bounds errors caused by photovoltaic fluctuations. Finally, the average network loss of the system under the two specifications is optimized to 0.096MW and 0.451MW, respectively, achieving good economic operating indicators while ensuring voltage safety.

[0121] In summary, as the number of distributed photovoltaic units increased from 6 to 22, and the load scale expanded, this method maintained stable reward value performance and a low level of out-of-bounds behavior. This indicates that the random consensus penalty term introduced in this invention can ensure that each agent achieves a high degree of policy coordination and parameter consistency in a complex networked environment, effectively solving the non-stationary operation problem caused by high-penetration photovoltaic access and possessing engineering applicability.

[0122] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. Typical known structures and common knowledge techniques in the preferred embodiments have not been described in detail here. Those skilled in the art can improve and implement the technical solutions of this invention based on the guidance provided in these embodiments and their own capabilities. Some typical known structures, known methods or common knowledge techniques should not be obstacles for those skilled in the art to implement this application.

[0123] The scope of protection claimed in this application shall be determined by the contents of its claims, and the contents described in the invention description, specific embodiments and drawings shall be used to interpret the claims.

[0124] Within the scope of the technical concept of this application, several modifications can be made to the specific implementation of this application, and these modified implementations should also be considered within the protection scope of this application.

Claims

1. A distributed photovoltaic scheduling method based on multi-agent consensus optimization, characterized in that, Applied to the sub-regional power distribution network containing distributed photovoltaic inverters, comprising the following steps: Step S1: in a power distribution network with nodes, including photovoltaic nodes, each photovoltaic inverter in each photovoltaic node is modeled as an independent agent, a target model is constructed, a dual-objective optimization model considering the safety and economy of the power distribution network is established, including a first target of minimizing voltage deviation and a second target of minimizing network loss; Step S2: mapping the reactive power decision process of the distributed photovoltaic inverter into a networked multi-agent Markov decision process, and constructing a mapping relationship connecting the double objectives and the reward function; Step S3: solving the multi-agent Markov decision process based on the multi-agent soft policy-critic algorithm, realizing the strategy learning and network parameter updating with exploration ability; Step S4: designing a consensus mechanism for multi-agent optimization, introducing a random consensus penalty term to ensure the consistency of the local strategy and the value function parameters of each agent in the distributed learning; Step S5: using the experience enhancement and priority calculation module to construct the constraint violation priority experience replay mechanism to improve the sampling efficiency of sparse and abnormal samples.

2. The method of claim 1, wherein, In step S2, the Markov decision process is constructed as follows: Treating each photovoltaic inverter as an independent intelligent agent, during the distributed execution phase, the intelligent agent... Decision Based on its local observations Local observation Defined as an intelligent agent At any moment Local runtime data vectors collected and acquired: wherein, the local observation area of the intelligent agent the node voltage amplitude vector of the local observation area ; the phase angle vector of the local observation area ; and the active power and reactive power vectors of the local observation area ; the active power output of the photovoltaic node where the intelligent agent ; the reactive power output of the photovoltaic node where the intelligent agent ; At time , the agent makes a decision according to the local observation , specifically, the decision of the agent is the reactive power output instruction value of its own photovoltaic inverter in the next time period, the decision space is defined as , the decision of each agent is mapped to its corresponding local decision space , that is ; the decisions of all agents together constitute the joint decision During the centralized training phase, each agent's local observations Together they form the global state vector Define the state space as Then the global state Mapped in the state space, i.e. reward function Define intelligent agents In global state Joint decision-making And cause the state to transition. The scalar feedback signal obtained subsequently maximizes the expected cumulative discount reward. It guides the updating and optimization of decisions made by various intelligent agents, among which This is the discount factor.

3. The method of claim 2, wherein, To achieve the dual goals of minimizing voltage shift and minimizing network loss, the reward function for agent i is is defined as follows: wherein, Vref is a system reference voltage; Vref is a system reference voltage; Vref is a system reference voltage; Vref is a system reference voltage; Vref is a system reference voltage; Vref is a system reference voltage; Vref is a system reference voltage; Vref is a system reference voltage; 4. The method of claim 1, wherein, In step S3, the agent contains a soft policy network and an evaluation network, and the multi-agent soft policy-critic solving and updating steps are as follows: Soft policy networks Objective function is to maximize a linear combination of the expected cumulative reward and the policy entropy: wherein, denotes the mathematical expectation, denotes the global state at a randomly drawn time instant from the buffer experience pool , , denotes the joint decision subject to a stochastic policy distribution defined by the soft policy network ; is a discount factor; is a reward function; is a temperature coefficient for the entropy used to control the randomness of the policy; is the entropy of the soft policy network at the global state . Evaluating network usage soft-decision-value functions An update is made with an update equation of: wherein, is the parameter of the network to be evaluated, and represent the global state at time and respectively; represents the expectation of the next state resulting from the state transition probability is the discount factor; is the soft state-value function at the next time, which is calculated as:​ wherein, denotes the log probability of taking a decision in state , denotes the mathematical expectation, at a soft policy network sampling the joint decision at time instant .

5. The method of claim 4, wherein, The multi-agent soft policy-critic algorithm adopts a double Q network mechanism, and introduces an online soft Q function and , and a corresponding target Q function and ; Q-network loss function The goal is to minimize the mean squared error between the online evaluation network's predicted value and the target value, which is defined as follows: where E represent the global state, joint decision, reward and randomly sampled experience tuples, respectively, represent global state at time t, joint decision, reward and global state at time t; to evaluate the decision value of the network prediction online; is the calculated target decision value, which is calculated as follows: wherein, is the reward function at time step is a discount factor; is the th target evaluation network, represents taking the minimum value of two target decision values, thereby suppressing the overestimation of the target decision value in the training process; is the temperature coefficient of entropy; represents the probability density of the joint decision of the soft policy network at time step .

6. The multi-agent consensus optimization based distributed photovoltaic scheduling method according to claim 5, characterized in that, The soft policy network minimizes a loss function that drives the soft policy network parameters towards a balance of expected cumulative reward and policy entropy: Based on the above loss function, the gradient corresponding to the objective function of the soft policy network is updated by gradient ascent The updating is as follows: wherein, is a parameter of the soft policy network, is a gradient operator on the parameter ; is an objective function of the soft policy network; is a log probability of taking a decision under the current policy; is a target decision value predicted by the online evaluation network, the gradient update formula aims to update the parameter in the direction of maximizing the expected cumulative reward and maximizing the policy entropy by the gradient ascent algorithm.

7. The multi-agent consensus optimization based distributed photovoltaic scheduling method according to claim 1, characterized in that, In step S4, the multi-agent consensus mechanism steps are as follows: The consensus mechanism aims to enable various intelligent agents to... Local parameterized copy Convergence, parameterized replicas Defined as: wherein, to evaluate network parameters; to the parameters of the soft policy network of the i-th agent; the consensus optimization problem is formalized as: in, For the first The local objective function of an agent without considering consensus constraints; This refers to the constraints requiring each agent to maintain its own copy of local parameters. During the optimization iteration process, the parameters tend to be consistent globally, where st represents the constraint that ensures the consistency of these replica parameters.

8. The method of claim 7, wherein, A stochastic consensus strategy is adopted, integrating equality constraints into the local objective through Lagrange relaxation and quadratic penalty terms. During this process, the global communication graph is determined using the connectivity relationships of physical photovoltaic nodes in the distribution network, and the adjacency matrix of the communication graph is used to... degree matrix Mathematical representation, intelligent agent Local consensus Lagrange function The definition is as follows: wherein, is the local update loss function for the agent is calculated as: The loss function is the Q-network loss function. For the first The loss function of a soft policy network for an agent; For intelligent agents Consensus punishment multiplier; It is a global state and joint decision-making As input, by parameters The output value of the determined nonlinear mapping function; Communication graph degree matrix The Kroneck block; For the adjacency matrix of the communication graph Kronecker block; quadratic penalty term through quantization of local output Information aggregated with neighborhood weights The deviation between them is penalized during training iterations to drive the agents to reach a global consensus in the feature representation space. For intelligent agents The local observation area; Wherein, the calculation method of the Kronecker product block is: wherein, is a parameterized copy of the agent is an identity matrix of the same dimension, is a communication graph degree matrix is the diagonal element of the is the element of the adjacency matrix; the parameters of the soft policy network of each agent are mapped from the scalar form of the communication topology weights to the high-dimensional neural network parameter space by the Kronecker product operation and the critic network parameters are strictly consistent in the distributed iterative process.

9. The method of claim 1, wherein, In step S5, the constraint violation priority experience replay mechanism steps are as follows: The experience enhancement module utilizes data symmetry of homogeneous agents to generate additional experience tuples through permutation operation, thereby improving sample collection rate of the multi-agent system ; Let be the set of all feasible permutation matrices, be any one permutation matrix element, when the agent obtains a ground truth experience tuple in the environment, a new experience augmented tuple is generated by matrix multiplication and stored in the experience replay pool : wherein, To The component parallel coaxial permutation of the elements in the vector, using the physical symmetry of homogeneous agents, will make the permutation matrix Act on each component of the tuple respectively, and by synchronously adjusting the data items of the corresponding positions of each agent in the vector, the enhanced samples are generated without changing the physical logic of the system.

10. The multi-agent consensus optimization based distributed photovoltaic scheduling method according to claim 9, characterized in that, The priority calculation module calculates the sampling priority by using the time difference error and the voltage violation of the sample , the sampling priority of the sample is defined as: wherein, sample a timing difference error; is a small constant for avoiding zero priority; is a timing difference error weight, controlling the contribution of the timing difference error to the priority; is a constraint violation weight; is a photovoltaic node a voltage amplitude at time ; are minimum and maximum voltage amplitudes allowed for the photovoltaic node; is a ReLU function, calculating a voltage violation quantity; is a set of all photovoltaic nodes of the power distribution grid.

Images