A distributed photovoltaic scheduling method based on multi-agent consensus optimization

Abstract

This invention relates to the field of distributed photovoltaic (PV) scheduling technology, specifically to a distributed PV scheduling method based on multi-agent consensus optimization. This invention models the real-time reactive power optimization decision of PV inverters as a networked multi-agent Markov decision process, mapping minimizing voltage deviation and minimizing network loss to reward functions. It employs a multi-agent soft actor-critic method based on a maximum entropy framework, adding a consensus penalty term to constrain the local policies and value function parameters of each agent to converge, ensuring the convergence and global coordination of distributed learning. A constraint violation priority experience replay mechanism is introduced, using an experience enhancement module and a priority calculation module to efficiently utilize sparse outlier samples, improving the model's learning efficiency and security under critical boundary conditions. This invention solves operational challenges in high-penetration PV-connected distribution network scenarios, such as voltage overflow and surges in network loss caused by bidirectional power flow, intermittency, and volatility.

Description

Technical Field

[0001] This invention relates to the field of distributed photovoltaic scheduling technology, and specifically to a distributed photovoltaic scheduling method based on multi-agent consensus optimization. Background Technology

[0002] With the large-scale integration of distributed photovoltaic (PV) energy into power distribution networks, energy utilization efficiency has been significantly improved, but new technical challenges have also emerged. Traditional unidirectional power flows have gradually evolved into complex bidirectional flows, rendering traditional protection mechanisms and voltage regulation ineffective, leading to problems such as voltage exceedances and a surge in network losses. Simultaneously, the intermittent and fluctuating nature of PV equipment significantly increases the difficulty of PV resource scheduling, further threatening the normal operation of the power system.

[0003] Existing optimization methods mostly employ centralized optimization models, which heavily rely on accurate modeling of the power grid system, resulting in complex solution processes. They lack robustness and resilience when faced with uncertain disturbances such as distributed photovoltaic systems. Furthermore, in distributed frameworks, model-free methods learn policies based on local observations and rewards, lacking global information exchange or collaborative mechanisms. Policy updates by individual agents can alter the dynamics of the environment, leading to non-stationarity from the perspectives of other agents, thus limiting convergence speed and learning stability, resulting in poor global collaborative performance. Finally, distribution networks have high safety requirements; however, in actual operation, abnormal samples causing voltage exceedances are sparse, leading to overtraining of models on normal samples and insufficient robustness when handling high-risk anomalies. Summary of the Invention

[0004] This invention aims to provide a distributed photovoltaic scheduling method based on multi-agent consensus optimization, which is intended to solve the problems of voltage overruns and surges in network losses caused by bidirectional power flow, intermittency and volatility in high-penetration distributed photovoltaic grid access scenarios.

[0005] A distributed photovoltaic (PV) scheduling method based on multi-agent consensus optimization, applied to a zoned distribution network containing distributed PV inverters, includes the following steps:

[0006] Step S1: In having In a distribution network with nodes, including One photovoltaic node, Each inverter in each photovoltaic node is modeled as an independent intelligent agent, and a target model is constructed. A dual-objective optimization model that takes into account both the safety and economy of the distribution network is established, including the first objective of minimizing voltage deviation and the second objective of minimizing network loss.

[0007] Step S2: Map the reactive power decision process of the distributed photovoltaic inverter into a networked multi-agent Markov decision process, and construct a mapping relationship connecting the dual objectives and the reward function;

[0008] Step S3: Solve the multi-agent Markov decision process based on the multi-agent soft policy-comment algorithm to achieve policy learning and network parameter update with exploratory capabilities;

[0009] Step S4: Design a consensus mechanism for multi-agent optimization, introduce a random consensus penalty term to ensure the coordination and consistency of local policies and value function parameters of each agent in distributed learning;

[0010] Step S5: Employ the experience enhancement and priority calculation modules to construct a violation priority experience replay mechanism to improve the sampling efficiency of sparse and anomalous samples.

[0011] This invention models real-time reactive power optimization as a networked multi-agent Markov decision process and uses a multi-agent soft policy-comment algorithm to solve the multi-agent Markov decision process. By introducing a consensus mechanism and constraint violation priority experience replay, a distributed cooperative control strategy with high communication efficiency and high sample efficiency is realized. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating the overall architecture of the distributed photovoltaic scheduling based on multi-agent consensus optimization of this invention.

[0013] Figure 2 This is a flowchart of the distributed photovoltaic scheduling method based on multi-agent consensus optimization according to the present invention;

[0014] Figure 3 , Figure 4 The figure shows the simulation results of the method of the present invention under the first network specification.

[0015] Figure 5 , Figure 6 The figure shows the simulation results of the second network specification using the method of the present invention. Detailed Implementation

[0016] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described below are only for explaining the present invention and do not limit the scope of protection of the present invention.

[0017] The present invention will be further described in detail below through preferred embodiments:

[0018] As attached Figure 1 and attached Figure 2As shown: This embodiment discloses a distributed photovoltaic scheduling method based on multi-agent consensus optimization, including the following steps:

[0019] Step S1: In having In a distribution network with nodes, including One photovoltaic node, Each inverter in each photovoltaic node is modeled as an independent intelligent agent, and a target model is constructed. A dual-objective optimization model that takes into account both the safety and economy of the distribution network is established, including the first objective of minimizing voltage deviation and the second objective of minimizing network loss.

[0020] Step S2: Map the reactive power decision process of the distributed photovoltaic inverter into a networked multi-agent Markov decision process, and construct a mapping relationship connecting the dual objectives and the reward function;

[0021] Step S3: Solve the multi-agent Markov decision process based on the multi-agent soft policy-comment algorithm to achieve policy learning and network parameter update with exploratory capabilities;

[0022] Step S4: Design a consensus mechanism for multi-agent optimization, introduce a random consensus penalty term to ensure the coordination and consistency of local policies and value function parameters of each agent in distributed learning;

[0023] Step S5: Using the experience enhancement and priority calculation module, a violation priority experience replay mechanism is constructed to improve the sampling efficiency of sparse anomalous samples.

[0024] Furthermore, in step S1, the steps for constructing the dual-objective optimization model are as follows:

[0025] In having In a distribution network with nodes, including One photovoltaic node, Each inverter in each photovoltaic node is modeled as an independent intelligent agent. Based on real-time operating data, the intelligent agent monitors its own reactive power. Make decisions to respond in real time to voltage fluctuations at photovoltaic nodes.

[0026] The overall objective function aims to minimize both voltage offset and network loss, making it a dual-objective optimization problem that balances safety and economy.

[0027]

[0028] This includes minimizing voltage offset. Minimize network loss Dual objectives.

[0029] The physical constraints are as follows:

[0030] 1) Active and reactive power balance constraints

[0031]

[0032] in, and Photovoltaic nodes (i.e., intelligent agent) At that moment Net injected active power and net injected reactive power; For photovoltaic nodes The active power output of the photovoltaic inverter; For photovoltaic nodes Active power of the load at the location; Photovoltaic nodes and At any moment Voltage amplitude; G ij To connect photovoltaic nodes With photovoltaic nodes The branch conductance, B ij To connect photovoltaic nodes With photovoltaic nodes Branch circuit susceptance; For photovoltaic nodes and The voltage phase angle difference between them; To connect with photovoltaic nodes A set of connected neighboring photovoltaic nodes; For photovoltaic nodes The reactive power output of the photovoltaic inverter; For photovoltaic nodes The reactive load power at the location.

[0033] 2) Apparent power capacity constraint

[0034] Active power output of photovoltaic inverter and reactive power The combination must be limited to the inverter's rated apparent power capacity. To ensure the safe operation of the equipment, the specific expression is as follows:

[0035]

[0036] 3) Voltage safety constraints

[0037] For photovoltaic nodes in the distribution network At any moment Minimum and maximum permissible voltage amplitudes. Dispatch-up photovoltaic node voltage. It must always be maintained within the safe operating range specified by the system. Inside.

[0038]

[0039] Furthermore, in step S2, the steps for constructing the Markov decision process are as follows:

[0040] Reactive power decision-making for distributed photovoltaic (PV) inverters is defined as a networked multi-agent Markov decision process, where each PV inverter... It is an independent intelligent agent that operates in a shared environment.

[0041] 1) State space and local observation space

[0042] During the distributed execution phase, due to limitations in communication bandwidth and privacy protection, the intelligent agent... Decision Based on its local observations Local observation From the local observation area Obtain This is a subset of photovoltaic nodes with strong voltage correlation characteristics with the agent. In a zoned distribution network, this set includes the branch-end photovoltaic nodes physically connected to the agent. Local observation. Defined as an intelligent agent At any moment The collected and acquired local runtime data vectors are specifically represented as follows:

[0043]

[0044] in, For intelligent agents Local observation area The node voltage magnitude vector; For local observation area The phase angle vector; and Local observation areas The vector of active and reactive power of the load; For intelligent agents Active power output of the photovoltaic node; For intelligent agents The reactive power output of the photovoltaic node.

[0045] The operation of the entire network architecture is divided into two phases: a distributed execution phase and a centralized training phase.

[0046] During the centralized training phase, each agent's local observations Together they constitute the global state Define the state space as Then the global state vector Mapped in the state space, i.e. .

[0047] 2) Decision-making space

[0048] At any moment intelligent agent Based on local observations Make a decision Specifically, the decision-making of intelligent agents Define the decision space as follows: [Value of reactive power output command for its own photovoltaic inverter in the next time period]. Decision-making of each agent Mapped to its corresponding local decision space In, that is The decisions of all agents together constitute joint decision-making. ;

[0049] During the centralized training phase, each agent's local observations Together they form the global state vector Define the state space as Then the global state Mapped in the state space, i.e. reward function Define intelligent agents In global state Joint decision-making And cause the state to transition. The scalar feedback signal obtained subsequently maximizes the expected cumulative discount reward. It guides the updating and optimization of decisions made by various intelligent agents, among which This is the discount factor.

[0050] 3) Reward function

[0051] reward function Define intelligent agents In state Joint decision-making And cause the state to transition. The scalar feedback signal obtained subsequently. The reward function of agent i. The definition is as follows:

[0052]

[0053] in, This is the system reference voltage; For agents to perform joint decision-making Afterwards, its local observation area Internal photovoltaic node exist Voltage amplitude at any given moment; For network losses; These are weighting coefficients used to balance voltage regulation and network loss optimization objectives.

[0054] The above reward function By minimizing the voltage offset for the first objective The second objective is to minimize network loss. By applying negative weighting, the original problem of minimizing physical constraints is transformed into a problem of maximizing cumulative reward in multi-agent reinforcement learning. This mapping method ensures that while agents pursue high reward signals, they can simultaneously achieve coordinated optimization of distribution network voltage smoothing and operational loss reduction at the physical level. Based on the defined reward function, the expected cumulative discount reward is maximized. It guides the updating and optimization of policies for each intelligent agent, among which This is the discount factor.

[0055] Network losses of power grid The calculation is the sum of active power losses across all branches:

[0056]

[0057] in For the collection of distribution network branches, For photovoltaic nodes and photovoltaic nodes The resistance of the branch between For photovoltaic nodes and photovoltaic nodes The square of the current in the branch between.

[0058] Furthermore, in step S3, the multi-agent soft policy-comment solution and update steps are as follows:

[0059] The constructed multi-agent Markov decision process needs to be solved using a neural network with exploratory capabilities. This invention employs a multi-agent soft policy-evaluation algorithm based on a maximum entropy reinforcement learning framework. Each agent contains a policy network and an evaluation network, and the updates and optimizations of these two networks follow the process described below.

[0060] 1) Optimization objective of soft policy networks

[0061] Soft policy network objective function It is a linear combination that maximizes the expected cumulative return and the policy entropy:

[0062]

[0063] in, Represents the mathematical expectation. Indicates the experience pool in the buffer. Random sampling time global state , Indicates joint decision-making Obeying the policy network Defined random policy distribution; Discount factor; For the reward function; The temperature coefficient of entropy is used to control the randomness of the strategy; For soft policy networks In global state Entropy;

[0064] 2) Soft decision-value function

[0065] Evaluating the use of soft decision-value functions in networks The update is performed, and the update equation is:

[0066]

[0067] in, To evaluate the network parameters, and Representing time respectively and The global state; Represents the state transition probability The next state generated Expectations; Discount factor; The soft state-value function for the next time step is calculated as follows:

[0068]

[0069] in, Indicates the state Make a decision The logarithmic probability, Represents the mathematical expectation. In policy network sampling Joint decision-making at any time .

[0070] 3) Q-network loss function

[0071] The multi-agent soft policy-comment algorithm employs a double-Q network mechanism and introduces an online soft Q function. and and the corresponding objective Q function and ;

[0072] Q network loss function The objective is to minimize the mean square error between the online evaluation network's predicted value and the target value, which is defined as follows:

[0073]

[0074] Among them, E Indicates in the experience buffer pool Randomly selected empirical tuples, representing respectively Global state at any given moment, joint decision-making, rewards, and Global state at all times; To evaluate the decision-making value of network predictions online; The calculation method for the target decision value is as follows:

[0075]

[0076] in, for Time-based reward function; Discount factor; For the first A target evaluation network, This means taking the minimum of the two target decision values, thereby suppressing the overestimation of target decision values ​​during the training process; The temperature coefficient of entropy; Indicating soft policy networks in Make decisions at all times The probability density.

[0077] 4) Policy Network Update

[0078] Soft policy networks minimize the loss function This drives the soft policy network parameters toward a balance between expected cumulative return and policy entropy.

[0079]

[0080] Based on the above loss function, gradient ascent is used to adjust the gradient corresponding to the objective function of the soft policy network. Update, the update method is as follows:

[0081]

[0082] in, For parameters of the soft policy network, For parameters The gradient operator; The objective function of the soft policy network; To make decisions under the current strategy The logarithmic probability; To evaluate the value of network-predicted target decisions online, this gradient update formula aims to improve the parameters through the gradient ascent algorithm. Update in the direction of maximizing expected cumulative reward and policy entropy.

[0083] Furthermore, in step S4, the consensus mechanism steps for multiple agents are as follows:

[0084] Within the maximum entropy reinforcement learning framework, to ensure that the local learning of distributed agents can achieve global collaboration, this invention introduces a consensus mechanism. The consensus mechanism aims to enable each agent to... Local parameterized copy Convergence, parameterized replicas Defined as:

[0085]

[0086] in, To evaluate network parameters; For the first The policy network parameters of each agent; the consensus optimization problem is formalized as:

[0087]

[0088] in, For the first The local objective function of an agent without considering consensus constraints; The constraint requires each agent to maintain its own copy of local parameters. During the optimization iteration process, the parameters tend to be consistent globally, where st represents the constraint that ensures the consistency of these replica parameters.

[0089] This invention employs a stochastic consensus strategy, integrating equality constraints into the local objective through Lagrange relaxation and a quadratic penalty term to achieve distributed optimization. In this process, the global communication graph is determined using the connection relationships of physical photovoltaic nodes in the distribution network, and the adjacency matrix of the communication graph is used to... degree matrix Perform mathematical representation.

[0090] Local consensus Lagrangian function of agent i The definition is as follows:

[0091]

[0092] in, For intelligent agents The local update loss function is calculated as follows:

[0093]

[0094] The loss function is the Q-network loss function. For the first The loss function of the policy network of each agent; For intelligent agents Consensus punishment multiplier; It is a global state and joint decision-making As input, by parameters The output value of the determined nonlinear mapping function; Communication graph degree matrix The Kroneck block; For the adjacency matrix of the communication graph Kronecker block; quadratic penalty term through quantization of local output Information aggregated with neighborhood weights The deviation between them is penalized during training iterations to drive the agents to reach a global consensus in the feature representation space. For intelligent agents The local observation area.

[0095] The calculation method for the Kronecker block is as follows:

[0096]

[0097]

[0098] in, It is a parameterized copy of the agent. Identity matrices of the same dimension Communication graph degree matrix The diagonal elements, The adjacency matrix is ​​the first Line number Column elements. Through the Kronecker product operation, the scalar form of the communication topology weights is mapped to a high-dimensional neural network parameter space, thereby realizing the network parameters of each agent's soft policy. With evaluation network parameters Strict consistency constraints in distributed iterative processes.

[0099] Furthermore, in step S5, the constraint violation priority experience replay mechanism steps are as follows:

[0100] In power systems, constraint violations that lead to voltage out-of-bounds or line overloads are sparse anomalous samples. To address the issues of sample sparsity and low training efficiency, this invention establishes a constraint violation priority experience replay mechanism, comprising two modules: experience enhancement and priority calculation.

[0101] 1) Experience Enhancement Module

[0102] By leveraging the data symmetry of homogeneous agents, additional empirical tuples are generated through permutation operations, thereby improving the sample acquisition rate of multi-agent systems. .

[0103] set up Let be the set of all feasible permutation matrices. Let be any element of the permutation matrix. When the agent obtains a fundamental empirical tuple from the environment... At that time, new empirically enhanced tuples are generated through matrix multiplication. Store in the experience replay pool :

[0104]

[0105] in, right The elements of the vector are subjected to coaxial permutations in parallel components. Leveraging the physical symmetry of homogeneous agents, the permutation matrix is... By applying the data items of each agent to the corresponding positions in the vector in a synchronous manner to each component of the tuple, enhanced samples can be generated without changing the physical logic of the system.

[0106] 2) Priority Calculation Module

[0107] To prioritize the playback of samples with constraint violation risk, this invention calculates sampling priority based on the sample's timing difference error and voltage violation amount. .sample Sampling priority Defined as:

[0108]

[0109] in, sample The timing difference error; A small constant used to avoid zero priority; The timing difference error weight controls the contribution of timing difference error to priority. To constrain violations of weights; For photovoltaic nodes At any moment The voltage amplitude; These are the minimum and maximum allowable voltage amplitudes for photovoltaic nodes; Use the ReLU function to calculate voltage violations; This refers to the set of all photovoltaic nodes in the distribution network.

[0110] Examples of hyperparameters involved in this invention are shown in Table 1.

[0111] Table 1

[0112]

[0113] Both the soft policy network and the evaluation network consist of three fully connected feedforward layers. The weights of the linear layers are randomly initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.1.

[0114] Online simulation was performed based on the method of this invention, and the selected network specifications are shown in Table 2.

[0115] Table 2

[0116]

[0117] The simulation results for the first network specification are attached. Figure 3 and attached Figure 4 As shown in the attached figure, the simulation results for the second network specification are as follows. Figure 5 and attached Figure 6 As shown in Table 3, the simulation data are presented under the first and second network specifications mentioned above.

[0118] Table 3

[0119]

[0120] Based on simulation data provided in the embodiments of the present invention, this method demonstrates technical advantages and beneficial effects in distributed photovoltaic (PV) scheduling. Under different network scales, the present invention maintains the average voltage of PV nodes within the allowable offset. Within the specified range. Meanwhile, the average voltage out-of-bounds rates for the first and second network specifications are 0.773% and 1.997%, respectively. Despite the increased system scale and complexity, the method of this invention effectively reduces the out-of-bounds rate, avoiding the risk of voltage out-of-bounds errors caused by photovoltaic fluctuations. Finally, the average network loss of the system under the two specifications is optimized to 0.096MW and 0.451MW, respectively, achieving good economic operating indicators while ensuring voltage safety.

[0121] In summary, as the number of distributed photovoltaic units increased from 6 to 22, and the load scale expanded, this method maintained stable reward value performance and a low level of out-of-bounds behavior. This indicates that the random consensus penalty term introduced in this invention can ensure that each agent achieves a high degree of policy coordination and parameter consistency in a complex networked environment, effectively solving the non-stationary operation problem caused by high-penetration photovoltaic access and possessing engineering applicability.

[0122] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. Typical known structures and common knowledge techniques in the preferred embodiments have not been described in detail here. Those skilled in the art can improve and implement the technical solutions of this invention based on the guidance provided in these embodiments and their own capabilities. Some typical known structures, known methods or common knowledge techniques should not be obstacles for those skilled in the art to implement this application.

[0123] The scope of protection claimed in this application shall be determined by the contents of its claims, and the contents described in the invention description, specific embodiments and drawings shall be used to interpret the claims.

[0124] Within the scope of the technical concept of this application, several modifications can be made to the specific implementation of this application, and these modified implementations should also be considered within the protection scope of this application.

Claims

1. A distributed photovoltaic scheduling method based on multi-agent consensus optimization, characterized in that, For use in zoned power distribution networks containing distributed photovoltaic inverters, the following steps are included: Step S1: In having In a distribution network with nodes, including One photovoltaic node, Each photovoltaic inverter in each photovoltaic node is modeled as an independent intelligent agent, and a target model is constructed. A dual-objective optimization model that takes into account both the safety and economy of the distribution network is established, including the first objective of minimizing voltage deviation and the second objective of minimizing network loss. Step S2: Map the reactive power decision-making process of the distributed photovoltaic inverter into a networked multi-agent Markov decision process, and construct a mapping relationship connecting the dual objectives and the reward function. The operation of the entire network architecture is divided into two stages: a distributed execution stage and a centralized training stage. In the distributed execution stage, due to limitations in communication bandwidth and privacy protection, the agent's decision is based on its local observations. In the centralized training stage, the local observations of each agent collectively constitute the global state. The reward function transforms the original problem of minimizing physical constraints into the problem of maximizing cumulative rewards in multi-agent reinforcement learning by negatively weighting the first objective of minimizing voltage offset and the second objective of minimizing network loss. Step S3: Solve the multi-agent Markov decision process based on the multi-agent soft policy-evaluation algorithm to achieve policy learning and network parameter update with exploratory capabilities; adopt the multi-agent soft policy-evaluation algorithm based on the maximum entropy reinforcement learning framework. The agent contains a soft policy network and an evaluation network. The objective function of the soft policy network is to maximize the linear combination of the expected cumulative reward and the policy entropy. The evaluation network is updated using the soft decision-value function. Step S4: Design a consensus mechanism for multi-agent optimization, introduce a stochastic consensus penalty term to ensure the consistency of local policies and value function parameters of each agent in distributed learning; adopt a stochastic consensus strategy, integrate equality constraints into local objectives through Lagrange relaxation and a quadratic penalty term, and penalize inconsistencies in training iterations by quantifying the deviation between local output and neighborhood weighted aggregate information, thereby driving each agent to reach a global consensus in the feature representation space; Step S5: Using an experience enhancement module and a priority calculation module, a violation priority experience replay mechanism is constructed to improve the sampling efficiency of sparse and abnormal samples. The experience enhancement module utilizes the data symmetry of homogeneous agents to generate additional experience tuples through permutation operations. The priority calculation module calculates the sampling priority based on the temporal difference error and voltage violation of the samples.

2. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 1, characterized in that, In step S2, the steps for constructing the Markov decision process are as follows: Treating each photovoltaic inverter as an independent intelligent agent, during the distributed execution phase, the intelligent agent... Decision Based on its local observations Local observation Defined as an intelligent agent At any moment The collected and acquired local runtime data vectors: in, For intelligent agents Local observation area The node voltage magnitude vector; For local observation area The phase angle vector; and Local observation areas The vector of active and reactive power of the load; For intelligent agents Active power output of the photovoltaic node; For intelligent agents Reactive power output of the photovoltaic node; At any moment intelligent agent Based on local observations Make a decision Specifically, the decision-making of intelligent agents Define the decision space as follows: [Value of reactive power output command for its own photovoltaic inverter in the next time period]. Decision-making of each agent Mapped to its corresponding local decision space In, that is The decisions of all agents together constitute joint decision-making. ; During the centralized training phase, each agent's local observations Together they form the global state vector Define the state space as Then the global state Mapped in the state space, i.e. reward function Define intelligent agents In global state Joint decision-making And cause the state to transition. The scalar feedback signal obtained subsequently maximizes the expected cumulative discount reward. It guides the updating and optimization of decisions made by various intelligent agents, among which This is the discount factor.

3. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 2, characterized in that, To achieve the dual objectives of minimizing voltage offset and minimizing network loss, the reward function of agent i... The definition is as follows: in, This is the system reference voltage; For agents to perform joint decision-making Afterwards, its local observation area Internal photovoltaic node exist Voltage amplitude at any given moment; For network losses; These are weighting coefficients used to balance voltage regulation and network loss optimization objectives.

4. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 1, characterized in that, In step S3, the agent internally includes a soft policy network and an evaluation network. The steps for solving and updating the multi-agent soft policy-evaluation network are as follows: Soft policy network objective function It is a linear combination that maximizes the expected cumulative return and the policy entropy: in, Represents the mathematical expectation. Indicates the experience pool in the buffer. Random sampling time global state , Indicates joint decision-making Obeying the soft policy network Defined random policy distribution; Discount factor; For the reward function; The temperature coefficient of entropy is used to control the randomness of the strategy; For soft policy networks In global state Entropy; Evaluating the use of soft decision-value functions in networks The update is performed, and the update equation is: in, To evaluate the network parameters, and Representing time respectively and The global state; Represents the state transition probability The next state generated Expectations; Discount factor; The soft state-value function for the next time step is calculated as follows: in, Indicates the state Make a decision The logarithmic probability, Represents the mathematical expectation. In soft policy networks sampling Joint decision-making at any time .

5. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 4, characterized in that, The multi-agent soft policy-comment algorithm employs a double-Q network mechanism and introduces an online soft Q function. and and the corresponding objective Q function and ; Q network loss function The objective is to minimize the mean square error between the online evaluation network's predicted value and the target value, which is defined as follows: Among them, E Indicates in the experience buffer pool Randomly selected empirical tuples, representing respectively Global state at any given moment, joint decision-making, rewards, and Global state at all times; To evaluate the decision-making value of network predictions online; The calculation method for the target decision value is as follows: in, for Time-based reward function; Discount factor; For the first A target evaluation network, This means taking the minimum of the two target decision values, thereby suppressing the overestimation of target decision values ​​during the training process; The temperature coefficient of entropy; Indicating soft policy networks in Joint decision-making should be adopted at all times. The probability density.

6. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 5, characterized in that, Soft policy networks minimize the loss function This drives the soft policy network parameters toward a balance between expected cumulative return and policy entropy. Based on the above loss function, gradient ascent is used to adjust the gradient corresponding to the objective function of the soft policy network. Update, the update method is as follows: in, For parameters of the soft policy network, For parameters The gradient operator; The objective function of the soft policy network; To make decisions under the current strategy The logarithmic probability; To evaluate the value of network-predicted target decisions online, this gradient update formula aims to improve the parameters through the gradient ascent algorithm. Update in the direction of maximizing expected cumulative reward and policy entropy.

7. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 1, characterized in that, In step S4, the consensus mechanism of the multi-agent system proceeds as follows: The consensus mechanism aims to enable various intelligent agents to... Local parameterized copy Convergence, parameterized replicas Defined as: in, To evaluate network parameters; For the first The parameters of the soft policy network of individual agents; the consensus optimization problem is formalized as: in, For the first The local objective function of an agent without considering consensus constraints; This refers to the constraints requiring each agent to maintain its own copy of local parameters. During the optimization iteration process, the parameters tend to be consistent globally, where st represents the constraint that ensures the consistency of these replica parameters.

8. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 7, characterized in that, A stochastic consensus strategy is adopted, integrating equality constraints into the local objective through Lagrange relaxation and quadratic penalty terms. During this process, the global communication graph is determined using the connectivity relationships of physical photovoltaic nodes in the distribution network, and the adjacency matrix of the communication graph is used to... degree matrix Mathematical representation, intelligent agent Local consensus Lagrange function The definition is as follows: in, For intelligent agents The local update loss function is calculated as follows: The loss function is the Q-network loss function. For the first The loss function of a soft policy network for an agent; For intelligent agents Consensus punishment multiplier; It is a global state and joint decision-making As input, by parameters The output value of the determined nonlinear mapping function; Communication graph degree matrix The Kroneck block; For the adjacency matrix of the communication graph Kronecker block; quadratic penalty term through quantization of local output Information aggregated with neighborhood weights The deviation between them is penalized during training iterations to drive the agents to reach a global consensus in the feature representation space. For intelligent agents The local observation area; The calculation method for the Kronecker block is as follows: in, It is a parameterized copy of the agent. Identity matrices of the same dimension Communication graph degree matrix The diagonal elements, The adjacency matrix is ​​the first Line number Column elements; through the Kronecker product operation, the scalar form of the communication topology weights is mapped to the high-dimensional neural network parameter space, thereby realizing the parameters of the soft policy network of each agent. With evaluation network parameters Strict consistency constraints in distributed iterative processes.

9. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 1, characterized in that, In step S5, the constraint violation priority experience replay mechanism steps are as follows: The experience enhancement module leverages the data symmetry of homogeneous agents to generate additional experience tuples through permutation operations, thereby improving the sample acquisition rate of multi-agent systems. ; set up Let be the set of all feasible permutation matrices. For any element of the permutation matrix, when the agent obtains a basic empirical tuple in the environment... At that time, new empirically enhanced tuples are generated through matrix multiplication. Store in the experience replay pool : in, right The elements of the vector are subjected to coaxial permutations in parallel components. Leveraging the physical symmetry of homogeneous agents, the permutation matrix is... By applying the data items of each agent to the corresponding positions in the vector in a synchronous manner to each component of the tuple, enhanced samples can be generated without changing the physical logic of the system.

10. The distributed photovoltaic scheduling method based on multi-agent consensus optimization according to claim 9, characterized in that, The priority calculation module uses the timing difference error and voltage violation of the samples to calculate the sampling priority. ,sample Sampling priority Defined as: in, sample The timing difference error; A small constant used to avoid zero priority; The timing difference error weight controls the contribution of timing difference error to priority. To constrain violations of weights; For photovoltaic nodes At any moment The voltage amplitude; These are the minimum and maximum allowable voltage amplitudes for photovoltaic nodes; Use the ReLU function to calculate voltage violations; This refers to the set of all photovoltaic nodes in the distribution network.

Images