Photovoltaic system voltage control method, device and medium based on deep reinforcement learning

Abstract

The present application discloses a photovoltaic system voltage control method, device and medium based on deep reinforcement learning, the method comprising: constructing a Markov decision process model; training the Markov decision process model based on preset training data to obtain a voltage control strategy network, wherein each agent in the Markov decision process model represents a photovoltaic power generation device; obtaining actual observation data of the photovoltaic power generation system, inputting the actual observation data into the voltage control strategy network, and obtaining a first action strategy corresponding to each agent; performing safety constraint processing on the first action strategy corresponding to each agent, and obtaining a second action strategy corresponding to each agent; and obtaining a voltage control strategy for the photovoltaic power generation system according to the second action strategy corresponding to each agent. By performing safety correction on the first action strategy, the probability of voltage exceeding the limit due to unsafe actions applied to the power grid is reduced, and the safety and stability of the power grid are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of photovoltaic power generation system voltage control, and in particular to a photovoltaic system voltage control method, device and medium based on deep reinforcement learning. Background Art

[0002] With the depletion of fossil fuels and increasingly serious environmental problems, renewable energy such as photovoltaics has been vigorously developed. The access of photovoltaic renewable energy to the power grid helps to increase the proportion of clean energy, but it also brings new challenges to the operation of the power grid. In order to maintain the stability and safe operation of the power grid, the voltage control strategy of photovoltaic power generation system has become a hot research topic.

[0003] In photovoltaic power generation systems, voltage control of reactive power is crucial to maintaining the stability and reliability of the power grid. However, in actual applications, due to changes in light conditions and fluctuations in grid load, the voltage of photovoltaic equipment may fluctuate or exceed the safe range, resulting in system instability and increased risk of damage.

[0004] The voltage control strategy of photovoltaic power generation system is mainly based on deep reinforcement learning, especially multi-agent deep reinforcement learning. Multi-agent deep reinforcement learning can automatically extract useful expressions from high-dimensional raw data, without the need to establish the identification format of the actual system, reducing the dependence on accurate system models. However, in multi-agent deep reinforcement learning, the agent's exploration of actions is relatively random, and the actions output by the agent's deep strategy network are directly applied to the power grid, which may cause the voltage to exceed the limit, thus affecting the safety of the power grid. Summary of the invention

[0005] In view of this, the present invention provides a photovoltaic system voltage control method, device and medium based on deep reinforcement learning. The main purpose is to solve the problem that the intelligent agent in the existing photovoltaic system voltage control method explores the action randomly, and the determined action is directly applied to the power grid, which may cause the voltage to exceed the limit and affect the safety of the power grid.

[0006] According to one aspect of the present application, a photovoltaic system voltage control method based on deep reinforcement learning is provided, the method comprising:

[0007] Based on the voltage control task of photovoltaic system, a distributed partially observable Markov decision process model is constructed;

[0008] The distributed partially observable Markov decision process model is trained based on preset training data to obtain a voltage control strategy network, wherein each agent in the distributed partially observable Markov decision process model represents a photovoltaic power generation device, and during training, the agent interacts with a preset power grid simulation environment for feedback and rewards;

[0009] Acquire actual observation data and actual state data of the photovoltaic power generation system to be adjusted, input the actual observation data into the voltage control strategy network, and obtain a first action strategy corresponding to each intelligent agent;

[0010] Based on the actual state data, a first action strategy corresponding to each intelligent agent is subjected to safety constraint processing to obtain a second action strategy corresponding to each intelligent agent;

[0011] According to the second action strategy corresponding to each of the intelligent agents, a voltage control strategy of the photovoltaic power generation system is obtained.

[0012] Optionally, performing safety constraint processing on the first action strategy corresponding to each agent based on the actual state data to obtain the second action strategy corresponding to each agent includes:

[0013] Based on the actual state data, the first action strategy corresponding to each intelligent agent and the trained sensitivity network, constructing the optimization constraint conditions of the safety constraint model;

[0014] Constructing an optimization objective function of the safety constraint model, solving the optimization objective function based on the optimization constraint conditions, and obtaining a second action strategy corresponding to each intelligent agent if the optimization objective function has a solution;

[0015] If the optimization objective function has no solution, the solution of the optimization objective function is calculated based on the penalty constraint and the relaxation constraint to obtain the second action strategy corresponding to each intelligent agent.

[0016] Optionally, the optimization objective function of the safety constraint model is:

[0017]

[0018] The optimization constraints of the safety constraint model are:

[0019]

[0020] Among them, a is the first action strategy corresponding to the agent, is the joint strategy of all agents, N is the number of agents, c j (s) is the constraint function in the current state, c j is the constraint limit, h(s|u j ) is the sensitivity network in the current state, u j is the weight.

[0021] Optionally, the method of calculating the solution of the optimization objective function based on the penalty constraint and the relaxation constraint to obtain the second action strategy corresponding to each agent is:

[0022]

[0023] Where ε={ε1,...,ε Z} is a slack variable, ρ is the penalty weight for violating the constraint, and a * ,ε * are the optimal safety action and the optimal slack variable respectively.

[0024] Optionally, the distributed partially observable Markov decision process model includes a sextuple represented by (S, A, O, Ω, T, r), wherein S is used to represent the state, O is used to represent the observation, A is used to represent the action, r is used to represent the reward, T is used to represent the state transition, and Ω is used to represent the observation transition.

[0025] Optionally, the distributed partially observable Markov decision process model is trained based on preset training data to obtain a voltage control strategy network, including:

[0026] Acquire group training data, and use the training data to train the distributed partially observable Markov decision process model to generate an initial action strategy at a current moment;

[0027] Perform security constraint processing on the initial action strategy to obtain the constrained action strategy;

[0028] Inputting the constrained action strategy into the power grid simulation environment, so that the power grid simulation environment updates the state according to the initial action strategy to obtain the next state data and the next observation data at the next moment;

[0029] Calculate the reward value at the current moment based on the state data at the next moment;

[0030] The agent is continuously trained according to the next state data, the next observation data and the reward value at the current moment until a preset stop condition is reached, thereby obtaining a voltage control strategy network.

[0031] Optionally, the reward value at the current moment is calculated using the following method:

[0032]

[0033] Where N is the number of nodes in the power grid simulation environment, f v (·) is the voltage barrier function, v k is the voltage amplitude at node k, v ref is the reference voltage, a, b, c, d>0 are hyperparameters used to adjust the function shape and smoothness, l q (q PV) is the reactive power generated by the photovoltaic device to regulate the voltage, K is the number of photovoltaic devices in the power grid simulation environment, q PV is the reactive power emitted by the photovoltaic device inverter at the previous moment, and α∈(0,1) is the Lagrange multiplier.

[0034] According to another aspect of the present application, a photovoltaic system voltage control device based on deep reinforcement learning is provided, comprising:

[0035] Model building module, which is used to build a distributed partially observable Markov decision process model based on the voltage control task of the photovoltaic system;

[0036] A training module, used for training the distributed partially observable Markov decision process model based on preset training data to obtain a voltage control strategy network, wherein each agent in the distributed partially observable Markov decision process model represents a photovoltaic power generation device, and during training, the agent interacts with a preset power grid simulation environment for feedback and rewards;

[0037] A first action strategy acquisition module is used to obtain actual observation data and actual state data of the photovoltaic power generation system to be adjusted, input the actual observation data into the voltage control strategy network, and obtain a first action strategy corresponding to each intelligent agent;

[0038] A second action strategy acquisition module is used to perform safety constraint processing on the first action strategy corresponding to each intelligent agent based on the actual state data to obtain a second action strategy corresponding to each intelligent agent;

[0039] The voltage control strategy acquisition module is used to obtain the voltage control strategy of the photovoltaic power generation system according to the second action strategy corresponding to each of the intelligent agents.

[0040] Optionally, the second action strategy obtaining module is further used for:

[0041] Based on the actual state data, the first action strategy corresponding to each intelligent agent and the trained sensitivity network, constructing the optimization constraint conditions of the safety constraint model;

[0042] Constructing an optimization objective function of the safety constraint model, solving the optimization objective function based on the optimization constraint conditions, and obtaining a second action strategy corresponding to each intelligent agent if the optimization objective function has a solution;

[0043] If the optimization objective function has no solution, the solution of the optimization objective function is calculated based on the penalty constraint and the relaxation constraint to obtain the second action strategy corresponding to each agent.

[0044] According to another aspect of the present application, a storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned photovoltaic system voltage control method based on deep reinforcement learning.

[0045] According to another aspect of the present application, a computer device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0046] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned photovoltaic system voltage control method based on deep reinforcement learning.

[0047] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0048] The present application provides a photovoltaic system voltage control method, device and medium based on deep reinforcement learning, which converts the voltage control problem of the photovoltaic system into a distributed partially observable Markov decision process, trains the distributed partially observable Markov decision process model based on preset training data, obtains a voltage control strategy network, inputs the actual observation data of the photovoltaic power generation system to be adjusted into the voltage control strategy network, obtains a first action strategy corresponding to each intelligent agent, performs safety constraint processing on the first action strategy corresponding to each intelligent agent based on the actual state data of the photovoltaic power generation system to be adjusted, obtains a second action strategy corresponding to each intelligent agent, obtains the voltage control strategy of the photovoltaic power generation system according to the second action strategy corresponding to each intelligent agent, performs voltage control on the photovoltaic power generation system, and reduces the probability of voltage exceeding the limit due to unsafe actions applied to the power grid by correcting the safety constraints of the first action strategy, thereby improving the safety, voltage stability and reliability of the power grid.

[0049] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Also, the same reference symbols are used throughout the accompanying drawings to indicate the same components. In the accompanying drawings:

[0051] Figure 1A flowchart of a photovoltaic system voltage control method based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0052] Figure 2 A flowchart of another photovoltaic system voltage control method based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of the interaction process between deep reinforcement learning and a power grid simulation environment of a photovoltaic system voltage control method based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0054] Figure 4 A multi-agent reinforcement learning schematic diagram of a photovoltaic system voltage control method based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0055] Figure 5 A schematic diagram of a multi-agent deep deterministic policy gradient algorithm with safety constraints for a photovoltaic system voltage control method based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0056] Figure 6 A block diagram of a photovoltaic system voltage control device based on deep reinforcement learning provided in an embodiment of the present application is shown.

[0057] in,

[0058] Figure 6 In: 602 - model building module; 604 - training module; 606 - first action strategy acquisition module; 608 - second action strategy acquisition module; 610 - voltage control strategy acquisition module. DETAILED DESCRIPTION

[0059] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0060] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention application are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form.

[0061] In view of the problem that the intelligent agent in the existing photovoltaic system voltage control method explores actions randomly, and the determined actions are directly applied to the power grid, which may cause the voltage to exceed the limit and affect the safety of the power grid, the embodiment of the present application provides a photovoltaic system voltage control method based on deep reinforcement learning, such as Figure 1 As shown, the method includes:

[0062] 102: Based on the voltage control task of photovoltaic system, a distributed partially observable Markov decision process model is constructed;

[0063] 104: training the distributed partially observable Markov decision process model based on preset training data to obtain a voltage control strategy network, wherein each agent in the distributed partially observable Markov decision process model represents a photovoltaic power generation device, and during training, the agent interacts with a preset power grid simulation environment for feedback and rewards;

[0064] 106: Acquire actual observation data and actual state data of the photovoltaic power generation system to be adjusted, input the actual observation data into the voltage control strategy network, and obtain a first action strategy corresponding to each intelligent agent;

[0065] 108: Based on the actual state data, perform safety constraint processing on the first action strategy corresponding to each intelligent agent to obtain a second action strategy corresponding to each intelligent agent;

[0066] 110: Obtain a voltage control strategy for the photovoltaic power generation system according to the second action strategy corresponding to each of the intelligent agents.

[0067] Specifically, by establishing a grid simulation environment for reactive power continuously adjustable photovoltaic equipment voltage control, including photovoltaic equipment, grid network and related sensors, the grid simulation environment is ensured to be consistent with the actual scenario and to provide accurate state observation and action execution. The model for reactive power continuously adjustable photovoltaic equipment voltage control is built as a distributed partially observable Markov decision process model. Each agent in the distributed partially observable Markov decision process model represents a photovoltaic device, and each agent has a partially observable state. In order to enhance the security of the power grid, all agents in the area share the observation of the area, and realize the collaboration and interaction between multiple agents to effectively control the voltage of the photovoltaic equipment.

[0068] The historical operation data sampled from the regional power grid is used as training data. The distributed partially observable Markov decision process model is trained based on the training data. After the training is completed, a voltage control strategy network suitable for reactive continuous adjustable photovoltaic equipment is generated. The actual observation data and actual state data of the photovoltaic power generation system to be adjusted are obtained, and the actual observation data are input into the voltage control strategy network to obtain the first action strategy corresponding to each intelligent agent. The first action strategy output by the voltage control strategy network is corrected by the safety constraint to obtain the second action strategy corresponding to each intelligent agent. According to the second action strategy corresponding to each intelligent agent, the voltage control strategy of the photovoltaic power generation system is obtained. The second action strategy output by the intelligent agent is the ratio of the reactive power generated by the photovoltaic device to its maximum reactive power. When performing the flow calculation, this value will be multiplied by the maximum reactive power and converted into the reactive power generated by the photovoltaic device, and used as the input of the flow calculation. Through the collaboration and learning between intelligent agents, a voltage control strategy that meets the safety constraints is generated to solve the technical problems existing in the voltage control of reactive continuous adjustable photovoltaic equipment, so that the reactive power continuously adjustable photovoltaic device can operate more efficiently and reliably in different operating scenarios.

[0069] Compared with the prior art, the present application provides a photovoltaic system voltage control method based on deep reinforcement learning, which converts the voltage control problem of the photovoltaic system into a distributed partially observable Markov decision process, trains the distributed partially observable Markov decision process model based on preset training data, and obtains a voltage control strategy network, inputs the actual observation data of the photovoltaic power generation system to be adjusted into the voltage control strategy network, obtains the first action strategy corresponding to each intelligent agent, and performs safety constraint processing on the first action strategy corresponding to each intelligent agent based on the actual state data of the photovoltaic power generation system to be adjusted to obtain the second action strategy corresponding to each intelligent agent, and obtains the voltage control strategy of the photovoltaic power generation system according to the second action strategy corresponding to each intelligent agent, and performs voltage control on the photovoltaic power generation system. By correcting the safety constraints of the first action strategy, the probability of voltage exceeding the limit due to unsafe actions applied to the power grid is reduced, thereby improving the safety, voltage stability and reliability of the power grid.

[0070] In one embodiment of the present invention, Figure 2 As shown, based on the actual state data, the first action strategy corresponding to each intelligent agent is subjected to safety constraint processing to obtain the second action strategy corresponding to each intelligent agent, including:

[0071] 202: Based on the actual state data, the first action strategy corresponding to each agent and the trained sensitivity network, construct the optimization constraint conditions of the safety constraint model;

[0072] 204: Construct an optimization objective function of the safety constraint model, solve the optimization objective function based on the optimization constraint conditions, and if the optimization objective function has a solution, obtain the second action strategy corresponding to each intelligent agent;

[0073] 206: If the optimization objective function has no solution, based on the penalty constraint and the relaxation constraint, the solution of the optimization objective function is calculated to obtain the second action strategy corresponding to each intelligent agent.

[0074] Specifically, a safety signal model is first established, and the voltage constraint function (which can be regarded as taking the state as the independent variable) is transformed into a set of linear constraints related to the state and action through first-order approximation. The safety signal model is:

[0075] c j (s')≈c j (s)+h(s|μ j ) T as′ represents the new state after applying action a in state s, h(s|μ j ) is the sensitivity network, μ j are weights, and the neural network learns the sensitivity of the constraints to the action in a given state.

[0076] Before the safety signal model is established, the sensitivity network needs to be trained. Using the power grid simulation environment, in the current state, the sensitivity agent (different from the agent of the photovoltaic device used for voltage control) randomly takes action a. After being applied to the power grid, the power grid transfers to the next state s' (the power flow calculation can be used to simulate the environmental transfer process), and the state transfer data group (s, a, s') is obtained. By minimizing the loss function Train the sensitivity network.

[0077] After the sensitivity network training is completed, the safety signal model is based on the value c of the constraint function in the current state. j (s) and the constraint limit C j , the calculation result is the variation range of action a, that is, the action set.

[0078] According to the action set, the safety constraint model is converted into the optimization constraint conditions of the safety constraint model, and the optimization objective function of the safety constraint model is constructed. The objective function is to minimize the Euclidean distance between the action vector output by the strategy network and the safety action vector. Based on the optimization constraint conditions, the optimization objective function is solved.

[0079] Solving the safety layer is a convex quadratic optimization problem, which calculates the minimum distance projection of the action output by the voltage control strategy network on the safety action set (where the safety action set is calculated based on the above-mentioned linearized safety signal model, that is, the constraint condition). This is the action that meets the safety constraints, which serves as the first action strategy corresponding to each intelligent agent, thereby obtaining an action that meets the safety constraints.

[0080] When the original quadratic optimization problem (hereinafter referred to as the hard-constrained quadratic optimization problem) is infeasible, that is, when there is no suitable solution, a soft-constrained quadratic optimization problem is introduced to find a solution by penalizing constraint relaxation. Slack variables are introduced to the constraints of the hard-constrained quadratic optimization problem, and the penalty factor is added to the optimization objective of the soft-constrained quadratic optimization problem. The optimization objective of the soft-constrained quadratic optimization problem is to minimize the action value and the value of the slack variable. If the value of the penalty factor is larger than the optimal Lagrange multiplier corresponding to the hard-constrained optimization problem, the soft-constrained problem and the hard-constrained problem are almost equivalent, and it can ensure that when multiple constraints cannot be satisfied at the same time, only minor violations of the constraints are made.

[0081] The optimization objective function of the safety constraint model is:

[0082]

[0083] The optimization constraints of the safety constraint model are:

[0084]

[0085] Among them, a is the first action strategy corresponding to the agent, is the joint strategy of all agents, N is the number of agents, c j (s) is the constraint function in the current state, c j is the constraint limit, h(s|u j ) is the sensitivity network in the current state, u j is the weight.

[0086] Based on the penalty constraint and the relaxation constraint, the solution of the optimization objective function is calculated to obtain the second action strategy corresponding to each agent as follows:

[0087]

[0088] Where ε={ε1,...,ε Z} is a slack variable, ρ is the penalty weight for violating the constraint, and a * ,ε * are the optimal safety action and the optimal slack variable respectively.

[0089] In one embodiment of the present invention, the distributed partially observable Markov decision process model includes a sextuple represented by (S, A, O, Ω, T, r), wherein S is used to represent the state, O is used to represent the observation, A is used to represent the action, r is used to represent the reward, T is used to represent the state transition, and Ω is used to represent the observation transition.

[0090] Specifically, mathematically, the distributed partially observable Markov decision process model includes a sextuplet represented by (S, A, O, Ω, T, r).

[0091] State set: The state set includes the set of active power and reactive power of the load, including the set of active power emitted by PV (photovoltaic power generation equipment), the set of reactive power emitted by PV inverter (photovoltaic inverter) in the previous step, the voltage set, and the voltage phase vector set measured in radius. The voltage amplitude set, voltage phase set, load active power and reactive power set in the state set all include all nodes in the power grid. State set in, It is the combination of load active power and reactive power. It is the collection of active power generated by PV (photovoltaic equipment). is the set of reactive power generated by the PV (photovoltaic device) inverter in the previous step, V = {(v,θ):v∈(0,∞),θ∈[-π,π]} is the voltage set, v is the voltage magnitude vector, and θ is the voltage phase vector measured in units of radius. i , are vectors v and p respectively. L ,q L , p PV ,q PV The amount.

[0092] Observation set, each component measures the quantity measurement of the area where a single agent is located, including the active power set emitted by the PV in the area where the agent is located, the reactive power set emitted by the PV in the previous step in the area where the agent is located, the node voltage amplitude set in the area where the agent is located, and the node voltage phase set in the area where the agent is located; the observation set is defined as O = × i∈ I O i , Represents the quantity measurement of the area where the i-th agent is located; the observation set is a subset of the state set. Due to the limited measurement range of a single device and the need to reduce the input dimension of the agent strategy network, each agent is set to only obtain the quantity measurement of the area where it is located, which is different from the corresponding quantity measurement dimension in the state set. The union of each agent's observation set is the observation set.

[0093] Action set: Each agent There is a continuous action set The continuous action represents the ratio of the reactive power emitted by the agent to its maximum reactive power, distinguishing between PV inverters that inject reactive power into the distribution network and those that absorb reactive power. The reactive power output of the kth PV inverter (i.e., the kth agent) is in, is the maximum apparent power of the kth node determined by the physical capacity of the PV inverter. k >0, reactive power is injected into the distribution network; if a k <0, reactive power is absorbed from the distribution network. The value of c is usually selected according to the load capacity of the distribution network to ensure the safe and reliable operation of the power grid. The joint action set is expressed as A = × i∈I A i .

[0094] State transition probability function: The state transition probability function describes the dynamics of the environment. Define the state transition probability function that conforms to the Markov decision process In this problem, in the current state, the photovoltaic equipment outputs reactive power through the inverter according to the strategy, which changes the power flow distribution of the power grid. The node voltage changes accordingly and generates network losses. The photovoltaic active output and load state fluctuate randomly, and the power grid transfers to the next state.

[0095] Observation probability function: In the context of power systems, the observation probability function It describes the probability distribution of the measured values ​​of the power system under a given real state, describes the possible measurement errors of the sensors, and uses the probability density function to represent the observation probability function.

[0096] Reward function: r is a global reward function that describes the reward of all agents for a given decision. The goal of the agent’s action is to control the voltage to v ref The voltage deviation from the reference value and the reactive power emitted by the PV for voltage regulation are added to the reward function as penalty items, that is, the reward function is negative.

[0097] In one embodiment of the present invention, a distributed partially observable Markov decision process model is trained based on preset training data to obtain a voltage control strategy network, including:

[0098] Obtain group training data, use the training data to train a distributed partially observable Markov decision process model, and generate an initial action strategy at the current moment;

[0099] Perform security constraint processing on the initial action strategy to obtain the constrained action strategy;

[0100] Input the constrained action strategy into the power grid simulation environment, so that the power grid simulation environment updates the state according to the initial action strategy, and obtains the next state data and the next observation data at the next moment;

[0101] Calculate the reward value at the current moment based on the state data at the next moment;

[0102] The agent continues to be trained according to the next state data, the next observation data and the reward value at the current moment until the preset stopping condition is reached to obtain the voltage control strategy network.

[0103] Specifically, the power grid simulation environment is constructed by establishing a simulation environment for reactive and continuously adjustable photovoltaic equipment voltage control, including photovoltaic equipment, power grid network and related sensors. Select appropriate photovoltaic equipment according to actual conditions, and install and connect them according to specifications to ensure the correct installation and connection of the equipment, so as to accurately obtain the status information and control capabilities of the photovoltaic equipment, and ensure the communication and data transmission between the photovoltaic equipment and the power grid, which can be achieved by connecting with inverters or other equipment in the power grid to monitor the status and requirements of the power grid in real time; deploy relevant sensors in photovoltaic equipment and power grid networks to obtain various status information in the environment. Set up a data acquisition system to regularly collect data obtained by sensors, and process and analyze them. It includes steps such as data storage, data cleaning, and feature extraction to obtain accurate status observations.

[0104] Construct the target network and deep policy network of the distributed partially observable Markov decision process model, learn and generate voltage control strategies, the target network calculates the target value function to estimate the expected long-term return, and the deep policy network is used to learn and generate the action strategy of the intelligent agent; including the following steps:

[0105] Based on the Actor-Critic architecture of the MADDPG (Distributed Partially Observable Markov Decision Process) algorithm, a centralized Q network is constructed as the Critic network. The centralized Q network uses the observations and actions of multiple agents as input, calculates through a multi-layer neural network, and outputs the corresponding action-value function; at each time step, the agent observes the current state and selects an action to perform. After performing the action, the agent interacts with the environment, such as Figure 3 and Figure 4 As shown, the next state and the corresponding reward signal are obtained, and the target value is calculated using the next state, which can be achieved by selecting the optimal action from the next state and obtaining its corresponding value; the predicted value of the action-value function is calculated using the current state and the executed action, and the loss value is obtained by calculating the difference between the predicted value and the target value. The update of the action-value function is performed by minimizing the loss of the value network;

[0106] When defining the agent's reward function, the present invention introduces a voltage barrier function based on voltage constraints derived from traditional optimization models. This ensures that the agent considers voltage limitations when optimizing its control actions.

[0107] The stochastic gradient descent method is applied to iteratively update the weights of the policy network and the target neural network, where the weights in the target neural network are updated in real time, while the weights in the policy network are updated at set time steps.

[0108] In the multi-agent deep deterministic policy gradient algorithm, the non-stationarity of the environment is addressed by utilizing a series of centralized Q-networks that use complete state and action information to approximate the optimal Q-value function for each agent. The policy (Actor) network of agent i consists of a network with parameters The control law of the intelligent agent with Gaussian noise can be expressed as in represents the policy network, represents Gaussian noise, is the standard deviation, which is a parameter representing exploration. Defined as the value function of the initial state of the event

[0109] The policy network is updated by gradient ascent, which moves the gradient direction of the policy. This process is the process of policy improvement and can be regarded as maximizing the action-value function. The analytical form of the gradient is expressed as Among them is the playback pool that stores historical experience, is the action of the other agents. At each time step, the policy and value network of each agent are updated by uniformly sampling small batches from the buffer. This allows the algorithm to learn from a series of unrelated experiences to stabilize the learning process. Applying the chain rule, the gradient can be decomposed into the gradient of the action-value with respect to the action and the gradient of the policy with respect to the policy parameters.

[0110]

[0111] Action-Value It is a centralized policy evaluation function that considers not only the agent’s own actions but also the actions of other agents, which creates a stable environment for each agent that is not affected by policy changes.

[0112] Value (Critic) Network The action-value function is approximately parameterized by a neural network, and the update of the action-value function is performed by minimizing the following loss in The weight of the value network;

[0113] Introducing the target policy network and target value network The target network is a copy of the original policy and value network's parameters at an earlier time. and are the weights of the target strategy network and the target value network respectively. It is the Critic network The reference value that we want to track during training, estimated by the target policy network and the target value network. Thus, Stabilize and replace with target network By making the target network slowly track the learning network, the weights of the target network of agent i are updated

[0114] For specific training methods, see Figure 5 , the training method is as follows:

[0115] a. Define the structure of the policy network and target neural network (calculation value), using the neural network model in deep reinforcement learning;

[0116] b. Use the collected data as training samples, including observations, actions taken, reward signals, etc.; c. Initialize the weights of the policy network;

[0117] d. For each training cycle (epoch):

[0118] e. In each time step, based on the current state observation, use the policy network to generate actions, impose safety constraints on the actions, input the constrained actions into the power grid simulation environment, and interact with the power grid simulation environment;

[0119] f. After interacting with the power grid simulation environment, new states, observations, actions, and reward signals are collected; g. The policy network is used to calculate the probability distribution of actions and sample new actions based on the probability of action selection;

[0120] h. Calculate the target value function using the target neural network;

[0121] i. Calculate the loss function of the policy network;

[0122] j. Use stochastic gradient descent method to update the weights of the policy network and minimize the loss function;

[0123] k. Update the weights of the target neural network, such as by soft updating or hard updating;

[0124] l. Repeat the training cycle until the set number of iterations is reached to obtain the trained intelligent agent. After reaching the set number of iterations, stop the training process and save the weights and model parameters of the trained policy network.

[0125] Through the above implementation method, the collected data is used for training and a voltage control strategy is generated. The weight update and optimization during the training process use the stochastic gradient descent method, combined with the real-time update of the target neural network and the regular update of the policy network. After the training is completed, the trained agent can be applied to the actual environment to obtain the voltage control strategy to achieve voltage control of reactive continuously adjustable photovoltaic equipment.

[0126] In one embodiment, the following method is used to calculate the reward value at the current moment:

[0127]

[0128]

[0129] Where N is the number of nodes in the power grid simulation environment, f v (·) is the voltage barrier function, v k is the voltage amplitude at node k, v ref is the reference voltage, a, b, c, d>0 are hyperparameters used to adjust the function shape and smoothness, l q (q PV ) is the reactive power generated by the photovoltaic device to regulate the voltage, K is the number of photovoltaic devices in the power grid simulation environment, q PV is the reactive power emitted by the photovoltaic device inverter at the previous moment, and α∈(0,1) is the Lagrange multiplier.

[0130] Specifically, the reward function is a global reward function that describes the rewards of all agents for a given decision. The goal of the agent’s action is to control the voltage to v ref A safe range near the target while generating as little reactive power as possible. The reward function is defined as Among them, f v (·) is the voltage barrier function, is the reactive power generated by the photovoltaic device to regulate the voltage. α∈(0,1) is the Lagrange multiplier that transforms the constrained reward into an unconstrained reward.

[0131] The requirement that the voltage is always within the safe range sets a constraint for the control. This constraint is characterized by a voltage barrier function and added to the reward function. Considering that the reward function contains multiple optimization objectives, if the voltage barrier function is in the form of where v k is the voltage amplitude at node k. Then, in order to smooth out small voltage fluctuations within the voltage safety range, PV will generate unnecessary reactive power. If the voltage barrier function is in the form of When the voltage exceeds the safe range, the agent will also receive a small penalty, so the speed of guiding the agent to give a strategy within the safe range will be very slow. To solve the above problems, a segmented voltage barrier function is set Among them, a, b, c, d>0 are hyperparameters used to adjust the shape and smoothness of the function, v ref This voltage barrier function gives a steeper gradient outside the voltage safety range, so that the strategy given by the intelligent agent is close to the voltage safety range. re f has a gentler gradient, which reduces the control cost of making the voltage close to the reference value. In the active voltage control scenario, each agent cooperates to find an optimal joint strategy Maximize the discount accumulative reward Among them, γ∈(0,1) is the discount factor, which determines the importance of future rewards. The closer the discount factor is to 1, the greater the impact of future rewards on current decisions, which is conducive to learning to find the optimal strategy in the long run; the closer the discount factor is to 0, the more the learner values ​​current rewards.

[0132] Furthermore, as a response to the above Figure 1 The implementation of the method shown in the figure, the embodiment of the present invention provides a photovoltaic system voltage control device based on deep reinforcement learning, such as Figure 6 As shown, the device comprises:

[0133] A model building module 602 is used to build a distributed partially observable Markov decision process model based on a voltage control task of a photovoltaic system;

[0134] A training module 604 is used to train the distributed partially observable Markov decision process model based on preset training data to obtain a voltage control strategy network, wherein each agent in the distributed partially observable Markov decision process model represents a photovoltaic power generation device, and during training, the agent interacts with a preset power grid simulation environment for feedback and rewards;

[0135] A first action strategy acquisition module 606 is used to obtain actual observation data and actual state data of the photovoltaic power generation system to be adjusted, input the actual observation data into the voltage control strategy network, and obtain a first action strategy corresponding to each intelligent agent;

[0136] A second action strategy acquisition module 608 is used to perform safety constraint processing on the first action strategy corresponding to each intelligent agent based on the actual state data to obtain a second action strategy corresponding to each intelligent agent;

[0137] The voltage control strategy acquisition module 610 is used to obtain the voltage control strategy of the photovoltaic power generation system according to the second action strategy corresponding to each of the intelligent agents.

[0138] The present application provides a photovoltaic system voltage control device based on deep reinforcement learning. Compared with the prior art, the voltage control problem of the photovoltaic system is converted into a distributed partially observable Markov decision process, and the distributed partially observable Markov decision process model is trained based on preset training data to obtain a voltage control strategy network. The actual observation data of the photovoltaic power generation system to be adjusted is input into the voltage control strategy network to obtain a first action strategy corresponding to each intelligent agent. Based on the actual state data of the photovoltaic power generation system to be adjusted, the first action strategy corresponding to each intelligent agent is subjected to safety constraint processing to obtain a second action strategy corresponding to each intelligent agent. According to the second action strategy corresponding to each intelligent agent, the voltage control strategy of the photovoltaic power generation system is obtained, and the voltage of the photovoltaic power generation system is controlled. By correcting the safety constraints of the first action strategy, the probability of voltage exceeding the limit due to unsafe actions applied to the power grid is reduced, thereby improving the safety, voltage stability and reliability of the power grid.

[0139] In one embodiment, based on the actual state data, the first action strategy corresponding to each agent is subjected to safety constraint processing to obtain the second action strategy corresponding to each agent, including:

[0140] Based on the actual state data, the first action strategy corresponding to each agent and the trained sensitivity network, the optimization constraints of the safety constraint model are constructed;

[0141] Construct the optimization objective function of the safety constraint model, and solve the optimization objective function based on the optimization constraints. If the optimization objective function has a solution, obtain the second action strategy corresponding to each intelligent agent;

[0142] If the optimization objective function has no solution, the solution of the optimization objective function is calculated based on the penalty constraint and the relaxation constraint to obtain the second action strategy corresponding to each agent.

[0143] In one embodiment, the optimization objective function of the safety constraint model is:

[0144]

[0145] The optimization constraints of the safety constraint model are:

[0146]

[0147] Among them, a is the first action strategy corresponding to the agent, is the joint strategy of all agents, N is the number of agents, c j(s) is the constraint function in the current state, c j is the constraint limit, h(s|u j ) is the sensitivity network in the current state, u j is the weight.

[0148] In one embodiment, based on the penalty constraint and the relaxation constraint, a solution of the optimization objective function is calculated to obtain the second action strategy corresponding to each agent as follows:

[0149]

[0150] Where ε={ε1,...,ε Z} is a slack variable, ρ is the penalty weight for violating the constraint, and a * ,ε * are the optimal safety action and the optimal slack variable respectively.

[0151] In one embodiment, the distributed partially observable Markov decision process model includes a sextuple represented by (S, A, O, Ω, T, r), where S is used to represent the state, O is used to represent the observation, A is used to represent the action, r is used to represent the reward, T is used to represent the state transition, and Ω is used to represent the observation transition.

[0152] In one embodiment, a distributed partially observable Markov decision process model is trained based on preset training data to obtain a voltage control strategy network, including:

[0153] Obtain group training data, use the training data to train a distributed partially observable Markov decision process model, and generate an initial action strategy at the current moment;

[0154] Perform security constraint processing on the initial action strategy to obtain the constrained action strategy;

[0155] Input the constrained action strategy into the power grid simulation environment, so that the power grid simulation environment updates the state according to the initial action strategy, and obtains the next state data and the next observation data at the next moment;

[0156] Calculate the reward value at the current moment based on the state data at the next moment;

[0157] The agent continues to be trained according to the next state data, the next observation data and the reward value at the current moment until the preset stopping condition is reached to obtain the voltage control strategy network.

[0158] In one embodiment, the following method is used to calculate the reward value at the current moment:

[0159]

[0160] Where N is the number of nodes in the power grid simulation environment, f v (·) is the voltage barrier function, v k is the voltage amplitude at node k, v ref is the reference voltage, a, b, c, d>0 are hyperparameters used to adjust the function shape and smoothness, l q (q PV ) is the reactive power generated by the photovoltaic device to regulate the voltage, K is the number of photovoltaic devices in the power grid simulation environment, q PV is the reactive power emitted by the photovoltaic device inverter at the previous moment, and α∈(0,1) is the Lagrange multiplier.

[0161] According to one embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the photovoltaic system voltage control method based on deep reinforcement learning in any of the above method embodiments.

[0162] According to a schematic diagram of the structure of a computer device provided by an embodiment of the present invention, the specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0163] The computer device may include: a processor, a communication interface, a memory, and a communication bus.

[0164] Wherein: the processor, the communication interface, and the memory communicate with each other via a communication bus.

[0165] Communication interface, used to communicate with other devices such as client or other server network elements.

[0166] The processor is used to execute the program, and specifically can execute the relevant steps in the above-mentioned photovoltaic system voltage control method embodiment based on deep reinforcement learning.

[0167] Specifically, the program may include program codes including computer operation instructions.

[0168] The processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0169] The memory is used to store programs. The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0170] The program can specifically be used to enable the processor to execute the photovoltaic system voltage control method based on deep reinforcement learning in any of the above method embodiments.

[0171] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. In one embodiment, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0172] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A photovoltaic system voltage control method based on deep reinforcement learning, characterized in that: include: Based on the voltage control task of photovoltaic system, a distributed partially observable Markov decision process model is constructed; The distributed partially observable Markov decision process model is trained based on preset training data to obtain a voltage control strategy network, wherein each agent in the distributed partially observable Markov decision process model represents a photovoltaic power generation device, and during training, the agent interacts with a preset power grid simulation environment for feedback and rewards; Acquire actual observation data and actual state data of the photovoltaic power generation system to be adjusted, input the actual observation data into the voltage control strategy network, and obtain a first action strategy corresponding to each intelligent agent; Based on the actual state data, a first action strategy corresponding to each intelligent agent is subjected to safety constraint processing to obtain a second action strategy corresponding to each intelligent agent; Obtaining a voltage control strategy for the photovoltaic power generation system according to the second action strategy corresponding to each of the intelligent agents; The step of performing safety constraint processing on the first action strategy corresponding to each intelligent agent based on the actual state data to obtain the second action strategy corresponding to each intelligent agent includes: Based on the actual state data, the first action strategy corresponding to each intelligent agent and the trained sensitivity network, constructing the optimization constraint conditions of the safety constraint model; Constructing an optimization objective function of the safety constraint model, solving the optimization objective function based on the optimization constraint conditions, and obtaining a second action strategy corresponding to each intelligent agent if the optimization objective function has a solution; If the optimization objective function has no solution, the solution of the optimization objective function is calculated based on the penalty constraint and the relaxation constraint to obtain the second action strategy corresponding to each intelligent agent.

2. The photovoltaic system voltage control method based on deep reinforcement learning according to claim 1, characterized in that: The optimization objective function of the safety constraint model is: The optimization constraints of the safety constraint model are: Among them, a is the first action strategy corresponding to the agent, is the joint strategy of all agents, N is the number of agents, c j (s) is the constraint function in the current state, c j is the constraint limit, h(s|u j ) is the sensitivity network in the current state, u j is the weight.

3. The photovoltaic system voltage control method based on deep reinforcement learning according to claim 2, characterized in that: The method of calculating the solution of the optimization objective function based on the penalty constraint and the relaxation constraint to obtain the second action strategy corresponding to each intelligent agent is: Where ε={ε1,...,ε Z } is a slack variable, ρ is the penalty weight for violating the constraint, and a * ,ε * are the optimal safety action and the optimal slack variable respectively.

4. The photovoltaic system voltage control method based on deep reinforcement learning according to claim 1, characterized in that: The distributed partially observable Markov decision process model includes a sextuple represented by (S, A, O, Ω, T, r), wherein S is used to represent the state, O is used to represent the observation, A is used to represent the action, r is used to represent the reward, T is used to represent the state transition, and Ω is used to represent the observation transition.

5. The photovoltaic system voltage control method based on deep reinforcement learning according to claim 1, characterized in that: The distributed partially observable Markov decision process model is trained based on preset training data to obtain a voltage control strategy network, including: Acquire group training data, and use the training data to train the distributed partially observable Markov decision process model to generate an initial action strategy at a current moment; Performing security constraint processing on the initial action strategy to obtain a constrained action strategy; Inputting the constrained action strategy into the power grid simulation environment, so that the power grid simulation environment updates the state according to the initial action strategy to obtain the next state data and the next observation data at the next moment; Calculate the reward value at the current moment based on the state data at the next moment; The agent is continuously trained according to the next state data, the next observation data and the reward value at the current moment until a preset stop condition is reached, thereby obtaining a voltage control strategy network.

6. The photovoltaic system voltage control method based on deep reinforcement learning according to claim 5, characterized in that: The current reward value is calculated using the following method: Where N is the number of nodes in the power grid simulation environment, f v (·) is the voltage barrier function, v k is the voltage amplitude at node k, v ref is the reference voltage, a, b, c, d>0 are hyperparameters used to adjust the function shape and smoothness, l q (q PV ) is the reactive power generated by the photovoltaic device to regulate the voltage, K is the number of photovoltaic devices in the power grid simulation environment, q PV is the reactive power emitted by the photovoltaic device inverter at the previous moment, and α∈(0,1) is the Lagrange multiplier.

7. A photovoltaic system voltage control device based on deep reinforcement learning, characterized in that: include: Model building module, which is used to build a distributed partially observable Markov decision process model based on the voltage control task of the photovoltaic system; A training module, used for training the distributed partially observable Markov decision process model based on preset training data to obtain a voltage control strategy network, wherein each agent in the distributed partially observable Markov decision process model represents a photovoltaic power generation device, and during training, the agent interacts with a preset power grid simulation environment for feedback and rewards; A first action strategy acquisition module is used to obtain actual observation data and actual state data of the photovoltaic power generation system to be adjusted, input the actual observation data into the voltage control strategy network, and obtain a first action strategy corresponding to each intelligent agent; A second action strategy acquisition module is used to perform safety constraint processing on the first action strategy corresponding to each intelligent agent based on the actual state data to obtain a second action strategy corresponding to each intelligent agent; A voltage control strategy acquisition module, used to obtain the voltage control strategy of the photovoltaic power generation system according to the second action strategy corresponding to each of the intelligent agents; The second action strategy acquisition module is also used for: Based on the actual state data, the first action strategy corresponding to each intelligent agent and the trained sensitivity network, constructing the optimization constraint conditions of the safety constraint model; Constructing an optimization objective function of the safety constraint model, solving the optimization objective function based on the optimization constraint conditions, and obtaining a second action strategy corresponding to each intelligent agent if the optimization objective function has a solution; If the optimization objective function has no solution, the solution of the optimization objective function is calculated based on the penalty constraint and the relaxation constraint to obtain the second action strategy corresponding to each intelligent agent.

8. A storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the photovoltaic system voltage control method based on deep reinforcement learning as described in any one of claims 1-6.

Images