Reactive voltage control processing method and device of power distribution network, computer equipment, readable storage medium and program product
By using multiple rounds of operation and integrated decision network methods in the distribution network, the reactive voltage control scheme is generated and evaluated, and the problem of large reactive voltage control error in the prior art is solved, and more efficient and accurate reactive voltage control is achieved.
Patent Information
- Application Number
- CN202411580334.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2025-03-21
AI Technical Summary
The existing reactive voltage control method of distribution network uses a single evaluation network, with large estimate errors, resulting in large errors in reactive voltage control.
By using multiple rounds of operation, by obtaining considerable state data of the target distribution network, using several pre-built integrated decision-making networks and integrated evaluation networks, multiple reactive voltage control schemes are generated and evaluated, candidates with relatively large evaluation mean are selected, and parameters are adjusted according to the actual execution results until the reward conditions are met.
Through the use of multiple rounds of operation and the use of integrated decision network, the prediction error is reduced, the accuracy and efficiency of reactive voltage control is improved, and the error of the target reactive voltage control scheme is reduced.
Smart Images
Figure CN119675005A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of voltage regulation in distribution networks, and in particular to a reactive voltage control processing method, device, computer equipment, computer-readable storage medium and computer program product for distribution networks. Background Art
[0002] With the increasing popularity of renewable energy such as photovoltaic and wind turbines, the distribution network has become more complex and faces the operational challenges of voltage crossing and increased power loss. Through reactive voltage control of the distribution network, the distribution network can be more flexible to respond to various operating conditions and improve the performance and safety of the distribution network.
[0003] Currently, reactive voltage control of distribution networks can be performed based on deep reinforcement learning. However, the existing reactive voltage control method of distribution networks uses a single evaluation network to evaluate the reactive voltage control scheme available in the current round, which results in a large estimation error and a large error in reactive voltage control. Summary of the invention
[0004] Based on this, it is necessary to provide a reactive voltage control processing method, device, computer equipment, computer readable storage medium and computer program product for a distribution network in order to solve the above technical problems.
[0005] In a first aspect, the present application provides a reactive voltage control processing method for a distribution network, comprising:
[0006] Acquire initial observed state data obtained by monitoring observable state data of the target distribution network;
[0007] Perform multiple rounds of operations according to the initial observation state data;
[0008] In each round of operation, the following steps are performed: according to the current round observation state data and several pre-built integrated decision networks, several available reactive voltage control schemes for the current round are obtained; the evaluation means corresponding to the several available reactive voltage control schemes for the current round are obtained, and according to the relative size of the evaluation means, the candidate reactive voltage control scheme for the current round is determined from the several available reactive voltage control schemes for the current round; according to the actual execution result of the candidate reactive voltage control scheme for the current round in the target distribution network, the current round reward value corresponding to the candidate reactive voltage control scheme for the current round is obtained; when the current round reward value does not meet the conditions, the next round observation state data is obtained according to the actual execution result; when the reward value meets the conditions, the candidate reactive voltage control scheme is used as the target reactive voltage control scheme.
[0009] In one embodiment, before performing multiple rounds of operations according to the initial observation state data, the method further includes:
[0010] Perform multiple rounds of training based on the initial observation state data samples;
[0011] In each round of training, the following steps are performed: according to the current round observation state data samples and several current round first parameter to be adjusted networks, several current round available reactive voltage control scheme samples and several current round first loss values are obtained; according to the several current round available reactive voltage control scheme samples and several current round second parameter to be adjusted networks, the evaluation means corresponding to the several current round available reactive voltage control scheme samples and several current round second loss values are obtained; the current round candidate reactive voltage control scheme samples are determined from the current round available reactive voltage control scheme samples corresponding to the maximum value of the sum of the evaluation mean and the evaluation standard deviation; according to the current round candidate reactive voltage control scheme samples Based on the actual execution result of the target distribution network, a current round reward value sample corresponding to the current round candidate reactive voltage control scheme sample is obtained; when the current round reward value sample does not meet the conditions, a next round observation state data sample is obtained according to the actual execution result; according to the current round first loss value, the current round first parameter to be adjusted network is adjusted; according to the current round second loss value, the current round second parameter to be adjusted network is adjusted; when the reward value sample meets the conditions, according to several current round first parameter to be adjusted networks and several current round second parameter to be adjusted networks, several pre-constructed integrated decision networks and several pre-constructed integrated evaluation networks are obtained.
[0012] In one embodiment, obtaining the evaluation mean values corresponding to the plurality of currently available reactive voltage control schemes includes:
[0013] Obtaining an integrated evaluation application network according to the pre-constructed integrated evaluation network and the amplification factor;
[0014] For any current round available reactive voltage control scheme, input the any current round available reactive voltage control scheme into a plurality of integrated evaluation application networks to obtain an evaluation value of the any current round available reactive voltage control scheme corresponding to each integrated evaluation application network;
[0015] The evaluation value of each integrated evaluation application network corresponding to any of the reactive voltage control schemes available in the current round is solved for mean, so as to obtain the evaluation mean corresponding to any of the reactive voltage control schemes available in the current round.
[0016] In one embodiment, the step of inputting any of the available reactive voltage control schemes for the current round into a plurality of integrated evaluation application networks to obtain an evaluation value of any of the available reactive voltage control schemes for the current round corresponding to each integrated evaluation application network includes:
[0017] For any integrated evaluation application network, input any of the reactive voltage control schemes available in the current round into any of the integrated evaluation application networks to obtain a power loss state estimate and a voltage violation state estimate corresponding to any of the reactive voltage control schemes available in the current round;
[0018] According to the power loss state estimated value and the voltage violation state estimated value, an evaluation value of any current round available reactive voltage control scheme corresponding to any integrated evaluation application network is obtained.
[0019] In one embodiment, determining the candidate reactive voltage control scheme for the current round from the plurality of available reactive voltage control schemes for the current round according to the relative size of the evaluation mean value includes:
[0020] Randomly selecting the plurality of reactive voltage control schemes available in the current round to obtain a plurality of candidate reactive voltage control schemes to be waited for in the current round; the number of the candidate reactive voltage control schemes to be waited for in the current round is less than the number of the reactive voltage control schemes available in the current round;
[0021] The current round candidate reactive power voltage control scheme corresponding to the maximum evaluation mean value is determined as the current round candidate reactive power voltage control scheme.
[0022] In one embodiment, obtaining the current round reward value corresponding to the current round candidate reactive voltage control scheme according to the actual execution result of the current round candidate reactive voltage control scheme in the target distribution network includes:
[0023] Execute the candidate reactive voltage control scheme of the current round in the target distribution network environment to obtain an actual execution result of the candidate reactive voltage control scheme of the current round in the target distribution network;
[0024] According to the actual execution result, the active power loss state observation value and the voltage violation state observation value corresponding to the current round candidate reactive voltage control scheme are obtained to obtain the current round reward value corresponding to the current round candidate reactive voltage control scheme.
[0025] In a second aspect, the present application also provides a reactive voltage control processing device for a distribution network, comprising:
[0026] An observation state data acquisition module is used to acquire initial observation state data obtained by monitoring observable state data of the target distribution network;
[0027] A multi-round operation module is used to perform multiple rounds of operations according to the initial observation state data; in each round of operation, the following steps are executed: according to the current round observation state data and several pre-built integrated decision networks, several available reactive voltage control schemes for the current round are obtained; the evaluation means corresponding to the several available reactive voltage control schemes for the current round are obtained, and according to the relative size of the evaluation means, the candidate reactive voltage control scheme for the current round is determined from the several available reactive voltage control schemes for the current round; according to the actual execution result of the candidate reactive voltage control scheme for the current round in the target distribution network, the current round reward value corresponding to the candidate reactive voltage control scheme for the current round is obtained; when the current round reward value does not meet the conditions, the next round observation state data is obtained according to the actual execution result; when the reward value meets the conditions, the candidate reactive voltage control scheme is used as the target reactive voltage control scheme.
[0028] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.
[0029] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the above method.
[0030] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to execute the above method.
[0031] The reactive voltage control processing method, device, computer equipment, computer-readable storage medium and computer program product of the distribution network described above obtain initial observed state data obtained by monitoring the observable state data of the target distribution network; perform multiple rounds of operations based on the initial observed state data; in each round of operation, execute the following steps: obtain several available reactive voltage control schemes for the current round based on the observed state data of the current round and several pre-built integrated decision networks; obtain the evaluation means corresponding to several available reactive voltage control schemes for the current round, and determine the candidate reactive voltage control scheme for the current round from several available reactive voltage control schemes for the current round based on the relative size of the evaluation means; obtain the current round reward value corresponding to the candidate reactive voltage control scheme for the current round based on the actual execution result of the candidate reactive voltage control scheme for the current round in the target distribution network; when the reward value does not meet the conditions, obtain the observed state data for the next round based on the actual execution result; when the reward value meets the conditions, use the candidate reactive voltage control scheme as the target reactive voltage control scheme. The present application determines the candidate reactive voltage control scheme for the current round according to the relative sizes of the evaluation means corresponding to several available reactive voltage control schemes for the current round, which can reduce the estimation error, thereby reducing the error of the candidate reactive voltage control scheme for the current round, and thus reducing the error of the target reactive voltage control scheme. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1 An application environment diagram of a reactive voltage control processing method for a distribution network in one embodiment;
[0034] Figure 2 It is a flowchart of a reactive voltage control processing method of a power distribution network in one embodiment;
[0035] Figure 3 A schematic diagram of a flow chart of determining a sample of a candidate reactive power voltage control scheme for a current round by adopting an optimistic exploration strategy in one embodiment;
[0036] Figure 4 A schematic diagram of a training process of a network to be adjusted for the second parameter in the current round in an embodiment;
[0037] Figure 5 A schematic diagram of a flow chart of randomly selecting an available reactive voltage control scheme for the current round in one embodiment;
[0038] Figure 6A schematic diagram of an integrated reinforcement learning framework in one embodiment;
[0039] Figure 7 It is a structural block diagram of a reactive voltage control processing device of a power distribution network in one embodiment;
[0040] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0042] The embodiment of the present application provides a reactive voltage control processing method for a distribution network, and the embodiment can be executed by a computer device, such as Figure 1 As shown, the computer device can obtain the initial observed state data obtained by monitoring the observable state data of the target distribution network, and perform multiple rounds of operations based on the initial observed state data to obtain the target reactive voltage control scheme. It can be understood that the computer device can be implemented through a server, a terminal, or an interactive system between a terminal and a server. In this embodiment, the method includes Figure 2 The steps shown are:
[0043] Step S201, obtaining initial observed state data obtained by monitoring observable state data of a target distribution network.
[0044] The target distribution network can be an unobservable distribution network, which belongs to an active distribution network. Some state data of the target distribution network cannot be observed, and some state data can be observed. The observable state data is called observable state data. The observable state data of the target distribution network can include bus voltage amplitude, branch current amplitude, bus active power and bus reactive power.
[0045] By monitoring the observable state data of the target distribution network, initial observed state data can be obtained.
[0046] The inverter-based reactive voltage of the target distribution network can be controlled according to the observable state data of the target distribution network, and the reactive voltage control process of the target distribution network can be modeled as a reactive voltage control process with time-delay reward. Partial observable markov decision process (POMDP), where is the state space, is the observation space, which can also be called observation state data. is the action space, which can also be called the reactive voltage control scheme. is the reward value (reward function). is in a given state and actions In case of transition to state probability.
[0047] The status It reflects the working conditions of the target distribution network and contains sufficient information for the deep reinforcement learning agent to make decisions. is the observable state of the target distribution network, including active load , reactive load , active output of new energy generators and reactive power output , reactive power output of reactive power adjustable device , line current I, bus voltage amplitude measured in real time by node voltage , branch current amplitude , Bus active power and bus reactive power , also includes controllable reactive power output determined by a deep reinforcement learning agent, , are the observable node voltages and branch currents. is a linear mapping of actions of a deep reinforcement learning agent, without the need for measurement. is output by the integrated decision network. The activation function of the integrated decision network is the hyperbolic tangent function (Tanh). The controllable variables executed in the target distribution network is the reactive voltage control scheme from (-1, 1) to the output limit The reward includes the reward for active power loss of the target distribution network and the reward for voltage violation, which cannot be obtained from limited real-time measurement data and need to collect smart meter data to obtain these data.
[0048] Step S202, performing multiple rounds of operations according to the initial observed state data; in each round of operation, executing the following steps: obtaining several available reactive voltage control schemes for the current round according to the observed state data for the current round and several pre-built integrated decision networks; obtaining evaluation means corresponding to several available reactive voltage control schemes for the current round, and determining the candidate reactive voltage control scheme for the current round from among several available reactive voltage control schemes for the current round according to the relative sizes of the evaluation means; obtaining the current round reward value corresponding to the candidate reactive voltage control scheme for the current round according to the actual execution results of the candidate reactive voltage control scheme for the current round in the target distribution network; when the reward value does not meet the conditions, obtaining the observed state data for the next round according to the actual execution results; when the reward value meets the conditions, taking the candidate reactive voltage control scheme as the target reactive voltage control scheme.
[0049] Multiple rounds of operations can be performed according to the initial observation state data to obtain the target reactive voltage control scheme of the distribution network. Among them, the initial observation state data is used as the current round observation state data of the first round of operation. In each round of operation, the following steps are performed: the current round observation state data is input into several pre-built integrated decision networks to obtain several current round available reactive voltage control schemes; the evaluation means corresponding to several current round available reactive voltage control schemes are obtained, and according to the relative size of the evaluation means, the current round candidate reactive voltage control scheme is determined among several current round available reactive voltage control schemes; according to the actual execution result of the current round candidate reactive voltage control scheme in the target distribution network, the current round reward value corresponding to the current round candidate reactive voltage control scheme is obtained; when the reward value does not meet the conditions, it indicates that after the execution of the current round candidate reactive voltage control scheme, the performance and safety of the distribution network are poor, then according to the actual execution result, the next round of observation state data is obtained; when the reward value meets the conditions, it indicates that after the execution of the current round candidate reactive voltage control scheme, the performance and safety of the distribution network are good, then the candidate reactive voltage control scheme is used as the target reactive voltage control scheme.
[0050] Specifically, at each time step, that is, at each round of operation, the state of the target distribution network environment is s t ∈S, the deep reinforcement learning agent can obtain the current round of observation state data s ot ∈O, and then execute the current round candidate reactive power voltage control scheme a according to the strategy π:O→A t ∈A. After executing the current round of candidate reactive power and voltage control schemes, the target distribution network environment transfers to the next state s （t+1） , the deep reinforcement learning agent receives the next round of observation state data s （o，t+1） .
[0051] In the reactive voltage control processing method of the above-mentioned distribution network, the candidate reactive voltage control scheme for the current round is determined according to the relative size of the evaluation mean values corresponding to several available reactive voltage control schemes for the current round, which can reduce the estimation error, thereby reducing the error of the candidate reactive voltage control scheme for the current round, and thus reducing the error of the target reactive voltage control scheme.
[0052] In one of the embodiments, before performing multiple rounds of operations based on the initial observed state data, the method provided by the present application also includes: performing multiple rounds of training based on the initial observed state data samples; in each round of training, executing the following steps: obtaining a number of current round available reactive voltage control scheme samples and a number of current round first loss values based on the current round observed state data samples and a number of current round first parameter to be adjusted networks; obtaining a number of current round available reactive voltage control scheme samples and a number of current round second parameter to be adjusted networks, and obtaining evaluation means and a number of current round second loss values corresponding to a number of current round available reactive voltage control scheme samples; obtaining the current round available reactive voltage control scheme corresponding to the maximum value of the sum of the evaluation mean and the evaluation standard deviation. Determine the current round candidate reactive voltage control scheme samples from the samples; obtain the current round reward value samples corresponding to the current round candidate reactive voltage control scheme samples according to the actual execution results of the current round candidate reactive voltage control scheme samples in the target distribution network; when the reward value samples do not meet the conditions, obtain the next round observation state data samples according to the actual execution results; adjust the current round first parameter to be adjusted network according to the current round first loss value; adjust the current round second parameter to be adjusted network according to the current round second loss value; when the reward value samples meet the conditions, obtain several pre-built integrated decision networks and several pre-built integrated evaluation networks according to several current round first parameter to be adjusted networks and several current round second parameter to be adjusted networks.
[0053] Multiple rounds of operations can be performed based on the initial observation state data sample, wherein the initial observation state data sample is used as the current round of observation state data of the first round of operations.
[0054] In each round of training, the following steps are performed: the current round observation state data samples are input into several current round first parameter adjustment networks to obtain several current round available reactive voltage control scheme samples and several current round first loss values.
[0055] Several networks with the first parameters to be adjusted in the current round can use integrated decision making To learn random strategies ,in The integrated decision is a compressed Gaussian strategy, and its parameterization is shown in Equation (1).
[0056] (1)
[0057] in is the output of the integrated decision, tanh is the hyperbolic tangent function, , It is The mean and standard deviation of the neural network for the first decision, also known as the The mean and standard deviation of the first parameter of the current round to be adjusted network, is the dot product.
[0058] According to a number of current round available reactive voltage control scheme samples and a number of current round second parameter to-be-adjusted networks, evaluation means corresponding to a number of current round available reactive voltage control scheme samples and a number of current round second loss values are obtained.
[0059] Several current round second parameter adjustment networks can use integrated evaluation The state-action value functions for power loss and voltage violation are learned separately, where is the index of the double-Q network, is the index of the integrated evaluation. The integrated target evaluation network can also be introduced , as the second parameter to be adjusted network in the current round, the initial model parameters of the integrated target evaluation network are obtained from the integrated evaluation network The model parameters copied from are then soft-updated as shown in Equation (2).
[0060] (2)
[0061] in Soft update operation is used to slowly track the ensemble evaluation network , the error accumulation and estimation variance can be reduced by the mean value estimation.
[0062] During the training process of the first parameter to be adjusted network, an effective exploration method should generate current round candidate reactive voltage control scheme samples to explore promising areas and help the agent collect useful current round candidate reactive voltage control scheme samples about the environment to discover high-return strategies. The first parameter to be adjusted network obtains the current round observation state data samples and outputs K current round candidate reactive voltage control scheme samples, as shown in formula (3).
[0063] (3)
[0064] Due to the random training process of the first parameter to be adjusted network, such as the parameters of the first parameter to be adjusted network that are randomly initialized, these first parameter to be adjusted networks will have a certain degree of diversity, where each current round candidate reactive voltage control scheme sample is slightly different. Some of these current round candidate reactive voltage control scheme samples may have better exploration capabilities. Since each step only executes the current round candidate reactive voltage control scheme samples for the active distribution network environment, it is necessary to select the best current round candidate reactive voltage control scheme sample for execution.
[0065] An optimistic exploration strategy can be used to determine the candidate reactive power and voltage control scheme samples for the current round, such as Figure 3 As shown, the first parameter to be adjusted network obtains the current round of observed state data samples and outputs the current round of candidate reactive voltage control scheme samples; the second parameter to be adjusted network includes an integrated power loss evaluation network and an integrated voltage violation evaluation network, and the current round of candidate reactive voltage control scheme samples are input into the integrated power loss evaluation network and the integrated voltage violation evaluation network respectively to obtain the evaluation mean and evaluation standard deviation corresponding to the current round of available reactive voltage control scheme samples; the current round of candidate reactive voltage control scheme samples are determined from the current round of available reactive voltage control scheme samples corresponding to the maximum value of the sum of the evaluation mean and the evaluation standard deviation, wherein the calculation formula for the sum of the evaluation mean and the evaluation standard deviation is shown in formula (4).
[0066] (4)
[0067] in is a hyperparameter that controls the intensity of optimistic exploration, and the evaluation value The calculation formula is shown in the formula, the evaluation mean The calculation formula of is shown in formula (6), the standard deviation The calculation formula of is shown in formula (7).
[0068] (5)
[0069] (6)
[0070] (7)
[0071] in is the standard deviation of the reactive power voltage control solution samples available in the current round, which is used to represent the reactive power voltage control solution samples available in the current round. The estimated uncertainty of . Expressing higher uncertainty, the estimated value is often not accurate enough. The optimistic exploration technique selects reactive voltage control scheme samples in less explored areas, which helps the deep reinforcement learning agent to obtain more data in the area, improve the estimation accuracy, speed up the learning process, and thus alleviate the problem of delayed rewards.
[0072] According to the actual execution result of the current round candidate reactive power and voltage control scheme sample in the target distribution network, the current round reward value sample corresponding to the current round candidate reactive power and voltage control scheme sample is obtained.
[0073] When the reward value sample does not meet the conditions, the next round of observation state data samples are obtained according to the actual execution results. By maximizing the first loss value of the current round, the first parameter of the current round to be adjusted network is adjusted, as shown in formula (8).
[0074] (8)
[0075] in ,and:
[0076] (9)
[0077] (10)
[0078] All networks whose first parameters are to be adjusted in the current round share the same temperature parameter , the temperature parameter can be adjusted by minimizing the loss , as shown in formula (11).
[0079] (11)
[0080] in is the entropy target. Temperature parameter It is used to balance the exploration intensity of the network whose first parameter is to be adjusted in the current round. In this way, the temperature parameter can be adjusted adaptively. , so as to achieve a better balance between exploration and utilization of the network with the first parameter to be adjusted in the current round during training.
[0081] Figure 4 The training process of the second parameter to be adjusted network in the current round, the second parameter to be adjusted network includes an integrated power loss evaluation network and an integrated voltage violation evaluation network. The second loss value of the current round can be a time difference (TemporalDifference, TD) loss function. The second parameter to be adjusted network in the current round can be adjusted by minimizing the second loss value of the current round, as shown in equations (12) and (13).
[0082] (12)
[0083] (13)
[0084] in is the time difference loss function of the integrated power loss evaluation network, is the time difference loss function of the integrated voltage violation evaluation network, The definition of is shown in equations (14) and (15).
[0085] (14)
[0086] (15)
[0087] , Is the second parameter of the current round to be adjusted network Parameters, It is the temperature parameter of the entropy term of the network to be adjusted for the second parameter in the current round. and The mean value of is defined as shown in equations (16) and (17).
[0088] (16)
[0089] (17)
[0090] The mean value of the entropy term of the network whose second parameter is to be adjusted in the current round is shown in formula (18).
[0091] (18)
[0092] When the reward value sample meets the conditions, it indicates that after executing the current round of candidate reactive power and voltage control scheme samples, the performance and safety of the distribution network are good. Then, several networks with the first parameters to be adjusted in the current round are used as several pre-built integrated decision networks, and several networks with the second parameters to be adjusted in the current round are used as several pre-built integrated evaluation networks.
[0093] In this embodiment, multiple rounds of training are performed based on the initial observation state data samples to obtain several pre-built integrated decision networks and several pre-built integrated evaluation networks with better performance.
[0094] In one of the embodiments, the evaluation means corresponding to several available reactive voltage control schemes of the current round are obtained, and the specific steps are as follows: according to the pre-constructed integrated evaluation network and the amplification factor, an integrated evaluation application network is obtained; for any available reactive voltage control scheme of the current round, any available reactive voltage control scheme of the current round is input into several integrated evaluation application networks to obtain the evaluation value of any available reactive voltage control scheme of the current round corresponding to each integrated evaluation application network; for the evaluation value of any available reactive voltage control scheme of the current round corresponding to each integrated evaluation application network, the mean is solved to obtain the evaluation mean corresponding to any available reactive voltage control scheme of the current round.
[0095] In the existing deep reinforcement learning-based reactive voltage control (smart grid virtual voltage-reactive voltage control) method, the penalty coefficient for voltage violation needs to be adjusted carefully. If the penalty coefficient is smaller, the deep reinforcement learning policy will focus more on learning to minimize power loss and ignore eliminating voltage violations. If is too large, the learning process of the deep reinforcement learning policy may become unstable.
[0096] To solve this problem, the reactive voltage control processing method of the distribution network provided in this application sets a slightly smaller penalty coefficient when training the deep reinforcement learning algorithm. To mitigate the unstable behavior of the learning process, an amplification factor is set during the application phase of the deep reinforcement learning algorithm. To expand the penalty coefficient of voltage violation so as to select a safe action, thereby having better performance in eliminating voltage violations. Specifically, the pre-constructed integrated evaluation network includes an integrated power loss evaluation network and an integrated voltage violation evaluation network. According to the pre-constructed integrated evaluation network and the amplification factor, an integrated evaluation application network can be obtained, as shown in formula (19). In addition, in the reactive voltage control processing method of the distribution network provided in the present application, the reactive voltage control scheme available in the current round is evaluated from the two aspects of power loss and voltage violation, which improves the flexibility of adjusting the penalty factor of violation evaluation.
[0097] (19)
[0098] in, is the mean evaluation value output by the integrated evaluation application network, is the power loss evaluation mean value output by the integrated power loss evaluation network, is the voltage violation evaluation mean value output by the integrated voltage violation evaluation network, is the magnification factor.
[0099] For any current round of available reactive voltage control scheme, any current round of available reactive voltage control scheme is input into several integrated evaluation application networks to obtain the evaluation value of any current round of available reactive voltage control scheme corresponding to each integrated evaluation application network; for the evaluation value of any current round of available reactive voltage control scheme corresponding to each integrated evaluation application network, the mean is solved to obtain the evaluation mean corresponding to any current round of available reactive voltage control scheme.
[0100] In this embodiment, an integrated evaluation application network is obtained based on a pre-constructed integrated evaluation network and amplification factor; based on the integrated evaluation application network, evaluation means corresponding to several current round available reactive voltage control schemes are obtained. Compared with the standard soft decision evaluation algorithm using a single evaluation network, the integrated evaluation application network reduces the estimated variance of the current round available reactive voltage control schemes by averaging its output values, which can alleviate the overestimation problem and unstable convergence behavior in the temporal difference learning process, thereby reducing the error of the target reactive voltage control scheme.
[0101] In one of the embodiments, any current round available reactive voltage control scheme is input into several integrated evaluation application networks to obtain an evaluation value of any current round available reactive voltage control scheme corresponding to each integrated evaluation application network. The specific steps are as follows: for any integrated evaluation application network, any current round available reactive voltage control scheme is input into any integrated evaluation application network to obtain a power loss state estimated value and a voltage violation state estimated value corresponding to any current round available reactive voltage control scheme; based on the power loss state estimated value and the voltage violation state estimated value, an evaluation value of any current round available reactive voltage control scheme corresponding to any integrated evaluation application network is obtained.
[0102] For any integrated evaluation application network, any current round of available reactive power voltage control scheme is input into any integrated evaluation application network, and the integrated evaluation application network can use the integrated evaluation The state-action value functions of power loss and voltage violation are learned respectively, and the power loss state estimate and voltage violation state estimate corresponding to any reactive voltage control scheme available in the current round are obtained. is the index of the double-Q network, It is the index of integrated evaluation.
[0103] The sum of the power loss state estimation value and the voltage violation state estimation value may be taken as the evaluation value of any current round available reactive power voltage control scheme corresponding to any integrated evaluation application network.
[0104] In this embodiment, the integrated evaluation application network evaluates the available reactive voltage control scheme for the current wheel from the two aspects of power loss and voltage violation, and can evaluate the available reactive voltage control scheme for the front wheel more accurately.
[0105] In one of the embodiments, a candidate reactive voltage control scheme for the current round is determined from among several available reactive voltage control schemes for the current round according to the relative size of the evaluation mean, and the specific steps are as follows: several available reactive voltage control schemes for the current round are randomly selected to obtain several candidate reactive voltage control schemes for the current round; the number of candidate reactive voltage control schemes for the current round is less than the available reactive voltage control schemes for the current round; and the candidate reactive voltage control scheme for the current round corresponding to the maximum value of the evaluation mean is determined as the candidate reactive voltage control scheme for the current round.
[0106] like Figure 5 As shown, several reactive voltage control schemes available in the current round are randomly selected to obtain several candidate reactive voltage control schemes in the current round, as shown in formula (20). The number of candidate reactive voltage control schemes in the current round is less than the number of available reactive voltage control schemes in the current round.
[0107] (20)
[0108] in .
[0109] The current round candidate reactive power voltage control scheme corresponding to the maximum evaluation mean value is determined as the current round candidate reactive power voltage control scheme.
[0110] In this embodiment, by randomly selecting several reactive voltage control schemes available in the current round, the optimization capability of scheme selection is improved, and the scope of exploring different schemes is increased, which helps to find a better candidate reactive voltage control scheme for the current round.
[0111] In one of the embodiments, based on the actual execution results of the current round candidate reactive voltage control scheme in the target distribution network, a current round reward value corresponding to the current round candidate reactive voltage control scheme is obtained, and the specific steps are as follows: the current round candidate reactive voltage control scheme is executed in the target distribution network environment to obtain the actual execution results of the current round candidate reactive voltage control scheme in the target distribution network; based on the actual execution results, the active power loss state observation value and the voltage violation state observation value corresponding to the current round candidate reactive voltage control scheme are obtained to obtain the current round reward value corresponding to the current round candidate reactive voltage control scheme.
[0112] After obtaining the current round of candidate reactive voltage control schemes, the current round of candidate reactive voltage control schemes can be executed in the target distribution network environment. The active power injection and voltage amplitude after the current round of candidate reactive voltage control schemes are executed in the target distribution network environment can be measured by a smart meter to obtain the actual execution result of the current round of candidate reactive voltage control schemes in the target distribution network; based on the actual execution result, the active power loss state observation value and the voltage violation state observation value corresponding to the current round of candidate reactive voltage control schemes are obtained.
[0113] The smart meter data is uploaded to the control center at a delay time interval T, and at time t, only the reward r_(tT) is obtained.
[0114] Active power loss state observation (reward for active power loss) The calculation formula of is shown in formula (21).
[0115] (twenty one)
[0116] in represents the power of the i-th bus before executing the current round of candidate reactive power and voltage control schemes, represents the power of the i-th bus after executing the current round of candidate reactive voltage control scheme, is the set of all buses.
[0117] Voltage violation state observation value (reward for voltage violation) As shown in formula (22).
[0118] (twenty two)
[0119] in, is the set of all buses, is the penalty factor for voltage violation, It is The node voltage of each node, is the upper limit of the normal operating range of voltage, is the lower bound of the normal operating range of voltage. The negative sign is added because deep reinforcement learning aims to maximize the reward.
[0120] The current round reward value corresponding to the current round candidate reactive voltage control scheme can be obtained according to the active power loss state observation value and the voltage violation state observation value corresponding to the current round candidate reactive voltage control scheme.
[0121] In this embodiment, according to the actual execution results of the current round of candidate reactive voltage control schemes in the target distribution network, the active power loss state observation value and the voltage violation state observation value corresponding to the current round of candidate reactive voltage control schemes are obtained to obtain the current round reward value corresponding to the current round of candidate reactive voltage control schemes. The actual execution results of the current round of candidate reactive voltage control schemes in the target distribution network are evaluated from the two aspects of power loss and voltage violation, and the actual execution results of the current round of candidate reactive voltage control schemes in the target distribution network can be evaluated more accurately.
[0122] In one embodiment, a time delay reward is obtained based on the measured value of the smart meter. Because the smart meter has a time delay characteristic, in order to mitigate the adverse effects of the observed state data and the time delay reward, an off-strategy integrated decision-evaluation deep reinforcement learning method is proposed. Figure 6 As shown, the delayed smart meter measurement value and limited real-time measurements , constructed as a delayed reward and some observations . In order to solve the problem of asynchronous rewards and partial observations, the data buffer first collects this data and then constructs it into Markov decision process data based on the timestamp, which is different from the standard deep reinforcement learning setting where Markov decision process data can be collected simultaneously. Due to the delay of the reward, the Markov decision process data collected to the data buffer is also delayed. The integrated deep reinforcement learning agent is trained off-policy, sampling a batch of data from the data buffer to train the behavior and evaluation network, and obtains the reactive voltage control scheme based on the observed state data. Among them, the off-policy integrated decision-evaluation deep reinforcement learning method is based on the soft decision evaluation algorithm and the dual evaluation framework, and an integrated soft decision dual evaluation deep reinforcement learning is designed, which is also applicable to other decision-evaluation deep reinforcement learning methods.
[0123] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0124] Based on the same inventive concept, the embodiment of the present application also provides a reactive voltage control processing device for a distribution network for implementing the reactive voltage control processing method for the distribution network involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more reactive voltage control processing devices for distribution networks provided below can refer to the limitations of the reactive voltage control processing method for distribution networks above, and will not be repeated here.
[0125] In an exemplary embodiment, Figure 7 As shown, a reactive voltage control processing device for a distribution network is provided, wherein:
[0126] The observed state data acquisition module 701 is used to acquire initial observed state data obtained by monitoring observable state data of the target distribution network;
[0127] The multi-round operation module 702 is used to perform multi-round operations according to the initial observation state data; in each round of operation, the following steps are executed: according to the current round observation state data and several pre-built integrated decision networks, several available reactive voltage control schemes for the current round are obtained; the evaluation means corresponding to the several available reactive voltage control schemes for the current round are obtained, and according to the relative size of the evaluation means, the candidate reactive voltage control scheme for the current round is determined from the several available reactive voltage control schemes for the current round; according to the actual execution result of the candidate reactive voltage control scheme for the current round in the target distribution network, the current round reward value corresponding to the candidate reactive voltage control scheme for the current round is obtained; when the current round reward value does not meet the conditions, the next round observation state data is obtained according to the actual execution result; when the reward value meets the conditions, the candidate reactive voltage control scheme is used as the target reactive voltage control scheme.
[0128] In one of the embodiments, the device also includes a network training module, which is used to: perform multiple rounds of training based on the initial observation state data samples; in each round of training, perform the following steps: obtain a number of current round available reactive voltage control scheme samples and a number of current round first loss values based on the current round observation state data samples and a number of current round first parameter to be adjusted networks; obtain the evaluation mean values and a number of current round second loss values corresponding to the several current round available reactive voltage control scheme samples based on the several current round available reactive voltage control scheme samples and a number of current round second parameter to be adjusted networks; determine the candidate reactive voltage control scheme for the current round from the current round available reactive voltage control scheme samples corresponding to the maximum value of the sum of the evaluation mean and the evaluation standard deviation; According to the actual execution results of the candidate reactive voltage control scheme samples in the current round in the target distribution network, the current round reward value samples corresponding to the candidate reactive voltage control scheme samples in the current round are obtained; when the reward value samples in the current round do not meet the conditions, the next round observation state data samples are obtained according to the actual execution results; according to the first loss value in the current round, the network to be adjusted with the first parameter in the current round is adjusted; according to the second loss value in the current round, the network to be adjusted with the second parameter in the current round is adjusted; when the reward value samples meet the conditions, according to several networks to be adjusted with the first parameter in the current round and several networks to be adjusted with the second parameter in the current round, several pre-built integrated decision networks and several pre-built integrated evaluation networks are obtained.
[0129] In one embodiment, the multi-round operation module 702 is also used to: obtain an integrated evaluation application network based on the pre-constructed integrated evaluation network and the amplification factor; for any current round of available reactive voltage control schemes, input any current round of available reactive voltage control schemes into several integrated evaluation application networks to obtain an evaluation value of any current round of available reactive voltage control schemes corresponding to each integrated evaluation application network; perform mean solution on the evaluation value of any current round of available reactive voltage control schemes corresponding to each integrated evaluation application network to obtain an evaluation mean corresponding to any current round of available reactive voltage control schemes.
[0130] In one embodiment, the multi-round operation module 702 is also used to: for any integrated evaluation application network, input any current round available reactive voltage control scheme into any integrated evaluation application network to obtain a power loss state estimated value and a voltage violation state estimated value corresponding to any current round available reactive voltage control scheme; and obtain an evaluation value of any current round available reactive voltage control scheme corresponding to any integrated evaluation application network based on the power loss state estimated value and the voltage violation state estimated value.
[0131] In one embodiment, the multi-round operation module 702 is also used to: randomly select the several current round available reactive voltage control schemes to obtain several current round candidate reactive voltage control schemes; the number of the current round candidate reactive voltage control schemes is less than the current round available reactive voltage control schemes; and determine the current round candidate reactive voltage control scheme corresponding to the maximum evaluation mean as the current round candidate reactive voltage control scheme.
[0132] In one of the embodiments, the multi-round operation module 702 is also used to: execute the current round candidate reactive voltage control scheme in the target distribution network environment to obtain the actual execution result of the current round candidate reactive voltage control scheme in the target distribution network; obtain the active power loss state observation value and the voltage violation state observation value corresponding to the current round candidate reactive voltage control scheme according to the actual execution result, so as to obtain the current round reward value corresponding to the current round candidate reactive voltage control scheme.
[0133] Each module in the reactive voltage control processing device for the distribution network can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0134] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of an embodiment of a reactive voltage control processing method for a distribution network. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a reactive voltage control processing method for a distribution network is implemented.
[0135] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0136] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0137] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0138] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0140] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0142] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A reactive voltage control processing method for a distribution network, characterized in that: The method comprises: Acquire initial observed state data obtained by monitoring observable state data of the target distribution network; Perform multiple rounds of operations according to the initial observation state data; In each round of operation, the following steps are performed: according to the current round observation state data and several pre-built integrated decision networks, several available reactive voltage control schemes for the current round are obtained; the evaluation means corresponding to the several available reactive voltage control schemes for the current round are obtained, and according to the relative size of the evaluation means, the candidate reactive voltage control scheme for the current round is determined from the several available reactive voltage control schemes for the current round; according to the actual execution result of the candidate reactive voltage control scheme for the current round in the target distribution network, the current round reward value corresponding to the candidate reactive voltage control scheme for the current round is obtained; when the current round reward value does not meet the conditions, the next round observation state data is obtained according to the actual execution result; when the reward value meets the conditions, the candidate reactive voltage control scheme is used as the target reactive voltage control scheme.
2. The method according to claim 1, characterized in that: Before performing multiple rounds of operations according to the initial observation state data, the method further includes: Perform multiple rounds of training based on the initial observation state data samples; In each round of training, the following steps are performed: according to the current round observation state data samples and several current round first parameter to be adjusted networks, several current round available reactive voltage control scheme samples and several current round first loss values are obtained; according to the several current round available reactive voltage control scheme samples and several current round second parameter to be adjusted networks, the evaluation means corresponding to the several current round available reactive voltage control scheme samples and several current round second loss values are obtained; the current round candidate reactive voltage control scheme samples are determined from the current round available reactive voltage control scheme samples corresponding to the maximum value of the sum of the evaluation mean and the evaluation standard deviation; according to the current round candidate reactive voltage control scheme samples Based on the actual execution result of the target distribution network, a current round reward value sample corresponding to the current round candidate reactive voltage control scheme sample is obtained; when the current round reward value sample does not meet the conditions, a next round observation state data sample is obtained according to the actual execution result; according to the current round first loss value, the current round first parameter to be adjusted network is adjusted; according to the current round second loss value, the current round second parameter to be adjusted network is adjusted; when the reward value sample meets the conditions, according to several current round first parameter to be adjusted networks and several current round second parameter to be adjusted networks, several pre-constructed integrated decision networks and several pre-constructed integrated evaluation networks are obtained.
3. The method according to claim 2, characterized in that The obtaining of the evaluation mean values corresponding to the plurality of reactive voltage control schemes available in the current round includes: Obtaining an integrated evaluation application network according to the pre-constructed integrated evaluation network and the amplification factor; For any current round available reactive voltage control scheme, input the any current round available reactive voltage control scheme into a plurality of integrated evaluation application networks to obtain an evaluation value of the any current round available reactive voltage control scheme corresponding to each integrated evaluation application network; The evaluation value of each integrated evaluation application network corresponding to any of the reactive voltage control schemes available in the current round is solved for mean, so as to obtain the evaluation mean corresponding to any of the reactive voltage control schemes available in the current round.
4. The method according to claim 3, characterized in that: The step of inputting any of the available reactive voltage control schemes of the current round into a plurality of integrated evaluation application networks to obtain an evaluation value of any of the available reactive voltage control schemes of the current round corresponding to each integrated evaluation application network comprises: For any integrated evaluation application network, input any of the reactive voltage control schemes available in the current round into any of the integrated evaluation application networks to obtain a power loss state estimate and a voltage violation state estimate corresponding to any of the reactive voltage control schemes available in the current round; According to the power loss state estimated value and the voltage violation state estimated value, an evaluation value of any current round available reactive voltage control scheme corresponding to any integrated evaluation application network is obtained.
5. The method according to claim 1, characterized in that Determining the candidate reactive voltage control scheme for the current round from the plurality of available reactive voltage control schemes for the current round according to the relative size of the evaluation mean value comprises: Randomly selecting the plurality of reactive voltage control schemes available in the current round to obtain a plurality of candidate reactive voltage control schemes to be waited for in the current round; the number of the candidate reactive voltage control schemes to be waited for in the current round is less than the number of the reactive voltage control schemes available in the current round; The current round candidate reactive power voltage control scheme corresponding to the maximum evaluation mean value is determined as the current round candidate reactive power voltage control scheme.
6. The method according to claim 1, characterized in that The obtaining, according to the actual execution result of the current round candidate reactive voltage control scheme in the target distribution network, a current round reward value corresponding to the current round candidate reactive voltage control scheme comprises: Execute the candidate reactive voltage control scheme of the current round in the target distribution network environment to obtain an actual execution result of the candidate reactive voltage control scheme of the current round in the target distribution network; According to the actual execution result, the active power loss state observation value and the voltage violation state observation value corresponding to the current round candidate reactive voltage control scheme are obtained to obtain the current round reward value corresponding to the current round candidate reactive voltage control scheme.
7. A reactive voltage control processing device for a distribution network, characterized in that: The device comprises: An observation state data acquisition module is used to acquire initial observation state data obtained by monitoring observable state data of the target distribution network; A multi-round operation module is used to perform multiple rounds of operations according to the initial observation state data; in each round of operation, the following steps are executed: according to the current round observation state data and several pre-built integrated decision networks, several available reactive voltage control schemes for the current round are obtained; the evaluation means corresponding to the several available reactive voltage control schemes for the current round are obtained, and according to the relative size of the evaluation means, the candidate reactive voltage control scheme for the current round is determined from the several available reactive voltage control schemes for the current round; according to the actual execution result of the candidate reactive voltage control scheme for the current round in the target distribution network, the current round reward value corresponding to the candidate reactive voltage control scheme for the current round is obtained; when the current round reward value does not meet the conditions, the next round observation state data is obtained according to the actual execution result; when the reward value meets the conditions, the candidate reactive voltage control scheme is used as the target reactive voltage control scheme.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.